Words: 2989 Approximate Reading Time: 20-25 minutes
In the previous essay I examined review scores and the reasons why they don’t really mean anything. That is, even when we are trying our best as reviewers to be honest and forthright with our scoring, we end up coming up with scores that fail to truly match up to expectations. They wind up being vague, arbitrary, and internally inconsistent.
But looking at review scores from the perspective of the reviewer is actually not that valuable. Many reviewers understand these issues in one way or another, and are merely trying to navigate a complex environment. I ended that essay by arguing that attempting to score a game in these ways is not the actual issue. We merely need to be aware of the limitations.
The bigger target that we should be focused on is the way that everyone else interprets these scores, and reacts to them. Because the real concerns should begin when people are attacked for their reviews. If a reviewer gives their honest opinion, and that opinion does not line up with the impression of someone else, the result is that the reviewer must be attacked.
Understanding how review scores are meaningless is useful in all of this discussion. We normally think of review scores as some kind of indicator of actual underlying quality. A proper representation of whether a game is good or bad. And of course, when we play a game we know whether that game is good or bad. So if a reviewer’s ideas radically disagree with our own, the answer must be that the reviewer is just wrong – they are incompetent, or just trying to farm for clicks, or being paid off.
Which is not to say that those things don’t ever happen. But the evidence we need for those claims needs to be on point for any specific case. Back in 2007, GameSpot reviewer Jeff Gerstmann was fired after a publisher, Sony, threatened to pull ad funding from GameSpot in response to some negative reviews, and so the managers of the company terminated Gerstmann. The episode highlighted an underlying problem about the ways in which publishers can control access…but the target of the audience’s ire ended up being “games journalism” as a concept, and not the people controlling the access in the first place.
But usually the evidence of such “corruption” is just vibes. You liked a game, the reviewer gave it a bad score, therefore they must be seeking attention. You hated a game, the reviewer gave it a good score, therefore they must be bribed.
So I wanted to take this opportunity to examine the ways in which review scores are just…bad. Sometimes because of misunderstandings, sometimes because of deliberate manipulation, sometimes because of bad industry practices. There is no one single cause, but the answer generally lies in the same direction: we should definitely get rid of these scores.
Developer Bonuses
Perhaps the best place to start is the problem of attaching bonuses for game developers to review scores. One of the biggest stories of this issue was back in 2012, when it was revealed that Obsidian – the developer behind Fallout: New Vegas – was promised bonuses if the game reached a Metacritic score of 85. The game wound up with a score of 84 (on the PC and the Xbox 360, with an 82 on the PS3 version). Which meant no bonuses.
It’s not entirely clear how prevalent this practice is within the industry, as there are few stories about it. CD Projekt Red had originally planned to tie employee bonuses to Cyberpunk 2077’s critical reception, but changed those plans after the game’s release was…iffy. A decade ago there was a report that Irrational Games, which had developed the Bioshock games, was requiring job applicants to show that they had worked on a game with a Metacritic score of at least 85. But these stories appear to be the focus of discussion. So it’s impossible to tell how often scores are used in these way. It could very well be a rarity.
But even if it’s uncommon, it is still a practice that would be harmful and wrongheaded. The obvious harm is that developers – both on the individual level and a company level – lose out on money after pouring huge amounts of time into the project and often needing to crunch to do so. The wrongheadedness is that no individual critic score is a good indicator of the quality of a game, and trying to solve that by averaging the scores just replaces bad data with other bad data.
Certainly one of the core problems is that a review is easily built off of a first impression that can be marred by a bad release. Take Cyberpunk 2077, for example, though it is by no means the only one. Its sketchy release likely led to significantly lower scores than it might have garnered without all of the bugs. How much lower? It’s hard to say. But whatever that difference, it impacts the average.
And one facet of how these game reviews work is that there’s no editing. A score is given, and the reviewer doesn’t go back to replay the game and decide if the score should be changed. For no small part, because the job of a game reviewer doesn’t generally allow for that – replaying a 20 or 30 hour game, much less a 100 hour game, just isn’t viable with so many other games that need to be played and reviewed. But also because it’s hard to “rescore” a game in that way. A review is generally going to rely on your first impressions of a game. Once those impressions have been done, trying to recapture them is impossible.
But that means in turn that a developer that crunches and ends up pushing out a buggy game gets hammered in the reviews, works hard to fix the game, but by then everyone has lost out on the opportunity to get their bonuses.
Having this system also creates an unfair pressure for reviewers. Imagine if what you wrote about a game determined if someone got paid or not? If you truly and genuinely thought the person was being lazy and incompetent, you might not care about the outcome. But that is almost never the case with studios. So now in writing a review and giving it a score, you have to ask: could my score in some way contribute to these people not getting their bonuses? If you’re not sure, you may be more likely to err on the side of caution and inflate your score just to be safe. But then the score – and the average of those scores – becomes meaningless. Indeed, your inflation will likely lead to an inflation of the required average, which means further inflation, and so on. It is a cycle that just keeps undermining the value of the scores period.
Review Bombing
It’s impossible to talk about review scores without discussing the practice of review bombing. We have generally been focused on scores as part of professional reviews, but Metacritic and other commercial sellers allow people to provide scores on games. I’ll refer to these as “user scores” for clarity.
User scores are usually presented as a rhetorical weapon to prove the superiority of the audience over professional critics. Whenever critics say something is good or bad, and the audience disagrees, this is bandied about as proof of…something. Generally that critics don’t know what they’re talking about.
But user scores are immediately unreliable because of review bombing and the mechanisms behind it.
When I say that, I don’t just mean the actual phenomenon of review bombing where people put together a campaign to go and give a particular game a low rating. The attack on The Last of Us Part 2 based on how fans reacted to one of the major story beats and the game’s inclusion of LGBT+ representation, is a recent example of this.
Review bombing itself is a result of people seeing games as a medium as something that needs to cater to their specific needs. If a game fails to meet those needs in some way – often a fairly arbitrary way – then a person can decide that the game is terrible. The focus on the “politicization” of games, where a certain subsection of gamers perceive any representation of groups other than the standard straight white male as an “intrusion,” causes a significant amount of this. But other incredibly petty or arbitrary views, from a sequel being too different or being too much of the same, or a game not running at 60 frames per second, or a game being exclusive to a particular console. Any of these could conceivably be a reason for someone to go on Metacritic and give that game a 0/10.
Or there could even be “justifiable” reasons for such bombing. For example, as a way of voicing criticism of the practices of a publisher or developer. If you thought Activision firing tens of thousands of people was an unethical move, you might take to Metacritic and post a bad review specifically calling out the publisher’s actions. Because it might be the only way for people to publicly say “this is wrong and shouldn’t be ignored.” But even when those actions are justifiable, they also render the actual user scores – on an individual level and as an average – meaningless.
The point of this section isn’t to say that every attempt to utilize user scores as a tool for activism is bad. As tempting as it would be to make a blanket statement that user scores should be kept “pure,” the actual reality is that it’s often the only real tool available to consumers to effect change. Not that those attempts have much or really any impact. There are better ways to get that change, but that’s not the point.
Instead, what I’m aiming at is that user scores cannot be used as a rhetorical tool to prove anything. User scores as numbers cannot be trusted because there are too many influences on those numbers that exist outside of the game and the player’s experience with the game.
Objective & Subjective Reviews
Perhaps one of the most common arguments trotted out whenever a reviewer gives a game a bad score is to claim that the review is insufficiently “objective.” This can be especially true if the review criticizes elements of a game which are not part of “traditional” reviews – how well does the game capture characters of diverse backgrounds, did the games narrative and themes convey a good message, etc. – then it is “just subjective.”
The obvious and most common response is that all reviews are necessarily subjective. While this response is true, it isn’t helpful. It confuses what our heckler says with what they mean – a problem that is of their own making. In reality, what these hecklers want is a return to a particular style of reviewing that focuses on a very narrow set of variables. How good are the graphics? Does the game have good music? Do the controls make sense? Does it have a story that generally feels compelling and well-written? All of these are strictly speaking subjective questions, but they feel objective because they apply to components of the game that are very clearly there and which we can all see. Compared to the “political” elements of a game which our hecklers feel are imagined or imposed upon the game by the outside.
So really we want to respond to the idea that a review that focused on these very limited aspects would be good. And that a score constructed solely out of these elements would be more valid than one that took a bigger look at the game.
Two obvious issues arise here. The first is that we don’t actually get the desired “objectivity.” What constitutes good graphics? Bad sound design? Good control schemes? These issues become particularly noticeable when we think about the wide range of different possibilities for all of these things. How does one compare retro pixel graphics with cel-shaded graphics with the most up-to-date realistic graphics? The only way to come up with an “objective” solution is a technical one – something like “how many frames per second can you get?” But then the less intense the graphics, the more frames you get. Thus reviews would push games towards what gets good scores – the stuff that checks the right boxes, but might be otherwise dull or derivative. The idea of the “objective” review in this way becomes self-defeating.
The second issue is that any attempt at an “objective” review limits what we’re able to talk about. What is, for instance, an objective review of a game’s narrative? Of its environmental design? Either a review must supply the most rudimentary descriptions of these things which convey no useful information to the reader, or the reviewer needs to make comments on quality – necessarily making it subjective. And without discussions of these things at all, it’s practically impossible to tell whether the game is worth your time or not. Again, the objective review is self-defeating.
But moreover, a review that covers this wider array of topics and digs into the reviewer’s views on topics is more useful and interesting. Whatever you think of someone’s tastes, political views, incentives, and so on, if that person attempted to keep all of that subjectivity out of their discussions it would make for terrible reviews.
Firstly, because it would become impossible to tell if a game is worth your time. The purely technical details cannot help you, so you need some understanding of whether a particular game is enjoyable. And that requires people who have experience with similar games. Reading reviews requires you as the audience to gather information about the reviewer and their tastes. If a review is written to cover only the most “objective” details, then there is no way for that information gathering to occur. If a reviewer recommends or does not recommend a game as part of a review, you have no way of knowing whether that’s a game you want to play.
Secondly, the reviews would just be boring. The technical details not only tell you very little, but they’re not interesting to read about. The reviews – admittedly often negative – that would actually stick in our minds and make the whole enterprise feel worthwhile are the ones that are written from a “subjective” lens. When the reviewer is very clearly in the review. If we were to try and remove the reviewer entirely and insist on only “objective” reviews, we’d basically remove the market for reviews, because no one would ever want to read them.
And of course, surrounding all of this is that the discussion of “subjective” vs. “objective” is a reaction based on the score. Implying that there is a correct number to apply to every given game, and if a reviewer does not give a game that score, then they are “biased” or “subjective.” In contrast those who do hand out those “correct” scores, who must therefore be “objective.” Does the latest Zelda release deserve a 10/10? This is literally a question to be discussed and debated. There is not a correct answer. There is no “objective” review of these games. The only way that a review is designated as objective is because it agrees with what we think, which puts the cart before the horse – we’ve just imposed our own subjective experience on the game and assumed that that is the right answer.
Concluding Remarks
When it comes to video game reviews, it is easy to see them as just words on a page. Or a mere number. But the problem is that these reviews and the scores surrounding them are given a weight that they really shouldn’t have. Which is not to say that they’re worthless. They provide opportunities for critics to talk about issues or share games that might not get as much attention and so on. They are useful for readers who want to learn about new games or make decisions about what they want to buy/play. Getting rid of reviews entirely is definitely not the answer.
But getting rid of review scores is. Because so many of the problems that exist around reviews exist as a function of trying to reduce a game to a number. Those numbers should not be dangled in front of a developer to determine whether or not they get a bonus. Those numbers cannot and should not be treated as rhetorical weapons to attack developers. And reviewers should not be harassed just because they gave a game a 7 rather than a 10. Admittedly, they shouldn’t be attacked for writing a bad review of a game, period. But giving a game a score you don’t agree with is even worse.
Removing scores entirely will not and cannot solve all of the problems. The desire to see the things we enjoy be praised and the things we hate be denigrated is strong. People will still attack reviewers for liking or disliking a game.
But at the very least by getting rid of these numbers, we can stop a bit of it. When people are forced to engage with actual writing – when you are no longer allowed to react to a mere number – then there is some possibility for people to at least understand the reviewer’s position.
Abstracting Games 