Metacritic Matters: How Review Scores Hurt Video Games
Bugs in Fallout: New Vegas might have eaten your save file. Maybe they took away a few hours of progress, or forced you to reset a couple of quests. Maybe game-crashing bugs pissed you off to the point where you wished you could get your $60 back. But they probably didn’t cost you a million dollars.
Why Are Game Developer Bonuses Based On Review Scores?
Last night, Obsidian's Chris Avellone tweeted an interesting detail about his roleplaying game … Read…
Perhaps you've heard the story: publisher Bethesda was due to give developer Obsidian a bonus if their post-apocalyptic RPG averaged an 85 on Metacritic, the review aggregation site. It got an 84 on PC and Xbox 360, and an 82 on PS3.
“If only it was a stable product and didn't ship with so many bugs, I would've given New Vegas a higher score,” wrote a reviewer for the website 1up, which gave New Vegas a B, or 75 on Metacritic's scale.
“It's disappointing to see such an otherwise brilliant and polished game suffer from years-old bugs, and unfortunately our review score for the game has to reflect that,” said The Escapist's review, which gave the game an 80.
If New Vegas had hit an 85, Obsidian would have gotten their bonus. And according to one person familiar with the situation who asked not to be named while speaking to Kotaku, that bonus was worth $1 million. For a team of 70 or so, that averages out to around $14,000 a person. Enough for a cheap car. Maybe a few mortgage payments.
Those sure were some costly bugs.
This is not an anomaly: for years now, video game publishers have been using Metacritic as a tool to negotiate with developers. And for years now, observers have been criticizing the practice. But it still happens. Over the past few months, I’ve talked to some 20 developers, publishers, and critics about Metacritic’s influences, and I’ve found that the system is broken in quite a few ways.
There is something inherently wrong with the way publishers use Metacritic. And something needs to change.
Why Metacritic Matters
Hop into a debate with some video game fans on your favorite message board, and there’s one subject that will always come up: review scores. Which game scored the highest? Which scored the lowest? Which are the best review websites? Which are the worst?
Inevitably, at some point, someone will jump into the fray and say something like “lol review scores mean nothing anyway.” To some people, maybe that’s true. But to the people who make and sell video games, review scores are more important than many casual fans realize. Mostly because of Metacritic.
For the uninitiated: Metacritic is an aggregation website that rounds up review scores for all sorts of media, including video games. The people who run Metacritic take those scores, convert them to a 100-point scale, average them out using a mysterious weighting formula (more on that later), and spit out a number that they call a Metascore, meant to grade the quality of that game. The Metascore for BioShock Infinite, for example, is currently an 94. Aliens: Colonial Marines? 48.
To people who work in gaming, these Metascores can mean a lot. Say you’re a developer who needs money. You’ve got some ideas to pitch to publishers. You take some meetings. They’re going to ask: just how good have your games been?
“Typically, when you go into pitch meetings and whatnot, publishers are going to want to know your track record as far as Metacritic,” said Kim Swift, a game designer best known for helping create games like Portal and Quantum Conundrum. “As a company, what is your Metacritic average? As an individual, what is your Metacritic average?”
Swift works for Airtight Games, an independent studio that is tied to no publishers. Their Metacritic history: Dark Void, which has a 59 on Metacritic, and last year's Quantum Conundrum, which sits at 77.
In order to survive, studios like Airtight have to negotiate deals with big companies like Capcom and Square Enix. Often that means talking about Metacritic. Sometimes that means wearing their history of Metacritic scores like a scarlet letter.
This is common. An employee of a well-known game studio told me about a recent pitch meeting with a publisher, during which the publisher brought up the studio's last two Metacritic scores, which were both average. The studio employee asked that I not name the parties involved, but claimed the publisher used the Metascores as leverage against the studio, first to negotiate for less favorable terms, and then to turn down the pitch entirely.
Often, developer bonuses or royalties are tied to game review scores. Fallout: New Vegas is one high-profile example, but it happens fairly often.
“It’s pretty common in the industry these days, actually,” Swift told me. “When you're negotiating with the publisher for a contract, you build in bonuses for the team based on Metacritic score. So if you get above a 90, then you get X amount for a bonus. If you get below that, you don't get anything at all or get a smaller amount.”
In other words, a developer’s priority is sometimes not just to make a good game, but to make a game that they think will resonate with reviewers.
“When you're working on a game, part of what you want to do is have a high score,” said Swift. She said she’d never seen a developer change part of a video game just for the sake of raising scores, but the influence is undoubtedly there.
“It's usually some other thing like, ‘Hey, we could use another couple hours on this game because people perceive a longer game to be a higher value,’” Swift said. “It's never directly pointing back to, ‘This is gonna improve our score by X number of points.’”
Matt Burns, a longtime game designer who worked for a number of big shooter companies and now makes indies with his company Shadegrown Games, wrote about his personal experiences with Metacritic back in 2008. Burns said he watched firsthand as a development studio worked as hard as possible to make a game that would snag high review scores.
“Armed with the knowledge that higher review scores meant more money for them, game producers were thus encouraged to identify the elements that reviewers seemed to most notice and most like–detailed graphics, scripted set piece battles, 'robust' online multiplayer, 'player choice,' and more, more of everything,” Burns wrote.
“Like a food company performing a taste test to find out that people basically like the saltiest, greasiest variation of anything and adjusting its product lineup accordingly, the big publishers struggled to stuff as much of those key elements as possible into every game they funded. Multiplayer modes were suddenly tacked on late in development. More missions and weapons were added to bulk up their offering–to be created by outsource partners. Level-based games suddenly turned into open-world games.
“Before you cry in despair, keep in mind that all these people wanted in the end was the best game possible–or, more precisely, the best-reviewed game possible.”
And then there's this wry joke by Warren Spector, talking about the words that influenced his career during a talk at the DICE conference earlier this year. Powerful words. Legacy. Mentor. And...
While chatting with Obsidian head Feargus Urquhart for the profile I wrote last December, I asked him about what had happened with Fallout: New Vegas. For legal reasons, he couldn’t get into the specifics.
“I can’t comment on contracts directly,” he said. “But what I can say is that in general, publishers like to have Metacritic scores as an aspect of contracts. As a developer, that's challenging for a number of reasons. The first is that we have no control over that, though we do have the responsibility to go make a brilliant game that can hopefully score an 80 or an 85 or a 90 or something like that.”
According to Metacritic’s rating scale, any game above a 75 is considered “good,” but realistically, according to multiple developers I spoke with, publishers expect scores of 85 or higher. Sometimes, Urquhart told me, the demands can get unreasonable.
“A lot of times when we're talking to publishers–and this is no specific publisher–but there are conversations I’ve had in which the royalty that we could get was based upon getting a 95,” he said. “I’ve had this conversation with a publisher, and I explained to them, I said, ‘Okay, there are six games in the past five years who have averaged a 95, and all of those have a budget of at least three times what you’re offering me.’ They were like, ‘Well, we just don’t think we should do it if you don't hit a 95.’”
That’s the developer’s perspective. Now let’s look at this from the other side. Say you’re a publisher. You’re about to sign a seven- or eight-figure deal with a development studio, and you want to make sure they’re not going to hand you a clunker. Why not use Metacritic as a security blanket in order to minimize risks and ensure you get yourself a great game?
Here’s some very reasonable rationalization from a person who worked at a major publisher and asked not to be named:
“Let’s say that [a publisher] wanted to pay $1 million up front (through milestone payments over the course of development), but the developer wanted $1.2 million. If they wouldn’t budge, sometimes we would offer to make up the difference in a bonus, paid out only if the game hit a certain Metacritic.
“That conversation could happen during development too. Maybe a developer wanted more time and money in the middle of the production, to make a better game. So the counter was, ‘If you’re so sure it will make the game better, we’re gonna tie the additional funds to the Metacritic score.’ It was a way to minimize risk.”
But a different person who once worked for major publishers says that Metacritic scores are just an excuse publishers use in order to deprive developers of the bonuses they deserve.
“Well, generally the whole Metacritic emphasis originated from publishers wanting to dodge royalties,” that source said. “So even if a game sold well, they could withhold payment based off review scores... The big thing about Metacritic is that it's always camouflaged as a drive for quality but the intent is nothing of the sort.”
Multiple developers I spoke to echoed similar thoughts, although nobody could share hard evidence to back up this theory. I reached out to a number of major publishers including Activision, EA, and Bethesda, but none agreed to comment for this story.
Marc Doyle, the former lawyer who co-founded Metacritic in 2001 and keeps it running every day, told me during a phone conversation last week that he feels no responsibility for what video game publishers or developers do with his website.
“Metacritic has absolutely nothing to do with how the industry uses our numbers,” he said. “Metacritic has always been about educating the gamer. We’re using product reviews as a tool to help them make the most of their time and money.”
But gamers aren’t the only ones who use Metascores. Not by a long shot. Even the massive Japanese publisher Square Enix recently cited Metacritic as one of the factors they used to predict sales for their games.
"Let's talk about Sleeping Dogs: we were looking at selling roughly 2~2.5 million units in the EUR/ NA market based on its game content, genre and Metacritic scores," former Square Enix president Yoichi Wada wrote in a recent financial briefing. "In the same way, game quality and Metacritic scores led us to believe that Hitman had potential to sell 4.5~5 million units, and 5~6 million units for Tomb Raider in EUR/ NA and Japanese markets combined."
"Review scores are a part of our industry and it's something we pay attention to as developers,” said Swift. And they lead to trends. “Review scores of this year are gonna drastically affect what’s gonna be seen next year,” she said.
Even big retailers like Walmart and Target ask publishers for Metacritic predictions when deciding whether or not to feature certain games.
“One of the criteria [retailers] have is, ‘What’s the review score gonna be?’” said Tim Pivnicny, vice president of sales and marketing at Atlus USA. “That comes up a lot... They’re concerned if it’s going to be a good game.”
Metacritic has a significant influence on the way games are produced today. That's a problem.
