Why I Hate: Metacritic

Posted by Dave "shaolinjesus" Corvin
on May 7th, 2009
Filed Under: Articles

"Kids Love It!"

If you check my reviews for Civ Rev or Mighty Final Fight you are more likely to be met with a weird Youtube clip or a Rambo reference than a numeric score. Although at first glance this might be seen as the result of severe head trauma, there is actual reasoning behind this decision to eschew number scores.

I understand that many people like having an easy number that will explain everything, but in the end I think that a number score is the worst part of any review. I am not alone in thinking this way. Adam Sessler of X-Play has railed against the way his scores are interpreted by sites like Metacritic. And there are countless others, just about every reviewer hates giving numeric scores and just about every reviewer is forced to come up with them so that they can be used for sites like Metacritic.

Metacritic is a website that averages the scores from multiple reviews. I think at its best, a site like Metacritic can provide a quick way to evaluate the quality of any given game. At its worst it can skew perceptions. It places far too much emphasis on trying to quantify an ultimately unquantifiable experience. And if you can say that 3 times fast you unlock a portal into another dimension.

Publishers base multimillion dollar decisions on these scores. A higher aggregate score can mean the difference in licensing agreements. Warner Bros. Interactive Entertainment came up with a plan where developers whose games did not achieve at least a 70% average would be subject to higher royalty rates. Sounds like a great idea right? Why should there be sympathy for developers that make crappy games? Its not like 70% is an outrageously high number.

But what about something like “50 Cent: Bulletproof”? The game sold over a million copies and has a Metacritic score right around 50. Hardcore gamers and critics hated it. 50 Cent fans loved it. Does this mean that Bulletproof was a good game?

God no, and I am not trying to argue that popularity equals quality. But still, it is worth noting that critics and the general public often have wildly different opinions.

One of my favorite movies, Armageddon, has a Metacritic score of 42. What do these numbers ultimately mean? What kind of a score would the Sistine Chapel or the Mona Lisa receive? I am pretty sure that there would be at least two reviewers who think they are overrated and would give them scores of 40 or 50 out of 100.

I think a critic’s job is to provide an insightful analysis and evaluation of a given work of art. A numeric score gives no real insight or clarity. What is the difference between a 5.5 and a 6? Does one provide .5 more entertainment? What about the difference between a 3 or a 4 out of 10? At what point do the numbers become meaningless? I think that most reviewers would agree that coming up with a score is the worst part of trying to critique something. Trying to quantify any subjective experience is so far beyond impossible that we are venturing into the realm of ridiculousness. No 2 people have the same kind of measuring scale, because no 2 people are approaching a game in the same way. Just as 2 English professors might give wildly different grades for the same paper, 2 different reviewers might give wildly different scores for the same grade. Metacritic takes the absolute worst parts of criticism and celebrates them.

I might be able to get past my issues with numeric scores if I felt I could trust the sources. Because Metacritic is taking scores from all kinds of sites you end up with scores from sites I have never heard of. Check out these reviews of Resistance 2 Check out those scores down at the bottom and look at the sites they come from. Now I want you to think about this for a second, how much can traffic to your site increase if you throw out a score of 60 for the big holiday game. Is Edge magazine truly trying to say that Resistance 2, one of the most popular and best games on the PS3 really deserves a “D-”?

There is way too much temptation to give a low score just to drive traffic. When I first started writing for WingDamage, one of my friends (jokingly) suggested giving Killzone 2 a score of 5 out of 10 so fanboys would flood the site. Electronic Gaming Monthly gave Assassin’s Creed a 58. How many angry gamers spent hours on the 1up message boards virtually screaming about this low score? How much advertising did they see while they were there? Not that there is anything wrong with advertising revenue (check out our ads from project wonderful) but there is an obvious temptation to give a low score to a high profile game to drive traffic to your site.

In the end, I hate Metacritic because it celebrates the worst part of videogame reviews.

What do you think? What are your thoughts on numeric scores? Is a reference to an unfortunate fryer accident any better or clearer?

Tags: 50 Cent: Bulletproof, assassin's creed, Killzone 2, metacritic, review scores, Why I Hate

This entry was posted on Thursday, May 7th, 2009 at 5:00 am and is filed under Articles. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

8 Responses to “Why I Hate: Metacritic”

May 7th, 2009 at 5:30 am
Rob says:

Interesting write up. There was a similar article in last month’s Games TM in the UK, which agreed with a lot of the above.

I’m inclined to agree, too, with most of what you say. We, as core gamers, will probably not place too much trust in a Metacritic score, but it’s a decent source for the uninformed masses. If you have a look at the top of the 360 chart, you see GTA IV, Bioshock, Halo 3, CoD4, Rock Band – games I think we can all (in the most past) agree belong up there. So for those unwilling to read up a bit more on why and how the game’s successes and failures impact on your time spent with it, it’s a decent source.

However, when a publisher uses a Metacritic score to define when and how bonuses and royalties are paid, I think it can be a bit unfair. Most of the reviews used for a Metacritic score are from core gamer’s sites, so kid’s games tend to score low, as will movie and TV tie in games. However, many of these can sell extremely well, so is it fair to deny bonuses paid to the developer on a game which sold a couple of million because the Metacritic score is 50%? The people buying these games aren’t too worried about the quality as much as we are – they don’t care about going on endless fetch quests or the same mission reskinned over and over. If the game sells, the developer should get their royalties.

Also, regarding your point on low scores generating high traffic, I think with Edge in particular you may be off the mark. The reviews on Edge online come direct from the print mag, so when they’re writing that the ads are already in the magazine – they don’t need to generate extra traffic to get more hits. Also, their scale runs from 1 – 10, so a 5 is an average game – Resistance 2 received an above average score. Games TM do the same. These two are always included in Metacritic scores as they are well regarded magazines, so they will always throw the score out slightly due to the fact that most other publications and websites seem to run on a 60% – 100% scale. Edge’s score for Killzone 2 was a 7, and this caused a huge stink in many forums, with people branding their review and score (and hence their opinion of the game) as ‘wrong’. However, this is a score given to a good game – not exceptional, but good. Which I agree with when it comes to Killzone 2.

Stuff like this can make a huge difference when it comes to Metacritic, and as so much value is placed on this by publishers it can affect people’s jobs and livelihoods. Projects have been dropped due to a low Metacritic score on a previous game by that same developer, with no regard to the potential of the game in question.

So I think it’s a good site as a general guide for the public who have no other knowledge of the games they’re looking into, but publishers place too much emphasis on these scores when there can often be little correlation between a quality game and high sales (see Beyond Good & Evil, Oddworld: Stranger’s Wrath, Zak & Wiki, Psychonauts on one end, 50 Cent: Bulletproof, every FIFA before ’08 and Driv3r on the other end).
May 7th, 2009 at 6:39 am
Wesley says:

Here, here!

I hope 50 Cent doesn’t see what you wrote though.
May 7th, 2009 at 7:45 am
Scott Reeser says:

I wonder how many “uninformed” gamers get input from Metacritic. It’s relevance must be seen within certain audiences, though definitely not amongst those of us who don’t care about scores. If only it really were vendor-neutral and could produce relative scores for each of its contributors.

Either way, scores will always be around, so will those who don’t ascribe to them. I’d much rather see a system like Netflix’s scoring system: ratings based on what I’ve played and overall average ratings from all members.

It’ll happen once Netflix and Gamefly merge with OnLive
May 7th, 2009 at 9:09 am
Jesse "Main Finger" Gregory says:

I agree that numerical scores are pretty meaningless, I think every writer for Wind Damage does. However, what I use metacritic for is very quickly seeing review highlights and being able to click the “read full review” button for lots of different sites so I can read what critics liked and disliked about the game. I usually only bother reading them if it is a site Ive actually heard of. So ya, I’m not using it quite as the site would want me to, but I find it to be a convenient portal into finding out what people liked and disliked. To me, metacritic is no more than a links page.
May 8th, 2009 at 9:46 pm
Jacob says:

You lied to me…there was no portal dave.
July 8th, 2009 at 11:49 am
Kddr10 says:

Author’s credibility goes to zero when he says his favorite movie is Armageddon.
July 8th, 2009 at 12:01 pm
Jesse "Main Finger" Gregory says:

I think you’re kind of missing the point of the article. I think the key points are:

1) Review scores are trying to quantify an unquantifiable experience and

2) A review should just be an insightful analysis of the subject matter and not a score

Which things he likes doesn’t really affect the points brought out in the article.
July 8th, 2009 at 1:42 pm
shaolinjesus says:

Armageddon is NOT my favorite movie. But I thought it was a good example. It was just a very enjoyable explosion filled easy to watch spectacle. So many movies that critics go ape over are barely watchable one time. I prefer over the top melodrama and explosions to boredom any day. If I was to write a review of Armegeddon I would stress that it is a kind of ridiculous and over the top and not meant for everyone.

What I was trying to say in the article is that ultimately a score without any context is meaningless. My 9 out of 10 is not your 9 out of 10. For example I like my hot sauce spicy as all creation, so my favorite hot sauce is not going to be enjoyed by everyone. Does that mean my opinion is meaningless? No, instead it means you have to understand why I enjoyed hot sauce A over hot sauce B. Metacritic ignores the why.