December 09, 2005
Sell Out
by peterbIn addition to the writing I do for this weblog, and my day job, I occasionally get the opportunity to work on other projects. One of them has bourne fruit, and I am now a paid and published game reviewer. My review of Civliization IV appears in the latest issue of played.todeath.com. Please feel free to give it a read (PDF format).
Hopefully, you'll read it in time to take Civilization IV off of your holiday gift lists.
Which brings me, once again, to the topic of game reviews, game reviewers, and why so many of them are so absolutely terrible. I think that I'm in a position to explain this clearly now.
Some of you may recall that I addressed this issue earlier in the year, where I pointed out that Game Girl Advance was engaging in some unjustified apologetics for unethical behavior on the part of a game magazine. The howls from some paid game reviewers shook the rafters. Their defenses were many, but most of them centered around either recharacterizing my argument as claiming that they were all being paid off in cash in a parking garage somewhere, or by simply asserting that, since I was not a professional writer, I simply didn't understand the time pressures and tradeoffs involved in writing a quality review.
So now that I'm officially a professional writer — although, thank God, not one who has to write for a living — I still think these writers are deluded. Video game reviews, on the whole, are terribly written. Let's think about why.
It helps, in this context, to look at counterexamples from other media. Film is the example most people choose. But today, I'm going to talk about automobile reviews.
Your average, run-of-the-mill car review, found in your nondescript, middle-American newspaper, is of more use to the consumer than your average, run-of-the-mill videogame review. In part, this is because there is more at stake. Cars are durable goods that people evaluate carefully before purchasing, and if you engage in puffery people will tend to notice very quickly, and complain about it.
It also stems from a vehicle's nature as a functional item. This is not to say that cars don't entertain — as anyone who has bought one can tell you, aesthetics matter — but that every car can be described in terms of how well it does any number of tasks. How does the gearbox feel? Is it responsive when you step on the throttle? How much body roll is there going around turns? How much space is there? Are the seats comfortable?
Furthermore, everyone is very aware that different vehicles cost differing amounts of money. The key question we look to car reviewers to answer is not "Is this car good," but "Is this car a good value." A review that simply says "The BMW M6 is awesome!" without any reference to other vehicles you can get at the same price point is an utterly useless review.
Let's look at the BBC 2 TV show Top Gear. This show is incredibly addictive, and watched and enjoyed by a lot of my friends, many of whom don't even care about cars. The show's hosts are opinionated, to the point of being crushingly rude about vehicles that they don't like. But even in their most cruel and cutting reviews, they find time to mention that the car gets good fuel economy for its class, or that it has a lot of cargo space. Even in their most fawning, adulatory paeans to £400,000 cars, they find the time to mention that the fit and finish on the bodywork is garbage, or that the knobs on the radio are silly, or that the trip computer is hateful.
Game reviews — generally — are not written like this. Sure, they tend to divide the game into sections, and then purport to analyze them, but they don't, in my view, answer the fundamental questions. To pick just one example, none of the "best game EVAR" reviews of Civilization IV that you can find on Metacritic choose to mention that the game is amazingly crashy, or that it's user interface is worse than its predecessor. One excuse you find people making is that games are primarily entertainment content. That's sort of true, but sort of not. A game is not just like a movie. A game is like a movie, but it is also like a car. A game is entertainment, but it's also, in a very real and practical sense, a functional device.
It's not simply a matter of reviewers disagreeing with my opinion. More importantly, in none of the reviews I've read on Civ IV do I see a discussion of value. When discussing automobiles, it's easy to compare value because cars are sold at so many price points. Value is largely a matter of comparing the vehicle to other cars in its class, and to other cars at its price point. In games, the hidden variable is time. How much time will it take you to play the game? How many hours of enjoyment will the game offer, versus hours of mindless drudgery? How many hours will you spend on the tech support line, trying to find out why the sound stutters?
I'll reduce it to a simple rule of thumb. I don't care what game you're reviewing. I don't care how perfect (or flawed) it is. I don't care how pressed for time you are. If you can't find one bad thing to say about a game (or, in the case of a negative review, one good thing), then you are not doing your job.
So why don't most of the reviews of Civ IV condescend to mention that the game, as delivered in the box, is buggier than a prison cafeteria? I have a theory.
Game reviewers, like everyone else, are subject to the phenomenon of the latent object of desire. Most game reviewers are not software engineers. They don't see the meat that goes in to the sausage. As such, they are unreasonably optimistic about game publishers, particulary PC game publishers, ability to fix problems in patches. It's not just game reviewers. Everyone thinks that software engineering is easy. The other day I was visiting Slashdot and read an article whose thesis was, and I am not kidding, "Why doesn't Microsoft just fix the bugs?" I nearly choked on the sandwich I was eating, and several quarts of my own bile, out of sympathy for the person in Redmond who was going to read that and instantly have a brain aneurysm.
Game reviewers, I believe, approach games with this optimistic attitude, and they don't want to be a downer, so if there's a bug and if the game is by a reputable publisher, they just think "Hey, no big deal. They'll just fix that in the patch. Since the article I'm writing won't be published for another month, it might be fixed by the time people read it. Sure, the game doesn't actually work, but I'm sure that's just a simple matter of programming. I was having a lot of fun before the game crashed. I'll just think happy thoughts and say the game is a surefire candidate for Game of the Year."
I can't do that. I know you, Firaxis. I've been in your shoes. I was in the room with you when the architect said "Hey, I've got this great idea. Why don't we make all the objects in the game fully 3D?" I was reading the memo the junior engineer circulated that compared various 3D libraries, and recommended that you take the one that was cheapest because it had newer features, and besides, none of your customers would be using an old videocard anyway. I attended the presentation from the consultant who informed you that as long as your save game file format was XML, implementing it would take 3 days and it would be impossible for there to be any bugs. I was there. You dropped features, decided not to fix bugs, and skimped on Q/A because you had to make the Holiday season. I feel for you. I'm really really sorry you had to go through it, and I know that you don't feel any better about it than I do.
But there's one thing I can't do for you, Firaxis. I can't look at the pile of bugs you delivered and call it a good game. In the end, the choice about when to ship, and what to ship, was yours. And you blew it. So when people ask me what I think about the game, I have to be honest with them. I have to tell them the truth.
That seems, sadly, to make me an unusual game reviewer. If that also makes me a poor one, then so be it.
I approached the Civilization IV review, very deliberately, as an attempt to write a review about a game as if I was reviewing a car. I tried to write it as if I were one of the presenters on Top Gear.
I'm sure that my writing isn't as colorful as Jeremy Clarkson's. But I hope, in the end, that it is still fun to read, and that you find the review useful. If you have questions or comments on the played.todeath.com review, feel free to comment on it here, or send a letter to the editor.
Posted by peterb at December 9, 2005 04:34 PM | Bookmark ThisHmmm. Civ 4 is my game of the year. And that's the truth.
Not everyone had bug issues. I didn't, and I'm running it on a three year old laptop. There was a mislabeled disk and the annoying DirectX install prompt at the beginning, but none of that has detracted from the many hours I've spent on this game.
I think that most paid game reviewers (btw, congrats on joining the humble underpaid ranks) do tell the truth. I think that credibility is more important to reviewers than anything. I know I take mine very seriously.
But good reviews for games you have problems with are mysterious. I've commented on the surprisingly good reviews that Paradox's Diplomacy has gotten, but note that they are mostly from the "volunteer" websites. PCGamer is a notable exception.
Posted by Troy Goodfellow at December 9, 2005 07:04 PMI hate movies. On the rare occasion I get roped into going to the movies, I come up with a short list of films I'm willing to see. This list is usually only zero to one item in length. How do I decide what current movie I would be willing to waste $8 seeing? Simple: Mr. Cranky Rates the Movies. I go the CMU local movie page to see what's playing nearby, hop over to Clamen's MRQE, search for the movie titles, search the results page for the word "cranky", and read his reviews. Anything with more than 3 bombs doesn't even get read, unless the bigfont teaser blurb is funny or intriguing.
I like to think of you as Mr. Cranky of Video Games.
Posted by rlink at December 9, 2005 07:12 PMTroy,
There are a lot of good things one can say about Civ IV, and I try to hit the high points in the played.todeath review. I hope my rant here is not seen as the whole story.
In some ways, this is once again the PC vs. console debate writ small: apart from true outlying situations, if some guy plays a game on an Xbox, I'm reasonably confident that the _overall user experience_, albeit not the details of whether or not I like a game, will be the same.
But if we have different videocards on our PC, we might very well be effectively playing completely different games.
Posted by peterb at December 9, 2005 07:13 PMI read the whole review, so my reply had the whole thing in mind. (BTW, the last thing I need is another online magazine to read, but now I think I have to subscribe to this one...).
Your comment on the effects of hardware on PC game reviews is something that has been debated for years. I'm lucky in that I have multiple machines in my house, so if the game doesn't work on my primary computer I can test if the issue is the game or me.
I'm not sure that I can so far as to say that a videocard means an entirely different experience, but Age of Empires III certainly looks prettier on my wife's computer than it does on mine. I still have the same problems with it as a game.
Starforce has become a big divider in the review community. Some people experience so many problems that they refuse to review Starforce games on their home machines. I've never had an issue. Lots of readers wonder why we don't warn them about Starforce protection.
I wonder if Next Gen console reviewers will be required to have HDTV?
Posted by Troy Goodfellow at December 9, 2005 07:26 PMI hadn't considered the HDTV/non-HDTV divide, but you're right -- I suspect it will be an issue. Certainly, at the least, it will require more QA on each game.
Also, I really enjoy your weblog; I'm adding it to the blogroll. Thanks for commenting!
Posted by peterb at December 9, 2005 08:07 PMI'm not a Civ fanatic. I like the series, but I know that I'm no expert. I play on wimpy skill levels. I have no real idea how to optimize my tech progression. To date, I've still only finished a couple games of Civ4. As such, I feel a bit odd commenting on the game at all.
Still, I don't think your review tells the whole story.
The games graphical presentation IS unacceptably buggy, yes. It plain doesn't work on many machines, including those of several friends of mine. The patch appears to have at least partially fixed this, but the game should never have been released in that state.
However, I've had no problems with the graphics, either before or after the patch. It Just Plain Works on my machine. The sound is fine. The music is fine. It hasn't crashed on me yet.
Okay, yes, that's just me. But it makes me dubious of a claim that the game as a whole is the unending bugfest you describe. No, it doesn't work right on many computers. Yes, it should. No, you shouldn't buy it from any place without a good return policy. Yes, Firaxis should be taken to task for this. These are, however, compatibility problems, not fundamental gameplay bugs.
I just don't see the UI as being as bad as you describe, either.
You list two specific examples: Modal build popups and the Civilopdia's list-o-icons. Both are flaws, yes. Neither, however, is all THAT bad. Don't know what to build? Pick a random item and come back to the city later. Don't know what icon represents the you want? Click one at random, and then pick the unit from the nicely alphabetized list that is now available on the side of the window.
Both of these problems are examples of poor UI design, but neither one will slow you down for more than a couple seconds.
My real problem with your review, however, is not your conclusion--I disagree with it, but it's a fair one to come to. It's what's NOT in the review.
There's hardly a mention of the GAME in there. You talk about graphics. You talk about some UI nits. You have a paragraph on the music, three paragraphs on the history of the Civilization series. And then there's a brief paragraph that describes the addition of religion, and one bare sentence mentioning that a few large cities are more useful than many small ones. At no point do you cover how these changes impact the gameplay or feel of the game.
Perhaps this is fair: Firaxis certainly focused on graphics more than they should have, and released a game with major system compatibility problems as a result. In return, they get a review that focuses on the graphics and compatibility issues. But there's still a game in there--I know, I've played it--and it's a game that many people seem to be enjoying.
I don't read professional game reviews much any more. In part because, as you say, they have appallingly bad journalistic standards. But also because they've almost universally turned into a bland recitation of graphics and features with no actual REVIEW. Imagine a movie review that talked about the quality of CGI, provided a capsule summary of the plot, and ended. Okay, there are movie reviews like that out there--but there's a reason I read Ebert's reviews and not those.
Posted by Damien Neil at December 9, 2005 08:34 PMDamien,
It's definitely true that my review had a specific reader in mind: a reader who had played one of the earlier Civilization games. The implicit question I was trying to answer is: "OK, so I own Civ I, Civ II, and Civ III. Should I bother buying the new one?" A detailed description of all of the game mechanics would have been boring to that reader, and so I left it out.
That being said, it certainly makes the review unfair to anyone who has never played any of the games. Thanks for the constructive criticism. You raise a good point.
peterb
Posted by peterb at December 9, 2005 09:00 PMI thought the review quite accurate and enjoyable to read, being squarely in the "Has played all the incarnations of Civ since I was a wee lass" camp.
I also enjoy your "non-professional" writing here (unless it's about odd Italian beverages - we all have some inexplicable idiosyncracy or another that keeps us from talking about games all the time). So it feels rather rude to pipe up harping over a nit I doubt you have much control over - yet much space is dedicated on this blog to keenly pointing out just such silly nits and the little moments of reeling incomprehension and annoyance they cause that I just can't resist. I tried, I swear.
Why is it I have to download and search through an
11 Mb PDF with fades and fonts that look near-unreadable unless zoomed in to the nanometer level? Sure, I am, like many of us, clearly an
inveterate masochist for reading game reviews and game related things. I take the un-navigable sites, the howling flash ads, the often reprehensible quality of the writing, the outright obfuscations and lies. But a massive, busy, un-linkable, _page numbered_ (!?) PDF file that seems to not even have the excuse of being intended for print distribution? Why torture us with this perfect storm of all the shortcomings of printed and online text? Is this the price that must be paid for the well-meant attempt at better, fairer writing? Total terribleness must remain constant according to some immutable gaming press law, it seems.
-pvg
Posted by pvg at December 9, 2005 11:46 PMpeterb,
On the subject of game crashes, I was shocked that none of the big reviews of "Champions of Norrath" for the PlayStation 2 mentioned the lock-up/hang bugs that were discussed at great length in many online forums after the game was released. I asked those reviewers to comment, and while some saw the bug/hang others didn't. Those who did chose not to mention it in the review:
http://curmudgeongamer.com/article.php?story=20040323191154516
In retrospect, I'm less charitable toward the reviewers and probably should have given them a harder time.
Perhaps for comparison, I later gave Champions of Norrath a good thrashing. Here 'tis:
http://curmudgeongamer.com/article.php?story=20040411185227728
Congrats on your review and take care.
jvm
Posted by jvm at December 10, 2005 07:02 AMPVG, I totally hear your complaint, though at least once I zoomed into title page I could click on "Civilization 4" to go straight to the article. Though not using the PDF table of contents was irritating.
Unfortunately, this seems to be a growing problem. Allegedly "The Escapist" also has many good gaming articles, but I'll never read them. They're overly busy images and won't let me zoom in or adjust the font size, which basically means they're illegible for me. I hate when the graphics designers take over.
(This isn't all criticism; I'd like to personally thank PeterB for talking me out of Civ4. )
Posted by Adam Rixey at December 10, 2005 08:56 AMThe Escapist has a "print friendly" version of each of their HTML articles that strips all the page layout bullshit^H^H^H^H^H^H^H^H design that they do so you can actually read the text.
Still, even navigating to an article that you want to read is enough to make you claw your eyes out one by one with an ice pick.
I had not heard of The Escapist, thanks. Design-wise it fit perfectly in my Saturday morning hour of hatred. The bit that made my bile-sac swell painfully was the "Technical FAQ" section. It starts out boldly and unapologetically:
"The Escapist is based on the idea of a quality magazine, that of a fixed and artistically inspired layout with solid and thought provoking content. One of the earliest decisions in creating The Escapist was that we did not want to create another website. While PDFs provide all the magazine-like qualities we want [...]"
All of the magazine-like qualities? Not another website? One has to wonder what business people who neither want to design websites nor take magazines to the bathroom have, well, designing websites.
The rest of the FAQ consists of a few questions through which one can easily hear the muffled anguished wails of their readers. All of this is dismissed with glib talk of validated XHTML 1.0 Transitional and CSS, resolutions and a touching plea to try the PDF instead. Essentially what it boils down to is "We're fucking with you and we know it. Tough".
Perhaps it was the theme of the current issue but I found much of the writing a little precious and overwrought. It could be just me but I prefer my gaming readings to be a diversion from, not extension of that 7000 word Lewis Lapham essay I just force-fed myself.
I'd check The Escapist out again but I'm blind now.
-pvg
Posted by pvg at December 10, 2005 11:59 AM"Since the article I'm writing won't be published for another month, it might be fixed by the time people read it."
This was the one thing that stuck out of this blog. That one sentence seems to be the problem with what you're talking about.
You compared game reviews to auto mobile reviews and although you make the valid point that one is supirior to the other, one of the mai nreasons the auto mobile review was supirior was because they would review a finished product.
You wouldn't review a car without airbags thinking "By the time this is published the airbags are bound to be installed".
The same thing goes with games really. As far as I can tell, major review sites give a video game review a few days after it has been released. Hopefully this is an indication that they have the same copy the ordinary consumer will have.
Though if you were to put game magazines into context here then you point is valid.
There's one thing that bugs me when people talk about reviews and that is when they say that reviews on a certain website or in a certain magazine are more "accurate", which someone has already mentioned above.
"Accuracy" is pretty much impossible in the context of a review, which is almost by definition an expression of someone's opinion on something. A review may be "fair", "balanced", "impartial", "well-written", "entertaining", "thought-provoking", "insightful", and may even "exactly match my own opinion". (Your Civ IV review, by the way, seemed to be several of the above). But none of that has anything to do with accuracy.
It's why review scores always annoy me, as well. It's fair enough out of 5 stars - 5 means buy it, 1 means don't, somewhere in between means think about it. Simple. Out of 10 is starting to get overcomplex, in my opinion. Percentages? Say what? Our game just got 89% on one website. What, exactly, stopped it getting 90%? Or 88% for that matter? Which minor flaw in the game cost as that 1% that would have made us a 90%+ game (everyone wants to be 90%+ after all)? I mean... just what is a percentage trying to express?
Posted by at December 10, 2005 02:04 PM> just what is a percentage trying to express?
It's an expression of how well a game's marketing team did their job.
Unfortunately, I'm not really kidding.
Posted by peterb at December 10, 2005 04:15 PMGag...a pdf. I hate those for the same reason I disable flash by default in my browser, it slows a web page down (especially when loading).
For reviews I generally just grab the recommendations from forums like qt3 (which is also how I found out about obscure titles like Space Rangers 2).
As for the tiny font problem at the escapist it's not really a problem unless you have your monitor set to some really high resolution (I only use 1280x1024 on my 21"). Some newer web browsers even support zooming in on web pages(like Avant which I use).
Posted by garyh at December 12, 2005 03:05 PMCiv 4 is my game of the year too.
I've encountered one obvious bug (stuttering wonder movies), two performance issues (performance degrades in the late game when graphics settings are pumped up, noticeable annoying pause at end of AI turn), and zero crash bugs. Zero.
I've played the hell out of it. Fifty games at least, a few from beginning to Space Race or even turn limit without exiting the program. It's not a very buggy game, it certainly isn't a pile of anything except great new game mechanics, and while the interface has minor issues (your noting of them in your super dry, checklist-style review is the only part with any critical acumen in it), the game portion of it is near-perfect.
Take a look at Apolyton or CivFanatics, the forums are not filled with complaints about bugginess. Yes, the game is crashing for some people. Welcome to PC gaming, Mr. B. However, most Civ fans are having a great time with this game. This is nowhere near anything like when, say, Civ III: Play the World was released. That was a disaster. That had big bugs.
This, however, is a superlative game. Telling people not to buy it in your first "commercial" review is a hell of résumé builder. The only more off-the-mark review I've seen is the IGN review of Master of Orion 3. And that's including every review of Black & White ever.
Posted by Matthew Gallant at December 13, 2005 12:33 PMWhen I saw this post, I was wondering how long it'd take for a snotty "you suck for not liking my FAVORITEST GAME EVAR" fanboy comment to show up.
Four days is a lot longer than I'd expected.
Matthew Gallant writes:
> Yes, the game is crashing for some people. Welcome to PC gaming, Mr. B.
You, sir, are the reason why PC gaming is dying. You are willing to accept crashy, buggy software as the natural state of things. You are willing to accept that the game you just bought will need at least one patch downloaded and applied before you can even play it. And worse, you seem to blame the spotty QA process of Firaxis (and the industry in general) on Peter. How can it possibly be his fault?
Also, on a completely personal note, you're a condescending ass. I will bet dollars to donuts that Peter has been playing games on his PC for longer than you've been eating solid food. Welcome to PC gaming, indeed.
Posted by jeff at December 14, 2005 05:05 PMDamn straight. I've been playing PC games for, what, 18 years now? And I for one am SICK AND TIRED of crashes, patches, hangs, bugs and Steam.
Windows crashes much less than it used to (though we all know it's still not perfect), so it's not an unsolvable problem, and the market size, different hardware configuarations and number of bugs Windows has to deal with is WAY more than every PC game put together, blowing the usual excuses right out of the water. It's simply not good enough, it's preventing me from playing games I'm sure I would love (like Half Life 2 and Rome Total War), and it's killing PC gaming altogether.
Gah.
Posted by Andy P at December 15, 2005 06:28 AMCiv IV hasn't crashed on me once. The original Civ crashed more often.
None of the Civ bugs rise to the level of making the game unplayable - this isn't Darklands people. It's not even Harpoon.
I don't begrudge anybody not liking the game - I don't like Diablo all that much. But let's not operate on the assumption that Civ 4 is a bug ridden mess.
Posted by Troy Goodfellow at December 15, 2005 12:54 PMI've played a handful of games on Civ4 with the difficulty settings as high as Warlord. (I'm a new to the Civ world). So far the only serious bug has been stuttering in the audio when I build a World Wonder. I've also experienced a couple minor bugs with the hover text in specific cases. I've not had a single crash so far in probably 30 hours worth of game time. Is the main difference between people experiencing crashes the difficulty level they play under?
One factor that seems to be related is the size of the world. In particular, more scrolling over dark regions of the map seems to result in more slowdown, and eventually crashing. See the bug forum on civfanatics.com for more specific information.
Posted by peterb at December 19, 2005 05:58 PMpeterb, thank you for shooting the honest truth about Civ4. I've been playing Civ since Civ2, and I've been drooling over Civ4 for months. Finally I get it for Christmas, and have I finished ONE game yet? Endless load times, constant crashes, freezing every few turns (and this is on low graphics setting). I got the game, discovered that it didn't work, got the patch, discovered that it still didn't work, tried to play five turns at a time, and now I'm just about ready to just pop in Civ3, or better yet, just go play Alpha Centauri.
Don't get me wrong, what I can play of the game is excellent. I love every minute I play it. I have nothing against the UI. I think the gameplay is beautifully designed, and it fixed all of the many gameplay issues of Civ3. But all the game balance in the world doesn't do a bit of good if the game is unplayable.
To those for whom the game works: Congratulations. I wish I could say the same.
Posted by Shawn Summers at December 30, 2005 02:29 AMI've been playing CIV since #1 and am a big fan of the series. I bought my first computer mostly to play this game. What I'm able to play of CIV4 is great. But I'm experiencing continuous crashes too - even after the patch and in spite of meeting all the minimum requirements on the box and most of the recommended requirements. This is ridculous! I don't really have time to play PC games at all. I certainly don't have time to download patches and tweak graphics settings. I'm furious that I payed FIFTY dollars for this tease.
Posted by Tony at January 3, 2006 09:04 AMI can't agree with your review more. Further, I have written both to the support people and to Firaxis about this piece of crap and asked where I can get a refund.
I fell in love with games because of the original Civilization. I remember the first night I played it. I didn't realize how hooked I was until my wife asked if I wanted breakfast...
Civ 2 didn't disappoint either and Civ 3 was fantastic. It is still me favorite game of all time.
But this buggy, crashing piece of **** should NEVER have been shipped until it actually played well on most computers...and my P4 with 2 gig of memory and a top of the line GeForce isn't most, since most people own a P4 with 512k and onboard video....or whatever.
So, I am still waiting for Firaxis to tell me where I can get my money back. And sadly, I don't think I will be playing CIV4 again, even when they ship it.
Back to CIV3 and hope these guys learn their lesson about shipping before its ready.
Posted by Robin at January 13, 2006 11:42 PMPlease help support Tea Leaves by visiting our sponsors.
