Game Design, Programming and running a one-man games business…

Why valve should give jobs to journalists.

cliffski

I do not know a single developer who has not knocked back a neat whiskey and, in the manner of someone who has seen things no man should see, when discussing the woes of the battlefield and the horrors of war…found themselves mumbling to their comrades, wiping the dribble of 100% neat alcohol from their chin and expressing the common sentiment… “fucking steam reviews eh?” before sobbing and falling off their bar stool.

Its not just a problem for steam obviously, or games in general, biut all online reviews. Anyone who runs a restaurant or a bed-and-breakfast or small hotel will tell you that ‘consumer reviews’ are not the panacea they are sometimes described as by businesspeople who run online portals.

In a sense, the theory behind reviews given freely by users are perfectly sensible. By definition, the seller of a good (me) is only going to promote it in a positive light. We are going to tell you our game is amazing (even if it isn’t) and we will not draw your attention to defects or bugs, because we have a financial interest not to do so. The other side of this, obviously is the consumer, whose interests (in the short term) are the complete opposite. They have a financial incentive (they do not wish to waste money on an inferior product) to know all the faults of a product before purchase.

So to quote pretty much every character in a Tolstoy epic. “What is to be done?”

review1

The uniformly accepted decision is to allow customer reviews. Theoretically the customer is unbiased, and only interested in reporting facts, Thus the customers have a shared objective and can trust one another, as nothing is to be gained from leaving an unjustified bad review. Thus, sanity prevails, reviews are free, and everyone is happy.

The system described above is based on what is called ‘classical economics’ or what, when I was at the London School of Economics was called ‘Economics’, because back then, we didn’t know any different. Thankfully since then, the whole field of behavioural Economics has grown up, and we know now that Classical Economics, at least at a micro level, is 50% bollocks.

Classical Economics makes assumptions that turn out mostly to be wrong, and the design of customer reviews makes many of those assumptions.

review2

Assumption #1: The customer has nothing to benefit from dishonesty. This is 99.9% true, and thankfully I do not see it as a problem on steam (at least not personally), but the area of restaurants, books and so on suffer hugely from this error. Where reviews are anonymous, chaos reins! As a seller, in theory I have much to be gained from reviewing competing products badly, and theoretically, from creating shill accounts to review my own games positively. More likely (thankfully also rare) is the possibility of extreme fanboys of game A giving negative reviews of game B, because its a ‘rival’ game, or because the developer supported gamergate, or because the developer is female, or whatever…

review3

Assumption #2: The customer has perfect information. In terms of steam reviews, this would mean that the customer is fully aware of the product they are reviewing. To do this, they would need to experience it on multiple PCs, on Windows and Mac, and Linux (if supported), also in multiple languages, and to have completed the entire game, plus tried modding, co-op, multiplayer and so on. In short, they would have to dedicate several weeks of in depth research to completely evaluating the experience. In practice, many reviews are based on one PC, one platform, one language and a relatively short playtime of limited features. This is a partial experience, not an in depth evaluation.

rev4

Assumption #3: The reviews represent a representative sample of players. I’m tempted to just type HA! at this, but it requires more explanation. A game that draws people in to play for hours and hours may well get negative reviews in the first day of release, simply because the only people leaving reviews are people who hit a bug, or otherwise stopped playing. In other words, the silent majority who like the game are too busy playing to review it. This is definitely true on steam where you are forbidden from encouraging reviews from the app itself (unlike on mobile), and where there is no minimum playtime AFAIK.

Of the Democracy 3 sales, I can tell that roughly 1% of players have left reviews. The chances of the 1% being a poor cross section is very high. Angry people are more motivated to write reviews than happy people. People with time on their hands (kids & students) are more likely to leave reviews than busy gamers (20+ 30+ and parents).

So what is to be done?

what

 

I think if you want to keep reviews you have to accept that the only real way to fix 1) 2) and 3) is to have at least some paid reviews. If I owned steam (I’ll get this on a t-shirt one day), I think I’d take a chunk of the sales profits (steamspy suggest roughly $200 million this sale, so say 30% is $60 million, lets spend 2% of that, $1,200,000) and hire a bunch of reviewers, full-time. I’m sure the world of games journalism has a bunch of unemployed writers that would love the job. Lets pay them $60k each, with admin & health insurance and all that, we get 12x$100k reviewers. So that’s 12 full time games reviewers working for valve.  Not a lot, but not insignificant. They don’t have to review all the shovelware, just the games selling thousands of copies.

Suddenly we have a bunch of ‘pro’ reviews mixed in with the wider range of existing ones. Now a big part of the problem is solved, but I’d go further and do some weighting based on another metric.

Time played.

Frankly if you play a game for 40 hours then give it a negative review you need therapy, or a job. That makes no sense. Also, TBH if you play a game for 40 minutes, you really shouldn’t be reviewing it *at all*. The beauty of ‘time-played’ on a game is that steam already calculates and stores this for ALL the players, not just the vocal 1% who leave reviews. The stat is not perfect (some people ‘idle’ a game to get trading cards’, but its not bad. Presumably steam *could* write some code that detects a player being really AFK and fix that anyway.

I think some combination of pro review scores (let them give 0-100% not just positive or negative) combined with user review scores combined with player time will be a far better (and harder to ‘cheat) version of what we have now. Just my opinion, and I’;d like to read your thoughts.

BTW before anyone starts yelling, I’m not complaining specifically about my scores, I have generally good reviews, and to state it for the billionth time, my games sell fine. You can criticize without ‘whining’ if you lay out your arguments sensibly :D.

27 thoughts on Why valve should give jobs to journalists.

  1. A 40-minute-or-less review is potentially quite accurate if the reason for the negative review is terrible technical problems. If you check out the WORST-rated games on weekly sale on Steam every week, some of the most common reasons for terrible review scores are things that are apparent in a very short time of playing: crashes all the time, doesn’t even launch, controls are terrible and unbindable (sometimes requiring a touchscreen because the mobile port was THAT bad), doesn’t support a joystick even though it says it does, is literally a copy of the ‘My First Game’ tutorial from some game building kit, and so on.

    Of course, what you can’t tell from a user’s review is whether they made any effort to resolve their problems. Sometimes a technical issue is fixable but the user threw a fit and demanded a refund rather than wait for someone to reply to their request. Most players are more patient, though, so a game generally doesn’t get overwhelmingly neutral reviews unless it’s really broken and the developers have no intention of fixing it.

    As for forty-hour negative reviews, how reasonable they are depends to a great extent on why the player is angry. Since Steam only allows a “thumbs up” and a “thumbs down” with no in-between, if you play a game for thirty-nine hours finding it to be okay if not great, and then suddenly it doesn’t just have a bad ending but a grossly offensive ending that makes you hope the game developer goes out of business and never works in this town again, you’re certainly not going to leave it a thumbs-up, and you certainly do want to tell everyone how annoyed you are.

    Some method of sorting the reviews by playtime might help prospective players look for different kinds of feedback. Both short and long negative reviews MAY be useful, but they should be complaining about totally different kinds of things.

    1. point taken regarding broken/buggy games on the sub-40 minute mark. I do think that bugs and major problems aside, its hard for you to really be able to give a genuinely helpful review of gameplay below that number though.

  2. Ok, I mostly don’t agree…

    Reviewers paid by Valve couldn’t be trusted, it’s an obvious conflict of interest.

    User reviews are often of questionable quality, yes, and one review is the experience of one person on one computer, sure. But after reading a dozen reviews or two hopefully you should have acquired some useful information, and possibly more information than you get from a professional review, because pro reviewers also only test the games on one computer, in one language, with one keyboard layout… (And also they usually didn’t pay for the game.)

    Now on the topic of time played… I again don’t agree. If the game has a flaw that prevents the player from playing or enjoying it, and the flaw appears in the first minute of the game, then they don’t need to play 2 hours to make a useful negative review. Maybe 95% of the players do not care about that flaw but that doesn’t matter, it’s still useful for some people. (BTW Steam requires you to have 5 min “playtime” before leaving a review.)

    And not recommending a game that has been played dozens of hours is also perfectly valid, no need to say people need therapy… That’s quite insulting.

    1. How is it a conflict of interest for valve? A gamer who is put off buying game A is likely to spend the money on game B, so valve get their money either way, and a happier customer than one who feels cheated by poor quality reviews.

      I’m assuming someone who plays for 5 hours and KEEPS playing is having fun. Someone who has played for 40 hours has clearly been entertained by a game, so the idea of then invalidating all 40 hours over an element of dislike 40 hours in seems crazy to me.
      The average playtime of games is WAY below 40 hours, so anyone even getting close to that is already suggesting that the game is vastly more entertaining than the majority of games on steam.

      1. You’re probably right about the conflict of interest; a few negative reviews wouldn’t be the end of the world for Valve due to the huge amount of games on Steam.

        I wonder though, would they give a negative review to a big AAA game like GTA? That could result in a significant drop of sales on Steam, and annoy the publisher at the same time…

      2. It is a conflict of interest because valve owns the store. Even 12 full time reviewers are probably not going to be able to get through all of the games that are released these days, so only some games will be getting pro reviews. Which ones will these be? Over time, if pro reviews can be shown to drive sales, they will be given to games that Valve thinks have potential to drive a lot of business. In the same way that valve already uses it’s algorithms to manipulate what games are shown and which ones are shown first, pro reviews would potentially be another tool of manipulation of users.

        Not that that manipulation is wrong. That’s how stores work. Bookstores do this sometimes with their employee recommendations. It’s not quite the same, but I don’t think you could have pro reviewers working for valve and not have the feeling that they are reviewing games well that they think they can sell, and burying games that they don’t think will make them money. Customers know where they stand with customer reviews at least.

        My biggest worry with something like this is that your game gets a pro review that helps you double your sales, and mine doesn’t get any extra help so I continue to do poorly. There are enough factors like this already in play.

      3. Note: 40 hours is far over the median play time for Steam, but that doesn’t mean that the game is far better than average for that player. Many people are poor, and can’t afford to buy a new game if they are dissatisfied with their purchase. There are many reasons that a game could be played for a long time but still be unpleasant.

        I know when I was young and poor I stuck with games that sucked because that’s what I had, and I wasn’t getting it replaced anytime soon. Doesn’t mean it didn’t suck, just that I didn’t have many options.

        You haven’t been poor in a really long time Cliffy B. :-) The vast majority of the people in the world, and possibly even the majority of those on Steam, are poor. If I’m forced to live in a slum because that’s what I can afford that doesn’t mean I need to rate it highly based on how many years I’ve lived there.

    2. PS: what I would appreciate is if someone else could go through all the reviews to make a list of objective pros and cons people have found, and also write a short description of the gameplay (since publishers are often quite bad at describing their games).

  3. No no no and no

    Power of steam is user driven review

    Journalists proved they are just going to promote/review games for money, that’s their work no?

    At least make people to be able to review after xx hours played and few achievement unlocked

    But letting journalists do the reviews is bad idea

    Lot of indie games becomes famous because players gave them enough visibility

    If you let journalist do reviews, same story will be reproduced, only AAA games reviewed, game reviewed because people sent they press kit with good money

    1. but I am suggesting valve pay the journalists, not the developers. Are you saying journalists are currently bribed? Even if you think thats true, surely better to have valve solve that problem? in fact all the more reason to do so?

  4. While I agree with some aspects of the article (players are biased, also players who had negative experience will most likely post a review than those who had a good experience, etc.), I am not sure how Valve hiring journalists would supposedly help.

    First of all, Some of the assumptions are ridiculuos. I mean who assumes reviewer must have tried the game in multiple consoles/platforms, multiple language settings, modding, and all of the game modes (multiplayer, co-op, etc) to the fullest extent? Nobody has time for that. Not even journalists. I am not sure what the point of that was.

    Furthermore, there are journalists out there. Valve doesn’t need to hire anybody. Valve already gives a link to metacritic for each game. I understand if people want the link to the critic reviews to be more visible, I don’t understand why Valve would ever hire journalists when there are oversaturation of critics out in the internet.

    Lastly, critic reviews are not always portray a better representation of the game compared to user reviews. Today, games receiving updates is extremely common. What may have been buggy game at launch may be excellent currently. Players who have spent thousands of hours may return to the game they loved and be horrified by the changes (No, Cliff. They don’t need therapy or jobs.) Critics almost never return to the game they reviewed because they have other games to review. In a sense, current review system which also shows ‘recent reviews’ may be better representation of the game.

  5. Studying Economics A-Level is much different to when my teacher was studying it.

  6. While I agree that there is an element of absurdity when a negative review is posted by a user who has clocked hundreds or thousands of hours into a game, I think there is actually value in spending a sizeable amount of time with a game before lambasting it.

    If I see a negative review with forty hours played, I don’t think “get a job!” – I think that the reviewer has invested enough time to deliver a well-informed opinion, or has discovered nuanced issues that are only apparent after some in-depth experience with the game.

    There are exceptions, of course. These are readily identified within a few sentences, typically. It’s fairly easy to sort the dross from the worthwhile opinion.

    In short, negative reviews with high play-times can be valuable, and don’t always present a paradox as they might superficially appear to. A single play-through of some games may exceed 40 hours! Many professionally employed journalists/critics are held to the standard of first completing what it is they are critiquing – I don’t suppose these writers are told to attend therapy for attempting to be thorough.

    1. Just to add to this: Time played = value gained is a false equivalence. On average, “Gone Home” took players two hours to beat, but it’s a highly regarded game that means a lot to many people. “Lords of the Fallen” took players an average of 14.5 hours, but is considered critically and by consumers to be mediocre.

      1. certainly very true. I don’t think I would ever want ‘time played’ to become THE metric, but I think it should be in an equation somewhere. I suspect you would get good results with 33% time played, 33% user reviews 33% pro reviews.

  7. Since playtime itself can be a metric of continued fun, perhaps Valve could display playtime statistics as a basic game enjoyment metric. This could be combined with the ability to organize user reviews by their playtime.

    1. And thus Clicker Heroes would become the “most enjoyable” game on Steam overnight. That system clearly wouldn’t work, not least of which because game length and replayability vary greatly.

      Sorting reviews by playtime could be useful though. Reviews from players with more playtime tend to be a bit more coherent and accurate usually.

  8. As a journalist who has written about games for the past decade, this is a bad idea. Not because journalists can’t be trusted (they can’t), not because Valve can’t pay journalists to be accurate (they can’t guarantee it), and not because it’s unethical (it is unethical, but barely).

    It’s a bad idea because it would be extremely costly for Valve to spend money in this fashion when they could do dozens of other things to “fix” the problem of user reviews. This includes anything from “Pro” users, who hit certain milestones in a particular game they’re review (which can be set by the developer/Valve), using key metrics to define a review’s value (based on the individual user’s history as well as a composite of the given scores from all other users), etc. The list goes on and on.

    Hiring a team of journalists, or even just a small handful, is like saying let’s QA everything by hand with some well-paid professionals…instead of automating the testing. Sure, the quality may be higher (if you’re lucky and have grade-A people), but this is an issue of scale, not of opinion or one-person value. So to scale it properly would require having a team of not a dozen, but dozens, if not more. Paying them even $50K/y would be a stretch, with proper benefits. The economics breaks down so fast that no one in their right mind would want to bet on it.

    There’s also little to no economic benefit to Valve for doing this. Sure, positive reviews may lead to more game sales of good games, but it’ll be leveled out by negative reviews. Companies like Ubisoft, which have their own stores but also sell through Steam, will be less inclined to do so, forcing Valve to lose money on those games non longer in inventory. And end-users will lose out for obvious reasons.

    If I were to pitch something to Valve to fix this “problem”, it would never to be to hire journalists, or reviewers. Today’s game critics (and this has been true for at least 20 years) aren’t critics at all. Most are bloggers who like to play lots of games The industry has not improved because of the existence of that business. Valve hiring some is a good way for the company to show that it needs a crutch for something much larger. And it doesn’t need a crutch.

    No, I’d pitch something about determining the goals for the business regarding user reviews. What benefit do they serve? Do people who like positive reviews tend to buy? Negative reviews? What do the numbers show? And based off of that what can be done to improve the quality of the Steam experience for everyone? All hiring a team of journalists will do is add a high cost with no true value proposition attached to it.

    Unless it’s a charity thing…with so many bloggers out of work because too many are getting laid off, maybe they can band together and Valve can use that as a way to get out of paying some extra taxes. Now that would be something.

  9. How about no.
    While Steam has it’s problems, “journalists” would soon become another one in the pile.

  10. Re Assumption #1: your alternative is games journalism, which often benefits from dishonesty. (Paid reviews, anyone? even by Youtubers?)

    Re Assumption #2: why does it make a difference if you have 1 reviewer testing it on 10 platforms instead of having 100 users testing it on 100 platforms? Surely the second approach is superior? If my platform is in any way unusual, I’d wait a while before purchasing the game and then check out the tech support sub-forum.

    Re Assumption #3: Reviews often get written by people who like to write reviews, be they good or bad. This is actually not so different from games journalists, who wouldn’t be representing everyone either.

    If you have been playing games for a long time, you can often see the problems with a new game after a relatively short time. Since I own (and buy) many more games than I have time to play/finish, I will drop that game then, and possibly write a review labeled “first impression” if the game still doesn’t have many reviews. (I also do this for games I like if I may not have the time to finish them. Unlike a games journalist, I can always update my review later if I do finish it.) I assume in this that I am representative of other players who would drop a game once they found it is not fun. By restricting reviews to players who played the game for a while, you would be restricting reviews to people who liked your game, and thus bias the reviews. (The same goes for in-game review prompts: they increase the proportion of reviewers who are enjoying the game, and placing the prompt right after a positive experience would only play to that bias more.)

    Players who have hundreds of hours into a game and still leave a negative review may be reacting to a recent update. (Who changes their game 3 years after it’s been published, you may think, but it happens.) Again, barring reviews of bad updates because of some artificial metric seems like a bad idea from a customer perspective.

    If you want good user reviews on Steam, identify players who left substantive reviews for games similar to the ones you want to publish. Give them free Steam keys, and ask them to review your game. Not all of them will, but if your game is good, this can be an initial “stock” of good reviews that may offset some bad first impressions.

    If you expect bad reviews because of bugs, consider an “early access” roll-out. Reviews on Early Access are marked as such, and reviewers are usually much more forgiving of problems in the EA phase since they assume that you’ll fix them once customers have brought them to your attention. Another option is to leave developer replies on the reviews mentioning bugs once you have fixed them.

  11. Nope, don’t agree. Not even slightly. Just look at your examples of “bad reviews” you put on here. If these are seriously the worst reviews you could find then steam reviews must be in a pretty stellar state.
    1. Review: While this review is not very in-depth it clearly depicts the thoughts of the player about the game, namely that the game is very bad. Some more Information can be helpfull to people, but this is still valid information that can help people make a buying decision.
    2. Review: This is actually a pretty helpfull review. If the player is unable to start your game because of technical problems, then that is your fault and your fault alone. You are the one selling a product here, if your product is so defective that it can’t be played at all, then that’s pretty important information for potential buyers. If you don’t want to get reviews like that, don’t release broken games. Problem fixed, there.
    3. Review: I give you this one as it’s not really a good review for the game in any way. It doesn’t highlight any problems with the game and just makes a rather lame joke. It’s also a single review. Out of 14,5 THOUSAND reviews. There are bound to be bad apples there.
    But in general when I look at the most helpfull reviews I tend to get all the information I need about the game. You may not like it as a developer, but technical problems and pricing shemes are important to customers and they have a perfectly good reason to give you bad reviews for that kind of stuff no matter how good the gameplay may be.

  12. So that means, you say that everyone should either review POSITIVE of your games or they should just not.

    And who are you to say that “Playing” 40 Hours people need to go to therapie. The only one who should go to therapie is YOU.

    Do you really think people are that stupid to not buy your game because of steam reviews? NO!

    People buy it or not buy it because of youtube and other video Plattforms where they see the game.

    And the Second review is very accurate, if you can’t play the game because of technical difficulties that weren’t fixed it’s YOUR FAULT. And no one else.

    Here is the only thing i can say to you. “Keep your head low and watch out for all the shit that is gonna fly to you because what you just posted is such bullshit you should be kicked out of game development”.

  13. why on earth can I not play a game for an hour and say “man, I didn’t enjoy that at all”? core gameplay rarely changes after an hour into the game; sure, it builds up, but a rotten foundation won’t be saved by a flashier and cooler version of more of the same. Besides, play time is already listed on the review itself!

    to your concerns about “Windows and Mac, and Linux”, why not have the review list the operating system used as well? In fact, couldn’t steam do that automatically? Meanwhile, since the reviews state hours played, anybody reading can draw their own conclusions about how good the rest of the game, including the ending, multiplayer, and co-op, is. Granted, a mod might save the game, but is it really the job of the reviewer to pimp out content the developer himself didn’t even create?

    Meanwhile…you are aware that if I played a game in, say, Spanish, then people would read my review, which is probably in Spanish too, and draw their own conclusions? If anything

  14. The solution to “bad” information is rarely to suppress that information, but to instead add more and better information.

    Steam already reveals how much time the reviewer spent playing the game, so we, the players can judge for themselves if they had time to come to the conclusion that they are presenting.

    Yes, the review system has problems, and yes, Steam can (and has) make improvements, but all reviews must inevitably leave out much of the details. It is the price we pay for being able to quickly read and sort through the many options available to us.

  15. This post certainly poses an interesting thought to ponder. However there’s one thing i have to say you’re simple a little narrowminded on:

    > Frankly if you play a game for 40 hours then give it a negative review you need therapy, or a job. That makes no sense.

    There are games that go that long, and as a main task carry a story. It is entirely possible for a game to keep a player engaged for that long, and then deliver a twist, or ending, that recontextualizes the entire story before that point, or is just plain a massive disappointment. (And yes, games other than Mass Effect do that, for example one game got the player very invested in one of the non-player characters and then forced the player to kill said character, very gruesomely too, at the end, no choice given.)

    Sometimes only the journey matters.

    But sometimes the destination is important too.

Comments are currently closed.