I do not know a single developer who has not knocked back a neat whiskey and, in the manner of someone who has seen things no man should see, when discussing the woes of the battlefield and the horrors of war…found themselves mumbling to their comrades, wiping the dribble of 100% neat alcohol from their chin and expressing the common sentiment… “fucking steam reviews eh?” before sobbing and falling off their bar stool.
Its not just a problem for steam obviously, or games in general, biut all online reviews. Anyone who runs a restaurant or a bed-and-breakfast or small hotel will tell you that ‘consumer reviews’ are not the panacea they are sometimes described as by businesspeople who run online portals.
In a sense, the theory behind reviews given freely by users are perfectly sensible. By definition, the seller of a good (me) is only going to promote it in a positive light. We are going to tell you our game is amazing (even if it isn’t) and we will not draw your attention to defects or bugs, because we have a financial interest not to do so. The other side of this, obviously is the consumer, whose interests (in the short term) are the complete opposite. They have a financial incentive (they do not wish to waste money on an inferior product) to know all the faults of a product before purchase.
So to quote pretty much every character in a Tolstoy epic. “What is to be done?”
The uniformly accepted decision is to allow customer reviews. Theoretically the customer is unbiased, and only interested in reporting facts, Thus the customers have a shared objective and can trust one another, as nothing is to be gained from leaving an unjustified bad review. Thus, sanity prevails, reviews are free, and everyone is happy.
The system described above is based on what is called ‘classical economics’ or what, when I was at the London School of Economics was called ‘Economics’, because back then, we didn’t know any different. Thankfully since then, the whole field of behavioural Economics has grown up, and we know now that Classical Economics, at least at a micro level, is 50% bollocks.
Classical Economics makes assumptions that turn out mostly to be wrong, and the design of customer reviews makes many of those assumptions.
Assumption #1: The customer has nothing to benefit from dishonesty. This is 99.9% true, and thankfully I do not see it as a problem on steam (at least not personally), but the area of restaurants, books and so on suffer hugely from this error. Where reviews are anonymous, chaos reins! As a seller, in theory I have much to be gained from reviewing competing products badly, and theoretically, from creating shill accounts to review my own games positively. More likely (thankfully also rare) is the possibility of extreme fanboys of game A giving negative reviews of game B, because its a ‘rival’ game, or because the developer supported gamergate, or because the developer is female, or whatever…
Assumption #2: The customer has perfect information. In terms of steam reviews, this would mean that the customer is fully aware of the product they are reviewing. To do this, they would need to experience it on multiple PCs, on Windows and Mac, and Linux (if supported), also in multiple languages, and to have completed the entire game, plus tried modding, co-op, multiplayer and so on. In short, they would have to dedicate several weeks of in depth research to completely evaluating the experience. In practice, many reviews are based on one PC, one platform, one language and a relatively short playtime of limited features. This is a partial experience, not an in depth evaluation.
Assumption #3: The reviews represent a representative sample of players. I’m tempted to just type HA! at this, but it requires more explanation. A game that draws people in to play for hours and hours may well get negative reviews in the first day of release, simply because the only people leaving reviews are people who hit a bug, or otherwise stopped playing. In other words, the silent majority who like the game are too busy playing to review it. This is definitely true on steam where you are forbidden from encouraging reviews from the app itself (unlike on mobile), and where there is no minimum playtime AFAIK.
Of the Democracy 3 sales, I can tell that roughly 1% of players have left reviews. The chances of the 1% being a poor cross section is very high. Angry people are more motivated to write reviews than happy people. People with time on their hands (kids & students) are more likely to leave reviews than busy gamers (20+ 30+ and parents).
So what is to be done?
I think if you want to keep reviews you have to accept that the only real way to fix 1) 2) and 3) is to have at least some paid reviews. If I owned steam (I’ll get this on a t-shirt one day), I think I’d take a chunk of the sales profits (steamspy suggest roughly $200 million this sale, so say 30% is $60 million, lets spend 2% of that, $1,200,000) and hire a bunch of reviewers, full-time. I’m sure the world of games journalism has a bunch of unemployed writers that would love the job. Lets pay them $60k each, with admin & health insurance and all that, we get 12x$100k reviewers. So that’s 12 full time games reviewers working for valve. Not a lot, but not insignificant. They don’t have to review all the shovelware, just the games selling thousands of copies.
Suddenly we have a bunch of ‘pro’ reviews mixed in with the wider range of existing ones. Now a big part of the problem is solved, but I’d go further and do some weighting based on another metric.
Time played.
Frankly if you play a game for 40 hours then give it a negative review you need therapy, or a job. That makes no sense. Also, TBH if you play a game for 40 minutes, you really shouldn’t be reviewing it *at all*. The beauty of ‘time-played’ on a game is that steam already calculates and stores this for ALL the players, not just the vocal 1% who leave reviews. The stat is not perfect (some people ‘idle’ a game to get trading cards’, but its not bad. Presumably steam *could* write some code that detects a player being really AFK and fix that anyway.
I think some combination of pro review scores (let them give 0-100% not just positive or negative) combined with user review scores combined with player time will be a far better (and harder to ‘cheat) version of what we have now. Just my opinion, and I’;d like to read your thoughts.
BTW before anyone starts yelling, I’m not complaining specifically about my scores, I have generally good reviews, and to state it for the billionth time, my games sell fine. You can criticize without ‘whining’ if you lay out your arguments sensibly :D.