A few thoughts I’ve been mulling over for the last few weeks, concerning steam, and most relevantly, the steam review system.

I like the idea of product reviews. So many people try to sell you crap with a lot of marketing bullshit and vague promises. Reviews are a powerful weapon which allow little-known but great products to rise to the top, and punish the superficial, poor-quality crap with big marketing budgets. Done correctly, reviews are a win for the consumer and for the developer. Consumers get unbiased purchasing advice, (and lots of it), and also a voice for their opinions, and developers get a free marketing department for good products, and constructive feedback on why some customers are unhappy.

Of course, that is all theoretical, the real impact of reviews depends vastly on their implementation. In many ways, the steam implementation is extremely good, and actually better than the implementation you see on some other websites. Firstly, you have to actually have bought the product to review it, which eliminates 95% of dubious reviews. I could easily go and review my neighbours B&B on google and say it sucked, if he is someone I don’t like, but with steam, if I wanted to maliciously give bad reviews to every other developer’s strategy games, I’d have to buy them all, which acts as a powerful brake on people just acting like dorks.

Also, steam lets you rate reviews as helpful or unhelpful, which is cool, but AFAIK this has no impact on the extent to which a review is counted towards the review score. This is a tough line to walk, because AFAIK anyone can rate a review without being a customer. If steam allowed review ratings to influence review scores, then you are back to square one with the malicious review-manipulation issue. The review-rating system is presumably a nudge towards  encouraging thoughtful reviews, which probably works to an extent, but you still have a problem that people may leave a bad review for the wrong reason such as ‘Developer is a woman/gay/nazi/non-white…’. How can this be combated?

I think the solution is pretty simple, and obvious when you go back to first principles and ask yourself what a review is supposed to be. let me put forward this assertion:

“A review is an objective measure of the collective opinion of customers as to the quality of the product they have bought”.

That sounds pretty fair to me, and when you put it like that you realize that we try to collect such measure all the time in the real world, with question like this:

“If an election were held tomorrow, which candidate would you vote for?”

Yup, opinion polls are basically trying to do the same thing. They are trying to work out what people think of products, in this case politicians and parties. The key thing I’ve realized, is that there is a wealth of expertise and knowledge gained from such systems about ‘how to do it right’, where ‘right’ means predict the real opinion of everyone from a small subgroup. With this in mind, lets look at everything game reviews on steam do wrong:

Problem #1: A self-selecting electorate.

You don’t have to review a product on steam, and you get NOTHING for it, if you do. No steam points, no gems, no chance of a discount coupon, nothing. You take up your own time. As a result, steam reviews are basically like holding an opinion poll where people have to choose to take part, and then take their own time and effort to participate. Any pollster would laugh you out of the room if you tried to predict an election result by waiting for the public to come to you and tell you how they would vote. You get the activists, the extremists, the angry, and also more relevantly, you get people with time on their hands. You would have huge over-representation by the unemployed, the teenagers and the retired. The result is worthless.

Problem #2: A small sample size.

Ask 10 people who will win the next election and you will get a pretty useless result. Ask 100 and its closer, but for a really close election (48/52% style) you are going to need thousands, even assuming that you have carefully ensured its not self-selecting and that the gamers have been randomly polled.

Problem #3: People lie to themselves.

Some political opinions are widely held but publicly frowned upon. In the US, saying you supported trump would be unpopular in some circles, in the UK, supporting UKIP can be seen as signifying racism. In working-class towns, saying you vote conservative is downright dangerous in some places, and a labour sticker in some conservative villages will exclude you from dinner-parties. Pollsters try to find out what people really think and will do, not what they claim to think and do. In gaming, we dont have much in the way of ‘shame’ although its interesting that all reviews are public and non anonymous. How many people dont want to have a positive review of a gay-dating sim visible on their profile? How many gamers wont post a glowing review of a game they love when the developer gets hate due to their political views? Its probably not *that* many. However, we do have a problem where gamers routinely plough hundreds of hours into a game, then give a negative review. This seems…. weird. To some extent, steam should be able to factor this in. Maybe some fudge factor needs to take players median play time into account when computing a score? This is the trickiest area to fix.

 

So the first two problems are EASILY fixed. You just get more people to review a game. Don’t leave it to the bored (mostly young) or the incredibly outgoing, happy to write comments everywhere (again, mostly young) crowd, or the angry mob (people are more likely to review badly when something goers wrong than they are happily when something goes right). Steam needs to do a simple thing… Raise the percentage of gamers leaving reviews above 1%.

Proposal 1 (Meek). Make it easier to leave a review

You can see a big ‘write review’ box on the store page. So whats the problem? NOBODY visits the store page after they bought the game. Why on earth is that big box not on the games page within the steam app itself? This would be easy to do. Also…on the page for a game right now, ‘write review’ is TINY. I couldn’t find it the last time I looked. Even a different color or a bigger font would help. The current UI design for this is incredibly meek. There is a big fat piece of prime estate next to the play button where it could go instead!

Proposal 2 (Bold) Incentivize reviews

The minute you add any reward for anything on steam, you get side effects, so for now lets ignore the idea of giving out steam points, or gems or anything, and just keep it really simple. When you quit a game session lasting more than 30 minutes, if the player has not reviewed the game , pop up a dialog (like the screenshot uploader) asking them if they want to leave one. 95% of them will hit escape, but even if the other 5% leave a review, we have boosted the accuracy of steam reviews by 500% immediately. Concerned about the 30 minute hard limit? fine, make it random for each player/game combination between 30 minutes and 8 hours, so you get a random sampling of play-times.

I’ve been thinking about this a lot, and these are the solutions that I think are a) hard to cheat and b) easy to try. You can even A/B test steam users and see what the effects are before rolling out to everyone. I’m interested to hear peoples opinions on this, and think its always worth discussing this sort of stuff. It applies of course not just to steam, but GoG, Humble, Itch and everyone else. Its in everyone’s interests that game reviews are fair and accurate for all.

 

 

4 Responses to “Fixing / Improving game review systems, for gamers and developers.”

  1. CdrJameson says:

    Surely this is the old economics question of stated vs revealed preferences, so why not go to what people are actually doing, not what they claim their opinions are?

    – How many people have installed and played game for more than five minutes?
    – Of those, what is the average number of times they have played the game?
    – Of those, what is the total length of time played?

    Then you could say what percentile of similar games this particular game is in.
    (Epic RPGs would have massive numbers compared to a shooter, but a good shooter would have a lot of sessions, so they’re a bit apples-and-oranges)

    Might want to throw standard deviation in there too.

    This could supplement a review system, and would have the advantage of taking no effort on the part of the players.

  2. James says:

    Both those ideas are great, particularly the second one about a prompt. There are many games I would have left reviews for in the past if it had been suggested to me, otherwise it just doesn’t naturally occur to me.

    What about asking within the game itself? Or would players see that as too intrusive? Youtube content creators almost always ask for viewers to “please like and subscribe if you enjoyed this video”. No-one really seems to have a problem with this, people have become accustomed to helping out creators this way and understand why it helps.

  3. Sean G says:

    I’ve seen people put review requests in the game themselves. So like after X number of hours, a window will come up at start asking you to review the game. While that won’t resolve the situation for the system as a whole, it might be a good idea for Production Line and other upcoming games.

    To be fair, the review system works quite well for complex games like Rimworld, Factorio, KSP. While these games are exceptional to a degree that you shouldn’t beat yourself up for not being them, they are still complicated games that are not so noob-friendly yet they are among the top rated games on the platform.

  4. Tony Brice says:

    Sorry, a random prompt coming up after quitting a game session is not appealing to me at all. I’m all for a button the same size as the Play button being right next to it, but I’m sick of nagware. I do agree it needs to be more obvious that a review can be made but not to the extent that it’s intrusive.

    I’ll admit that I don’t leave that many reviews and I would definitely do it more for games that I’ve got either a very positive, or very negative, feeling from. A visible button next to the Play button would be more than enough to remind me to do so, if I felt like I had a useful opinion.

    I’m as sick as the next guy of 120+ hour players leaving a negative review as they’ve both had their money’s worth from playing it and must have enjoyed it enough to persevere that long. Certainly it seems a bit petty to knock a game that had entertained them for that length of time just because of something, possibly minor, that didn’t work for them.

    It’s even worse for the played-ten-minutes-brigade to slate the game mercilessly. How much of it did you realistically see? Sure, some people may have had access to play it outside of Steam in a beta, or similar, but how often is that? I think a review should maybe not be allowed until there’s at least 30 minutes of game time clocked up. They still have time to refund the game afterwards, if it’s really that much of a stinker.