Early access feedback surveys
Polls and surveys are very simple ways to collect targeted feedback from players during early access – even if you don’t have a researcher on your team. They can be super powerful tools, cheap to distribute and easy to analyse nowadays with online survey tools that do most of the hard work for you.
There’s a few traps that are easy to fall into however, which I’d like to discuss here.
Get data to make your own decisions
One common trap I’ve noticed developers often fall into, is directly asking players/participants to tell them what they want. Users are great at telling you what works and doesn’t work for them. On the other hand, they’re terrible at knowing what’s best for them. As a result, what they want is often not what they need.
For lack of originality, I’ll quote Ford, like everyone else: “If I had asked people what they wanted, they would have said faster horses.”
Don’t ask players to design features or prioritise your work for you. Sure you can use that as input, but ultimately, you should take a step back and do your own thing. Look at what they say they want with a critical eye and try to get to the bottom of why they want that, and how you’re going to satisfy that underlying need.
Ask the right audience for each question
It is easy to ask questions to the wrong audience and draw poor conclusions. If you’re interested in something, make sure you’re asking the people who are concerned about it. For example:
- Want to be more inclusive? Ask the minorities
- Want to know if your tutorial needs improvement? Ask players who just started playing, or better yet, reach out to those who stopped playing soon after trying the game.
- Want to learn about the store experience? Check in with both people who bought something and people who visited the store, but didn’t buy
- Your game is strategy game with a platformer twist? Get some fans from both genres to try and share feedback about your game to know if the mix works well for them
- Want to know if people are interested in your game? Ask people who play other, similar games. Don’t ask fans who come to see you at a conference!
It is very easy to fool yourself into thinking something works well by asking the type of audience that wouldn’t have a problem. For example, I recently saw these survey results being distributed on the dev blog of the game “Colony Survival” (which inspired this article).
First of all, kudos to them for getting feedback from real players. Still too few developers do that to begin with. The survey is also very to the point and seems practical and actionnable.
Now do you see the flaws in the survey’s design?
- This survey was sent out through the dev blog. This means the most likely type of respondents is the niche group of players that are engaged enough to actually find the link to the survey in the dev blog – it seems obvious they will not vote for better tutorials as a priority, since they already know how to play
- I’d expect a similar thing to happen with the “fix bugs and optimize performance” option – I’d imagine people who experienced bad enough bugs would not be filling out the survey because they’d have quit playing the game
- Generally, some of the options have clear immediate benefits for players, other seem more indirect, which I’d expect to predict what players would prioritise, based on my experience – if only because some options feel more abstract than others. Asking how often players experienced issues with a given system likely would give a better idea on what bugs / fixes to prioritise, if any.
- The question asks players to prioritise the development process, however I don’t think that’s up to the players. A better way of asking would have been to ask a Kano style survey. This basically means asking how people would feel about each feature being included, then ask about how they would feel about each feature being absent. This is a very good way to prioritise development efforts without asking users what they want.
- Finally – this you cannot see from just the answers – the question is asked as a matrix scale, where players can give the highest priority to every option. If the question is about the priority, I probably should use a question type that forces ranking. Since they used google forms, a good option here would be to limit answers to once per column.
Conclusion
Asking the right participants about topics is important because each segment of your audience can really only answer the questions that relate to them.
Asking survey questions the right way can be tricky, but is mandatory to get the feedback right. How you ask a question determines what answers you’re most likely to get – being aware of this helps getting the most out of your data
Ask questions that help you understand your user’s needs over what they want. It’ll reduce the risk of failing despite doing exactly what players tell you. Instead, read between the lines and know what’s best for your audience.
Bonus: I really liked how simple that survey was, so I made the edits I felt were relevant and made this template for others to use.
Anyone with the link can edit, so please make a copy before making changes!
https://docs.google.com/forms/d/1l0KLzzloXqlQ8tdj9Cp52f-yGDR8Li5Ab04ealLf_nw/edit?usp=sharing
En ook bedankt voor het lezen :)
Any thoughts?