Feedback Filtering

1. TLDR
2. Who provided the feedback?
   1. How well does it represent the target audience of your game?
   2. Developers, peers and partners
   3. Members from target audience
   4. How much of the target audience does it represent?
3. Is the feedback relevant and timely?
4. Does it serve your intent?
5. Has that ship sailed?
6. Efficient feedback analysis lies in the preparation

Collecting feedback from players is easy, but knowing what to do with it can be a whole other story, especially when there is a lot of it to parse. It is not rare that feedback from different players will be contradictory, or otherwise incompatible. Some may express strong feelings about very nitty gritty details. Others may be vague and hard to translate into something actionnable. This article aims to share some approaches I use to filter down feedback to what’s most helpful, actionnable and prioritise which feedback to address first, or at all.

TLDR

• When considering if feedback should be addressed or not, it is important to consider how representative the feedback is of the intended audience, which segment of that audience might benefit and how many players does it represent; Based on who provided feedback and where it was collected, it might be biased in a variety of ways, which should be taken in account.

• When considering if feedback should be addressed or not, it is also worth considering if the feedback supports the creative vision, and if so, how much effort it would take to implement and at what point it is too late.

• Analysing feedback takes time and skill. The most efficient way to distill insights from feedback efficiently is to collect the right type of feedback from the right persons from the start, so that less filtering is needed.

Who provided the feedback?

The first thing I like to look at is who provided the feedback:

How well does it represent the target audience of your game?

I usually discard feedback that comes exclusively from outside the target audience. Chances are high that this person would not appreciate the change even if it was implemented, by virtue of them not appreciating the game/genre overall to begin with. Their feedback may not only be different from your intended audience’s but it might conflict with it. Implementing it might end up resulting in a net negative change.

This is why it is important to get feedback from your actual audience. Feedback provided by other sources should always be taken with a grain of salt.

Developers, peers and partners

Developers working on the project may be able to provide feedback earlier thanks to being more comfortable looking at early builds and looking past bugs, but their expertise is what they’re building tends to make their feedback more polarised. They may be overly positive due to building hype around the project to keep teams motivated. they may be overly critical of their own work. They may not see the forest for the trees, and push personal agendas through their feedback.

Other professionals, peers and partners that do not work directly on the project may provide a fresh perspective. Getting their expert input may be valuable, but similarly to the development team, they are unlikely to be representative of the target audience. At the least, they may be more tech savvy, belong to a higher socio-economical class and represent a biased sub-set of the target audience even if they fall within its definition.

Friends and Family, random sampling Friends and family may be close in profile to the intended audience if they fit the profile, but they are more heavily impacted by desirability and confirmation bias: they tend to be less critical than actual players if they do fall in the defined audience profile.

Similar limitations apply with random sampling, for example if you are testing a game in a public place such as a mall, library or similar venues. Besides how well the people providing feedback fit the intended audience, their feedback may be biased by desirability bias, by who self-selects into providing feedback or even by simple time constraints.

Members from target audience

Actual players from the target audience can be met in different contexts and venues, which all have pros and cons as well.

Feedback collected from the target audience at game shows and events are likely to be representative only of a sub-sample of the most engaged players. These are fans who may be more or less critical of the game, depending on the community. The setting of a game show tends to create hype and for a fan to meet the developper of a game also may create strong desirability bias.

Feedback collected from the target audience from public forums, whether message boards, social media or comparable sources are all impacted by the fact they are public. You can not be sure how much of what is said is genuine feedback, the impact of social dynamics to fit in or fit a persona, how much is exagerated or censored due to it being public etc. Besides the effect of feedback being public, there is also a strong change only a vocal sub-set of the audience engages in these public discussions, and may not represent the whole picture of even a significant portion of it.

Feedback collected in the context of a study, with participants recruited specifically based on criteria to fit the target audience as best as possible are likely most representative of the broader audience, and therefor the most valuable. This typically implies recruiting specific members of the audience from a panel, based on a screener survey, and inviting only those meeting strict criteria to provide feedback, often in exchange of some gratuity.

Which leads us to the next consideration.

How much of the target audience does it represent?

When looking at feedback from the target audience, there is value in considering how much of the total audience any piece of feedback might represent. Feedback can then be prioritised based on the size of the segment it is most likely to impact, as well as how strategic that segment is for the developer.

This implies having a framework for segmenting the audience and having a good understanding of the size of each of those segments. The segments can be fairly simple, such as new or returning players, players who fit different motivations for playing the game, or very refined profiles such as elaborate models based on game telemetry, or personas crafted from profiling research.

This is especially when dealing with contradictory feedback coming from different places. Looking into the causes of the contradictions by audience segment may reveal differences in their needs that need to be articulated, might be offered as a choice, or require a conscious decision that either of the audience’s feedback will not be acted on. This also gives an opportunity to message that decision in a way to improve the reception of that decision and/or minimise backlash if possible.

Is the feedback relevant and timely?

Does it serve your intent?

Finally, and one of the most important lenses to look at feedback is ensuring that it serves you.

Players may provide feedback and ideas that are good, but don’t fit in with the game’s vision. Some aspects of the creative process may not be up for debate if feedback challenges the intention of the design. In other cases, maybe the feedback doesn’t challenge the intent, but also does nothing to support it, making it less important to get right. On the other hand, if feedback can be tied directly to how it negatively impacts an intended experience or points out how something works against the creative vision, that is likely to be some of the most valuable information.

In some cases, feedback that goes against your intent may be a sign that the base assumptions the vision was built on were incorrect. In those cases, it may be worth digging deeper into the feedback at least the first time it shows up, so that it can be confidently ignored or addressed - ideally this should happen very early in development only, during concept phase or early pre-production, as if these are caught late turn out to be a genuine concern, it may be very costly to address later.

Similar to fitting the target audience, looking at feedback through the lens of a project’s vision requires that it is well defined upfront. Clear project pillars or foundations that describe intended experience outcomes at a high level need to be identified upfront. It is also important to formalise them: it is easy to end up with moving goalposts if they are not clearly identified from the start. It is not only helpful for deciding what feedback to address and how, but also to provide clarity and consistency as a while for the project, to ensure all developers work towards the same thing.

Has that ship sailed?

When looking at any feedback, it is also valuable to know upfront what scope is open to change and what is not - just from a logistical / project management perspective.

Regardless of how relevant feedback is, depending on the stage of development, the validation process, how complex changes may be to implement and how much cascading work might be created by a change made upstream. For small changes, this may be the easiest “line in the sand” to use when deciding to not address feedback: it is not realistically possible.

It becomes more complex depending on how critical the feedback may be, and it can involve some tough decisions. Sometimes a major change and deadline extension is worth addressing game-breaking feedback, other times it means a whole feature might be cut, if it can not be realistically improved sufficiently to meet the bar for “acceptable” due to lacking a clear path to success, lacking time/resources and/or having too many knock-on effects.

Efficient feedback analysis lies in the preparation

Deciding if feedback is relevant, actionnable and assessing the potential impact of addressing it can be very time consuming, especially when there is a lot of it. The best way to efficiently distill feedback into action is to plan for how it will be used long before collecting any.

Often, filtering and prioritising feedback effectively requires having built a frame of reference upfront that can be used as a guide, such as audience definition, segmentation, and documenting design intent.

A deep understanding of the intended audience makes it much easier to collect feedback only from relevant participants, and ensure different segments are represented, even in case where sourcing feedback must be more opportunistic.

A deep understanding of the project vision, intended experience and development process also helps with focusing the feedback on what is relevant and actionnable by asking specifically for feedback about aspects of the game that are still flexible.

Hopefully with these combined, you will be able to both make the most of existing feedback sources and design feedback loops that have the most added value for your projects.