How can we de-risk long term retention?
The short answer is you can not prove through research that a game will succeed at retaining its players for a very long time. One wishes we could simply ask players how engaging a game was, how long they see themselves playing a game, or how likely they would be to keep playing a long time. Unfortunately, players are equally bad at predicting the future as they are at guessing how many hours they played a game in the past.
Even if we asked and players guessed how long they would keep playing a given game and we obtained strong positive feedback in favour of them engaging long term, there is no guarantee that this would translate into actual behaviour.
We can not prove that players would remain engaged in a game.
That said, if players want to stop playing early in the experience, chances are that would become a major churn point in the game once released. So we can kind of identify when players may not retain well and why. As those barriers to retention are identified and ironed out, it increases the potential for long term retention, by at least making sure nothing gets in the way of it unnecessarily.
Factors such as the player’s context, competition from other game releases and other external potential causes of churn remain unknowns but at least you can improve your odds by managing the aspects that are under the developper’s control.
So what can we actually do?
Onboarding
For players to engage with any experience long term, they need to feel successful early on. If they quit before even completing the out of box flow or initial tutorial, there is very little chance of them returning at all.
Evaluating the usability of the first time user experience, onboarding flow and initial tutorials, up to the first hour should help identify any points where players risk ejecting from the experience early due to barriers, blockers or frustrations.
This is best done in one on one usability sessions, where depending on the type of gameplay, a handful of participants from the target audience would play through the initial hour of gameplay while commenting on their actions and/or answering questions on the fly. Each additional participant is likely to uncover several new issues, with diminishing returns beyond a dozen participants. Depending on budget and time, it can be beneficial to run several smaller of such studies and fix issues in between to iteratively surfaced new ones, as some issues may obfuscate others.
Issues that affect even one player are likely to affect a significant number of end-users once live. Addressing issues by impact, regardless of how many participants experienced them, is the best way to iron out any early ejection points.
Fun
If players managed to onboard successfully, for them to remain engaged with an experience, it needs to be fun at its core. If the basic actions players perform in the game loop are not inherently enjoyable, satisfying, and rewarding, chances that players will keep wanting to repeat them are fairly low.
Evaluating fun requires a decent number of participants that are representative of the intended audience to play the game and provide ratings and in-depth feedback for various aspects of the experience. The more participants, the less ratings are subject to chance variations, although there are diminishing returns beyond 3 dozen. To give the best results, players should play sufficiently to overcome any learning barriers and be comfortable enough to provide feedback about the gameplay itself, beyond the initial learning curve.
The in-depth feedback helps surface quality issues that stem from natural player behaviours, expectations and reactions to the game, which don’t typically surface during QA testing due to their expertise and focus.
Ratings give a sense of which areas need the most work, and how much progress was made towards an improved experience by addressing issues based on the feedback. Repeated testing throughout development in general can give an overview of the progress, and applying the same method to competitor titles can set the quality bar to reach.
Repetition
If the basic interactions with the game are fun, then the question becomes if they remain fun when repeated over and over by themselves, and with an extra layer of motivation and goals from a meta-game layer.
To assess this, the idea is to have conduct a study similar to assessing fun, but extend it for as long as realistically possible - based on the participant availability, cost, duration that can be supported and options to deliver the game to participants.
- This can be done at a very small scale in the same conditions as an external playtest such as described above. This can be done initially to assess early if there are any major issues to iron out before testing a scale. If players want to quit early before the end of the study for reasons other than bugs, that is a good indicator that retention may suffer. The good news is you can capture those reasons and address them.
- Publishers may have build delivery system that grants selected users access to an early version of the game for limited periods of time, which can be used to collect actual player data at scale - from several hundred to several thousand participants. This can start yielding quantitative data that enables using telemetry to identify churn and drop out rates, which combined with feedback forms similar to those used in the playtest, can explain why players leave at certain points to reduce friction and improve retention.
- Beyond that, alpha, beta and early access options can be used to collect even larger samples of data where there starts being sufficient data for business insights teams to create predictions and test against their forecasts.