Developing an effective LTV model at the soft launch and keeping it valid further beyond.
1 like•628 views
The document discusses the development and validation of lifetime value (LTV) models in gaming, particularly during soft launches and after global launches. It emphasizes the importance of adapting LTV models based on user data, behavior, and game maturity, and suggests leveraging machine learning for enhanced predictions. Additionally, it highlights the need for regular validation and updating of models to maintain accuracy as game features evolve.
Developing an effective LTV model at the soft launch and keeping it valid further beyond.
- 1. DEVELOPING AN EFFECTIVE LTV MODEL AT SOFT LAUNCH AND KEEPING IT VALID FURTHER BEYOND Ivan Kozyev
- 2. IVAN KOZYEV Head of Analytics at Crazy Panda IVAN KOZYEV Head of Analytics at Crazy Panda
- 3. CRAZY PANDA PRODUCTS World Poker Club Casual Poker 74M registrations Stellar Age MMO Strategy 2.7M registrations The Household Social Farm 30M registrations Pirate Tales Party Battler 1.5M registrations
- 4. Math behind LTV Basic definitions LTV (also Life-Time Value, CLTV, CLV or LCV) is an amount of the profit attributed to the entire relationship with a user. Can be individual or aggregated (average) for a group of users Individual: LTV = profit from user Cohort average: LTV = total profit ÷ number of users Can be actual or predicted
- 5. Predicting LTV Game stages Soft launch Characterized by: Small number of users and limited data Unknown life-time length Uniform users (in terms of data) Suggested LTV approach: General LTV models Interpolate and extrapolate appropriate curves Be ready to apply some heuristics
- 6. Predicting LTV Game stages Some time after global launch Characterized by: Large amounts of data Settled user behaviour Big user diversity Suggested LTV approach: Specific LTV models for different use cases Combination of different approaches to LTV modelling Machine Learning as a way to increase efficiency
- 7. Predicting LTV Characterized by: Huge amounts of data Some or most of that data cannot be used for model training Challenging to keep models up-to-date with new features Suggested LTV approach: Having more than one model, working on different sets of features Regular validation of LTV models predictions The “slicing” technique Game stages Game maturity
- 8. Game stage: the soft launch Characterized by Small number of users and limited data Unknown life-time length Uniform users (in terms of data)
- 9. Game stage: the soft launch Suggested approaches Interpolate or extrapolate depending on data Choose correct metric to approximate Work around edge cases: LTV = F(t) LTV(tn+1) ≥ LTV(tn) LTV(t→∞) = LTVmax
- 10. Game stage: the soft launch Extra notes Product knowledge is crucial: Monetization limits User behavior Other options: LTV = ∫(Ret(t) * ARPU)dt LTV = ∫(Ret(t) * ARPU(t))dt
- 11. Model validation The most important step in LTV model development Always have a validation sample Your validation sample must be representative Do not overfit on validation sample: use it only once per model Forecast validation - verifying and determining the predictive power of model forecasts and predictions
- 12. Some time after global launch Improving the model Shortening the confidence interval for prediction Lowering the sample size for reliable results Reducing the amount of time/data needed for a model $9.5 ± $2.1 → $9.3 ± $0.9 890 users → 214 users 7D of data → 2D of data
- 13. Some time after global launch Improving the model Different LTV models for different country groups Different LTV models for different sources Different LTV models for different monetization types Tier 1 vs Tier 2 vs Tier 3 based on conversions Google Ads vs video networks based on optimization In-apps vs ad-based based on monetization
- 14. Some time after global launch Machine Learning approach Advantages: Disadvantages: Can solve and take into account all dependencies at once Can identify very complex relations Can give very accurate results or even per-user predictions Needs a lot of data for training Takes time and skill to develop The LTV itself as a metric is a very complex target
- 15. Some time after global launch Machine Learning approach Predict shorter periods; for example, 7-, 14- or 28-day ARPU Use along with existing LTV models Predict proxy metrics other than LTV: payers, number of payments, etc How can we use Machine Learning to build better LTV models?
- 16. Model improvement Summary Set target model metric for improvement: decrease in sample size, data needed for a prediction, or just general accuracy Try different approaches: predicting proxy things like payer conversion or classifying payers by their type Always think about how the LTV model will be used and what needs it will be covering Mind the validation
- 17. Mature game Games change with every update, while LTV models tend to become more and more complex. This complexity plus sudden changes in product features introduce a whole new challenge: keeping LTV models valid and up-to-date. Update is bad → we are losing money due to incorrect predictions Update is good → we are losing money due to not scaling our development or marketing What could possibly go wrong?
- 18. Mature game Dealing with issues Build a quick new model with rough “soft launch techniques” for quick validation Retrain Machine Learning models as soon as possible. It is quite handy to have a few models using very limited amount of data Do an A/B test, if possible Use product knowledge to measure impact on monetization and changes in user behavior
- 19. Mature game Case: Live-Ops events We started to do Live-Ops events in our game and have already done 3 or 4 of them. We want to understand their impact on LTV, but for classical approach we would need 10 times more. What can we do apart from waiting?
- 20. Mature game The “slicing” technique Slice the LTV curve into smaller periods For each slice, define a validation cohort Calculate the impact on LTV over that period Normalize those impacts and calculate the final improvement Do not forget about the novelty effect
- 21. Mature game The “slicing” technique Slice the LTV curve into smaller periods For each slice, define a validation cohort Calculate the impact on LTV over that period Normalize those impacts and calculate the final improvement Do not forget about the novelty effect
- 22. THANK YOU FOR YOUR ATTENTION! ANY QUESTIONS? IVAN KOZYEV Head of Analytics at Crazy Panda [email protected] telegram: @IvanKozyev https://www.linkedin.com/in/ivan-kozyev/