An architecture for evaluating recommender systems in real world scenarios
7 likes•1,415 views
The thesis presents a framework for evaluating recommender systems in real-world scenarios, identifying the challenges and the need for empirical studies to assess factors like accuracy and user preferences. It proposes building an evaluation platform with defined use cases, focusing on user interactions and roles within the system. The conclusion highlights the implementation of a scalable platform already attracting interest from companies and the potential for ongoing research and user interaction development.
An architecture for evaluating recommender systems in real world scenarios
- 1. An architecture for evaluating recommender systems in real world scenarios Master Thesis Manuel Blechschmidt 2011 Supervisor Prof. Dr. Christoph Meinel M.Sc. Rehab Alnemr
- 2. 2 Christmas 2009 ... Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
- 3. Agenda 3 ■ Motivation and Current Research ■ Solution □ Use Cases & Requirements □ Wireframes □ Implementation ■ Related Work ■ Conclusion Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
- 4. 4 Motivation and Current Research Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
- 5. 5 Experiment Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
- 6. Choice 6 Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
- 7. Motivation 7 ■ The choice overload problem is well known in psychology □ It is necessary to do a preselection for the customer ■ Recommender systems are already very successful to decrease the choice overload problem in some domains □ Product-to-Product Recommendation → Amazon.com □ Movie Recommendation → NetFlix ■ Algorithms already produce great results ■ Already research in soft factores like: Diversity, Serendepity, Trust, Explanations → not a lot of emprical studies how these influences customers → no cross domain data sets → not a lot of business intereset integration Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
- 8. Current Algorithms and Developments 8 ■ Matrix Factorization (best RMSE 0.855 for NetFlix Dataset) □ SVD □ SVD++ R.M.Bell, Y. Koren, and C. Volinsky □ TimeSVD++ R.M.Bell, Y. Koren, and C. Volinsky ■ Collaborative Filtering □ Item based □ User based ■ Performance gains □ ALS1 István Pilászy, Dávid Zibriczky, Domonkos Tikk ■ Some of the algorithms already implemented in a distributed manner Mahout, MyMedia Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
- 9. Empirical Studies 9 ■ Current empirical studies (RecSys 2010) □ Understanding Choice Overload in Recommender Systems 174 participants □ Eye-Tracking Product Recommendersʼ Usage 18 participants □ Recommender Algorithms in Activity Motivating Games 180 participants □ Group-Based Recipe Recommendations: Analysis of Data Aggregation Strategies 170 participants □ A User-Centric Evaluation Framework of Recommender Systems 807 participants □ Information Overload and Usage of Recommendations 466 participants □ ... Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
- 10. Current Problems 10 ■ Not a lot of big empirical studies how recommender quality influence consumer behavior especially □ Acurarcy □ Familiarity □ Serendipity □ Attractiveness □ Enjoyability □ Novelty □ Diversity □ Context Compatibility ■ Taken from A User-Centric Evaluation Framework of Recommender Systems Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
- 11. Evaluating in real world 11 ■ Most of the academia persons do not know enough persons which are willing to test the algorithms. Therefore the following things are difficult: □ Evaluating User Interfaces □ Evaluating Maintenance □ Evaluating Scalibility □ Evaluating Performance Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
- 12. 12 Solution Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
- 13. Master Thesis 13 ■ Building and maintaining an evaluation platform for recommender systems in real world scenarios ■ Maintenance challenges in running a recommender system ■ Empirical study about user behavior □ Brand loyalty □ Pricing □ Timing Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
- 14. Solution: Use Cases 14 Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
- 15. Roles 15 ■ 5 Roles with different point of views and different interests and goals ■ The roles are describeded with description and goals ■ Example: □ Provider □ A provider is a legal personality which has as primary goal to optimize a particular objective. In an economic context this is most of the time a business goal like raise profit or optimize conversion rates. … □ Goals: – optimizing an objective – get forecasts – ensure privacy of his data Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
- 16. Use Cases and Requirements 16 ■ Use Cases and Requirements are described based on IEEE 830 ■ A use case is defined by: □ Id □ Name □ Summary □ Roles □ Preconditions □ Postconditions □ Wireframes □ More optional attributes Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
- 17. Use Case Example C1 Design User Interaction 17 ■ Id: C1 Name: Design User Interaction ■ Summary: When a user interaction should be run like a newsletter or an item-to-item recommendation the consultant has to do the following steps: … ■ Roles: Consultant ■ Preconditions □ User is logged in □ User has the Consultant role □ At least one user interaction is implemented □ At least one provider is associated with the consultant □ The provider has the necessary data which is needed for the user interaction ■ Postconditions □ Provider received an email for approving the user interaction □ User interaction is created in the system Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
- 18. C1 Design User Interaction 18 Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
- 19. C1 Design User Interaction 19 Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
- 20. C1 Design User Interaction 20 Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
- 21. C1 Design User Interaction 21 Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
- 22. Implemented Architecture 22 Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
- 23. Logical Modularization 23 Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
- 24. Survey Module Entities 24 Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
- 25. Survey Module Services 25 Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
- 26. 26 Demo Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
- 27. Implemented User Interaction chocStore 27 Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
- 28. 28 Related Work Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
- 29. Related Work: Competition 29 ■ NetFlix Grand Prize 2006 – 2009 □ 1.000.000 $ to make CineMatch 10% better □ Lots research of papers ■ KDD Cup 2011 Recommending Music Items based on the Yahoo! Music Dataset ■ ECML/PKDD’2007 DISCOVERY CHALLENGE □ User 1 User’s behaviour prediction Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
- 30. Related Work: Platforms 30 ■ GroupLens Research of University of Minnesota □ MovieLens 1997 http://movielens.umn.edu/ ■ RichRelevance RecLab 2011 □ RecLab: A System For eCommerce Recommender Research with Real Data, Context and Feedback ■ Knowledge and Data Engineering Group of Uni Kassel □ 2006 BibSonomy is a system for sharing bookmarks and lists of literature. Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
- 31. Further Research 31 ■ Implement more user interactions □ Item-to-Item recommender ■ Prove that the platform is scalable ■ Run the platform for a long time and evaluate usage ■ Integrate more companies ■ Promote plattform in science and economics ■ Take part at research projects together with companies Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
- 32. 32 Conclusion Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
- 33. Conclusion 33 ■ An enterprise ready platform was defined and implemented ■ Companies already applied for using ■ One example user interaction was implemented □ chocStore ■ Statistical test can be applied to the data to give scientific results Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
- 34. Questions 34 Questions? Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
- 35. Backup: What is a recommender? 35 Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
- 36. 36 Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
- 37. 37 Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
- 38. 38 Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
- 39. 39 Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11