Creating a MongoDB Based Logging System in a Webservice Heavy Environment
6 likes3,571 views
The document discusses the creation of a MongoDB-based logging system tailored for a high-traffic web service environment. It highlights the advantages of MongoDB, such as schema flexibility, scalability, and real-time analytics capabilities while detailing deployment strategies and log entry schemas. Additionally, it emphasizes the importance of data sharing, anomaly detection, and proactive support in maintaining the system's efficiency.
Creating a MongoDB Based Logging System in a Webservice Heavy Environment
- 1. Creating MongoDB Based Logging System in Webservice Heavy Environment Arkadiusz Goralski @MongoDB London 2013 piątek, 29 marca 13
- 2. Web service Heavy Environment Web Application Web Server 1 Web Server 2 ... Web Server n piątek, 29 marca 13
- 3. Web service Heavy Environment Web Application Web Server 1 Web Server 2 ... Web Server n Gaming Providers Sportsbook Slots Live Mini betting games Casino Poker Live Account dealer Management piątek, 29 marca 13
- 4. Web service Heavy Environment Web Application Web Server 1 Web Server 2 ... Web Server n Gaming Providers Payment Providers Sportsbook Slots Skrill Ukash Ecocard Live Mini Neteller Paysafe Instadebit betting games Bank Casino Poker Visa Ecocard transfers Live Account Mastercard Envoy Webmoney dealer Management piątek, 29 marca 13
- 5. Web service Heavy Environment Web Application Web Server 1 Web Server 2 ... Web Server n Gaming Providers Payment Providers Other Subsystems Bonus Common Sportsbook Slots Skrill Ukash Ecocard engine Wallet Live Mini Neteller Paysafe Instadebit Price feeds Mobile API betting games Bank Casino Poker Visa Ecocard Anti fraud Geo IP transfers Live Account Mastercard Envoy Webmoney dealer Management piątek, 29 marca 13
- 6. Different dialects SOAP JSON RPC Web REST JSON XML RPC Application REST XML HTTP Calls piątek, 29 marca 13
- 7. How to keep sanity? • why the deposit failed for user ... ? • user ... claims he didn’t place this bet? • can you give me a number of users that ... ? • can you tell me what user ... was doing on ... ? • why user ... couldn’t log in? • why customer ... couldn’t open an account? piątek, 29 marca 13
- 8. Why MongoDB is a good fit • schemaless • fire and forget • scalable and easy to replicate • capped collections & tailable cursor • centralized • easy remote access piątek, 29 marca 13
- 9. MongoDB deployment Web Application Web Server 1 Web Server 2 ... Web Server n piątek, 29 marca 13
- 10. MongoDB deployment Web Application Web Server 1 Web Server 2 ... Web Server n Replica Set Payment Providers MongoDB MongoDB MongoDB MongoDB (host 1) (host 2) (host 3) (hidden) piątek, 29 marca 13
- 11. MongoDB deployment Web Application Web Server 1 Web Server 2 ... Web Server n Replica Set Payment Providers MongoDB MongoDB MongoDB MongoDB (host 1) (host 2) (host 3) (hidden) Log Viewer RealPayment Providers time Simple Simple Real time log viewer queries reports alerts Secondary preferred piątek, 29 marca 13
- 12. MongoDB deployment Web Application Web Server 1 Web Server 2 ... Web Server n Replica Set Payment Providers MongoDB MongoDB MongoDB MongoDB (host 1) (host 2) (host 3) (hidden) Log Viewer Data mining RealPayment Providers time Simple Simple Real time Payment Providers Heavy Analytics log viewer queries reports alerts queries Secondary preferred Hidden node only piątek, 29 marca 13
- 13. Log entry schema piątek, 29 marca 13
- 14. Log entry schema Prepare web service request piątek, 29 marca 13
- 15. Log entry schema Prepare web service request JSON-ize Send it piątek, 29 marca 13
- 16. Log entry schema Prepare web service request JSON-ize Send it JSON-ize Get the result piątek, 29 marca 13
- 17. Log entry schema Prepare web service request JSON-ize Send it JSON-ize Get the result Was is successful piątek, 29 marca 13
- 18. Log entry schema Prepare web service request JSON-ize Send it JSON-ize Get the result Was is successful Proceed with Save it execution piątek, 29 marca 13
- 19. So you're logging data to MongoDB, now what? • all data is searchable (including web service specific data) • you can use the aggregation framework • trace user sessions across different web services • just sit and watch the logs piątek, 29 marca 13
- 20. Detect anomalies in the data • realtime issue analysis • subsystem failures • fraud • traffic anomalies piątek, 29 marca 13
- 21. Data mining • create a replica for analytics • analyze customer behaviour • spot fraudulent activity • create reports • proactive support • deployment analytics • police enquiries piątek, 29 marca 13
- 22. Share the logs • don’t be a human interface to MongoDB • the more you share the more time is saved, your time • just a simple query interface is enough • give separate node to marketing people piątek, 29 marca 13
- 23. Push notifications When something goes really bad Alert Scenarios Email Tailable cursor Node.js Severity Text MongoDB app rating message Call people, it's serious! piątek, 29 marca 13
- 24. Open Source Alternatives • GrayLog (uses MongoDB) • LogStash (MongoDB output plugins) • Fluent (MongoDB storage) piątek, 29 marca 13
- 25. Thank you Have fun with MongoDB piątek, 29 marca 13