Slides: Polyglot Persistence for the MongoDB, MySQL & PostgreSQL DBA
Download as PPTX, PDF0 likes2,044 views
The document discusses polyglot persistence, which involves utilizing multiple specialized data storage systems instead of a single database to address complex data management needs. It outlines the advantages, challenges, and examples of using different databases for various applications while emphasizing the importance of selecting the right tool for specific tasks. Additionally, it includes a project management lifecycle and encourages the use of tools like ClusterControl for managing polyglot environments.
Slides: Polyglot Persistence for the MongoDB, MySQL & PostgreSQL DBA
- 1. Confidential Polyglot Persistence December 22nd, 2015 Art van Scheppingen Severalnines [email protected]
- 2. Confidential Logistics Webinar is recorded Replay available soon Feel free to ask questions at any time Use your control panel to contact us Or email us as well: [email protected] Copyright Severalnines AB 2
- 3. Confidential Agenda What is Polyglot Persistence? Example project Is Polyglot Persistence good or bad? Leverage ClusterControl for Polyglot Persistence (demo) Q&A 3 Copyright Severalnines AB
- 4. Confidential What is Polyglot Persistence? Copyright Severalnines AB 4
- 5. Confidential Polyglot Persistence Using multiple specialized persistent stores rather than one single general-purpose database 5 Copyright 2013 Severalnines AB
- 6. Confidential Where does the term come from? The way we work is changing Enterprise applications are becoming more complex Separate (devops/agile) teams Ownership of applications (Micro)services Everyone has their preference Various programming languages Various storage systems Copyright Severalnines AB 6
- 7. Confidential Where does the term come from? Monoglot Programming Only one programming language allowed Readability All code is in the same language Support One platform to support Knowledge Everybody is an expert Is there a jack-of-all-trades language? Copyright Severalnines AB 7
- 8. Confidential Monoglot Programming Does a carpenter only use just one tool? Copyright Severalnines AB 8
- 9. Confidential Monoglot Programming Carpenters use a broad variety of tools Copyright Severalnines AB 9
- 10. Confidential Polyglot Programming Polyglot Programming Use programming languages for what they are good at Flexibility Use Java for a secure API Use Scala for real time stream processing Use Python for text analysis Tie everything together using AngularJS Knowledge Everybody is expert at one or more languages Copyright Severalnines AB 10
- 11. Confidential Polyglot Programming Copyright Severalnines AB 11 User Website using micro services Profile API Billing API Recommendations API login microservice Java profile details microservice Python billing microservice Go recommendations microservice Scala
- 12. Confidential Monoglot Persistence Copyright Severalnines AB 12 User Website using micro services Profile API Billing API Recommendations API login microservice Java profile details microservice Python billing microservice Go recommendations microservice Scala MySQL
- 13. Confidential Data storage landscape changes Relational data stores (RDBMS) Key-Value data stores Columnar data stores Document data stores Graph data stores Copyright Severalnines AB 13
- 14. Confidential Data stores overview Software RDMBS MySQL, PostgreSQL, Oracle Key-Value Redis, Riak Columnar Cassandra, Hbase, InfiniDB Document MongoDB, Couchbase Graph Neo4J, Titan Copyright Severalnines AB 14
- 15. Confidential Data stores overview Software AWS Google RDMBS MySQL, PostgreSQL, Oracle RDS, Aurora CloudSQL Key-Value Redis, Riak DynamoDB Datastore Columnar Cassandra, Hbase, InfiniDB Redshift Big Query Document MongoDB, Couchbase SimpleDB Bigtable Graph Neo4J, Titan Titan + DynamoDB - Copyright Severalnines AB 15
- 16. Confidential Even Hadoop becomes a polyglot Copyright Severalnines AB 16
- 17. Confidential Polyglot Persistence Complex problems require different storage systems Use the right tool for the job, for example Use PostgreSQL for financial data Use MySQL for website contents Use MongoDB for user profiles Use Cassandra for real time streams Use Neo4J for recommendation analysis Copyright Severalnines AB 17
- 18. Confidential Use the right tool for the right job Document storage: MongoDB Copyright Severalnines AB 18 application + mongos MongoDB replica set shard1 objectsshard 1 { _id: <objectid1>, username: myuser, phone: 023 443 2234, email: [email protected] } { _id: <objectid2>, username: anotheruser, email: [email protected], age: 23, } MongoDB replica set shard2 replication objectsshard2
- 19. Confidential Use the right tool for the right job Columnar storage: Cassandra Copyright Severalnines AB 19
- 20. Confidential Use the right tool for the right job Graph storage: Neo4J Copyright Severalnines AB 20
- 21. Confidential Polyglot Persistence Copyright Severalnines AB 21 User Website using micro services Profile API Billing API Recommendations API login microservice Java profile details microservice Python billing microservice Go recommendations microservice Scala PostgreSQL Hadoop RedisMongoDBMySQL
- 23. Confidential Project management lifecycle Copyright Severalnines AB 23 Initiating phase Planning phase Executing phase Closing phase Requirements Analysis Design Develop Test Implement Handover to Ops Systems decomission
- 24. Confidential Project management lifecycle Copyright Severalnines AB 24 Initiating phase Planning phase Executing phase Closing phase Requirements Analysis Design Develop Test Implement Handover to Ops Systems decomission
- 25. Confidential Project management lifecycle Copyright Severalnines AB 25 Initiating phase Planning phase Executing phase Closing phase Requirements Analysis Design Develop Test Implement Handover to Ops Systems decomission
- 26. Confidential Development Requirements are gathered Create a recommendation system Storage must be (blazingly) fast Document must contain user details Data must be kept until the next visit Choose the right tool for the right job Store data as a document Use MongoDB Development Copyright Severalnines AB 26
- 27. Confidential But what happens after development? Delivery Ops sets up the application on the infrastructure Monitoring is added to see if the application works If everything is fine the product will be handed over What’s next? Systems monitoring Backups Scaling Software updates BI Copyright Severalnines AB 27
- 28. Confidential The ops perspective Copyright Severalnines AB 28 Initiating phase Planning phase Executing phase Closing phase Development Handover to Ops Involvement Maintenance: keep systems healthy Systems decomission
- 29. Confidential Systems monitoring Supported data stores so far: MySQL Memcached How do we monitor MongoDB? Integration with Nagios / Zabbix / Shinken Integration with Graphite / OpenTSDB / InfluxDB What should we monitor? Key performance metrics SQL vs Mongo queries Copyright Severalnines AB 29
- 30. Confidential Backups Supported backups so far Xtrabackup streams How do we backup MongoDB? What should we backup from MongoDB? How do we restore from a backup? How do we provide a backup for development? Copyright Severalnines AB 30
- 31. Confidential Scaling out We are data hoarders and like to keep everything Scaling options in MySQL Read-bound workloads: add more slaves Write-bound workloads: sharding or NDB Cluster Scaling in MongoDB? When should we scale out? How do we add a new shard? How do we redistribute between shards? How do we add a new read slave? Copyright Severalnines AB 31
- 32. Confidential Software updates When do we need to update with MySQL? Once we hit a bug? Every major release? Every minor release? When do we need to update with MongoDB? How do we know the impact? Copyright Severalnines AB 32
- 33. Confidential Business Intelligence BI with MySQL? Export full database from a slave Allow schemas/tables to be queried for specific tables Most BI tools can connect to MySQL BI with MongoDB? Native MongoDB analytics tools Write your own exports BI connector (since version 3.2) Copyright Severalnines AB 33
- 34. Confidential A new project emerges Copyright Severalnines AB 34 Initiating phase Planning phase Executing phase Closing phase Requirements Analysis Design Develop Test Implement Handover to Ops Systems decomission
- 35. Confidential With new requirements Requirements are gathered Allow users to view their earnings Data must always be 100% reliable Storage must be ACID compliant Choose the right tool for the right job Store data relational Use PostgreSQL Copyright Severalnines AB 35
- 36. Confidential Is Polyglot Persistence good or bad? Copyright Severalnines AB 36
- 37. Confidential The drawbacks More specific (domain) knowledge is needed Additional tooling may be necessary Need more hands to maintain the systems Copyright Severalnines AB 37
- 38. Confidential The benefits More flexibility to your company Driven and dedicated devops who take ownership Use the right tool for the right job Copyright Severalnines AB 38
- 39. Confidential The challenges Deploy various data stores Handle HA and failovers Scaling and sharding Interpret graphs and monitoring Schedule backups and handle recovery Handle updates/upgrades Copyright Severalnines AB 39
- 40. Confidential Polyglot Persistence is here to stay Variety of data storage needs keeps growing Polyglots in the cloud Better be prepared for it Copyright Severalnines AB 40
- 41. Confidential Leverage ClusterControl for Polyglot Persistence (demo) Copyright Severalnines AB 41
- 43. Confidential Thank You! ClusterControl www.severalnines.com/product/clustercontrol ClusterControl – Getting Started www.severalnines.com/getting-started Polyglot Persistence meetups http://goo.gl/64Ga5z Severalnines Blog www.severalnines.com/blog Contact: [email protected] 43