Augmenting MySQL with NoSQL options - Data Lifecycles
Download as PPTX, PDF0 likes505 views
The document discusses the different stages in a data lifecycle - transient, short to medium term, analytics, and archival. It outlines example workloads that fall into each stage. The key technologies that can be used at each stage are also presented, including Redis, MongoDB, Cassandra and Hadoop. The document provides guidance on how to fit the different technologies together based on requirements, available talent, and licensing. It emphasizes allowing experts to manage the technologies while focusing on building application features.
Augmenting MySQL with NoSQL options - Data Lifecycles
- 1. Augmenting MySQL with Big Data and NoSQL options The Data Lifecycle
- 2. Lead DBA @ Data Services / ObjectRocket by Rackspace 15+ years in data and information systems, ranging from application develop, data architecture, system design, and more. Primary focus – Helping business focus on using data not managing and storing it. David Murphy @davidmurphy_data www.linkedin.com/in/davidbmurphy/
- 3. True genius resides in the capacity for evaluation of uncertain, hazardous, and conflicting information. - Winston Churchill EVERYONE’S GOT TO HAVE A GREAT DATA QUOTE RIGHT?!
- 4. Lifecycle, say what Where are the technologies Why One isn't enough How to fit them together Outcomes We want you to leave here understanding:
- 5. This is NOT… a deep dive on any technology a comprehensive list a roadmap discussion the end of the journey What We Will Cover
- 6. What We’ll Cover Concepts What are the lifecycle stages How to classify your workloads Terminology Actions What technologies are there When to use them Fitting them together Why is this better
- 7. What are the lifecycle stages Transient • Sessions • Logins • Shop Cart Short - Medium • Feeds • E-Commerce • Video Game Stats Analytics • Reports • Summary Data • Dash boards Archival • Cold Storage • Seldom Access • Governances L i f e C y c l e
- 8. What are the lifecycle stages Transient • Sessions • Logins • Shop Cart Short - Medium • Feeds • E-Commerce • Video Game Stats Analytics • Reports • Summary Data • Dash boards Archival • Cold Storage • Seldom Access • Governances L i f e C y c l e
- 9. What are the lifecycle stages Transient • Sessions • Logins • Shop Cart Short - Medium • Feeds • E-Commerce • Video Game Stats Analytics • Reports • Summary Data • Dash boards Archival • Cold Storage • Seldom Access • Governances L i f e C y c l e
- 10. What are the lifecycle stages Transient • Sessions • Logins • Shop Cart Short - Medium • Feeds • E-Commerce • Video Game Stats Analytics • Reports • Summary Data • Dash boards Archival • Cold Storage • Seldom Access • Governances L i f e C y c l e
- 11. Updated frequently Ultra fast retrieval If missing is OK IS IS NOT Workloads - Transient Rich Query-able Durable Point of truth
- 12. Some to many updates Rich Query-able Durable + Point of Truth IS IS NOT Workloads - Short to Medium Built for short term 99% Write 1% Reads Heavy Aggregations
- 13. Heavy Aggregations More Latency Massive Parallelized IS IS NOT Workloads - Analytics Rich Query-able Good for many updates Point of truth
- 14. High / Extreme Latency Ultra Cheap Built for Retention IS IS NOT Workloads - Archival Rich Query-able Updateable Short Term Storage
- 22. Terminology: Documents Columns Rows Backups Partition s Scaling Geo & DR The dreaded polyglot persistence
- 23. Transient • Memcache • CouchBase • Redis • SQLite Medium • MySQL • Maria • PostgreSQL • Mongo DB • XtraCluster • NDB Analytics • Hadoop • InfoBright • Cassandra • Teradata Archival •Hadoop + External •Hadoop Snapshots •Cassandra using S3 Technologies
- 24. Fitting it together • What is the fewest technologies we can use • What will for new requests • Do I have plans to handle each stage of data? • If not can the technologies do a decent job on the odd case? • Have talent now? Can I get a service or person easily?
- 25. Fitting it together - tools Build a matrix with • Features needs ( Transactions, Persistent , Geo,…) • Importance ( 1- 5) • Current or Attainable Talent ( 1 -5 ) • Does its Licensing work for this project ( 0 or 1) (Features * Importance * Talent * License) = Combined Rank
- 26. Klout’s great example, but it’s polyglot!
- 28. How it should be…
- 29. How to scale – focus on what you know You scale your app by letting someone else • Build the hardware • Know the Ops side for the technology • Make the technologies pass data as its ages vs duplicating the data • Be the experts • You just focus on the features of your app and make $$$
- 30. Questions? WE ARE HIRING! ( DBA, DevOps, and more) https://rackertalent.com https://www.objectrocket.com/careers Twitter: @dmurphy_data @rackspace @objectrocket Email: [email protected] Github: https://github.com/dbmurphy SlideDeck: https://github.com/dbmurphy/presentations
Editor's Notes
- #12: Atomic: Everything in a transaction succeeds or the entire transaction is rolled back. Consistent: A transaction cannot leave the database in an inconsistent state. Isolated: Transactions cannot interfere with each other. Durable: Completed transactions persist, even when servers restart etc.
- #13: Atomic: Everything in a transaction succeeds or the entire transaction is rolled back. Consistent: A transaction cannot leave the database in an inconsistent state. Isolated: Transactions cannot interfere with each other. Durable: Completed transactions persist, even when servers restart etc.
- #14: Atomic: Everything in a transaction succeeds or the entire transaction is rolled back. Consistent: A transaction cannot leave the database in an inconsistent state. Isolated: Transactions cannot interfere with each other. Durable: Completed transactions persist, even when servers restart etc.
- #15: Atomic: Everything in a transaction succeeds or the entire transaction is rolled back. Consistent: A transaction cannot leave the database in an inconsistent state. Isolated: Transactions cannot interfere with each other. Durable: Completed transactions persist, even when servers restart etc.