Understanding and building big data Architectures - NoSQL
18 likes1,718 views
1. The document discusses various technologies for building big data architectures, including NoSQL databases, distributed file systems, and data partitioning techniques. 2. Key-value stores, document databases, and graph databases are introduced as alternatives to relational databases for large, unstructured data. 3. The document also covers approaches for scaling databases horizontally, such as sharding, replication, and partitioning data across multiple servers.
Understanding and building big data Architectures - NoSQL
- 1. Understanding and Building Big Data Architectures Ran.ga.na.than B, ThoughtWorks @ran_than Part 1 - Storage and NoSQL
- 2. Agenda
- 4. Basics Story of Sir William Osler, University of Oxford
- 7. Latency ● Hibernia Express ● 3,000-mile fiber-optic ● across the Atlantic Ocean to connect London to New York ● goal for 5ms latency ● To be used by Financial Institutes for trading Src: http://shop.oreilly.com/product/0636920028048.do
- 9. HDD Speed ❏ ~122MB per sec ❏ 1TB in 2hr 22 minutes ❏ SSDs are 2-3 times faster Multiple disks reading parallel: ❏ With 100 HDDs, it takes 6 minutes
- 10. Data and Compute together
- 13. Clusters 1. MultiNode: e.g: Hadoop, each node has some responsibility. 2. Peer-to-Peer: e.g: Cassandra, all nodes are equal
- 15. Expectations from Data Systems Non-functional parts of the application.
- 16. Expectations from Architecture ❏ Reliability ❏ Scalability ❏ Maintainability ❏ High Availability ❏ Fault Tolerance ❏ Security ❏ Compliance ❏ Compatibility
- 18. Components ❏ Storage ❏ Cache ❏ Search ❏ Stream Processing ❏ Batch Processing ❏ Data Exchange Protocols
- 19. Components ❏ Storage ❏ Cache ❏ Search ❏ Stream Processing ❏ Batch Processing ❏ Data Exchange Protocols
- 20. NoSQL Databases “Database Admins walked into a NoSQL bar. A little while later they walked out because they couldn’t find a table.”
- 21. Why NoSQLs? ❏ Scalability ❏ Cost ❏ Flexibility ❏ Availability ❏ Migrations
- 22. CAP theorem Strong Consistency, High Availability, and Partition-Tolerance Img-src:http://image.slidesharecdn.com/cap-131117230434-phpapp02/95/dynamo-and-bigtable-in-light-of-the-cap-theorem-12-638.jpg?cb=1384729712
- 23. CA
- 24. CP “when your business requirements dictate atomic reads and writes” Src: http://robertgreiner.com/2014/08/cap-theorem-revisited/
- 25. AP “when the system needs to continue to function in spite of external errors” Src: http://robertgreiner.com/2014/08/cap-theorem-revisited/
- 26. CAP story
- 27. Activity Design ticket booking with scenarios of CA, CP, AP
- 28. ACID Atomic, Consistent, Isolated, and Durable
- 29. BASE ● Basically Available: If a single node fails, part of the data won't be available, but the entire data layer stays operational. ● Soft state: Soft state means data that is not persisted on the disk, yet in case of failure it could be possible to restore it. ● Eventually consistent: indicates that the system will become consistent over time, given that the system doesn't receive input during that time.
- 30. Key-Value DBs Memcached, Redis, Riak, Voldemort, ...
- 32. Implementation 1 - Arrays ● Only int as key ● Values are of same type
- 33. Implementation 2 - Associative Arrays Key Value user1 Mike user2 Mary user3 Nina On hotspace
- 34. Simple Storage Design - put key value - will add content to file in one line - get key - will grep for key and return the value from the file What are the problems with this? Activity How can we improve this?
- 35. Simple Storage Design - Add in memory index, with key and value as byte offset. What are the problems with this? Activity How can we improve this?
- 36. Simple Storage Design - Segments - Compaction What are the problems with this? Activity How can we improve this?
- 37. Simple Storage Design - Sorted Key-Value - Sparse index - SSTable What are the problems with this? Activity How can we improve this?
- 38. Simple Storage Design - Memtable - Segments as SSTable Img-src: https://www.igvita.com/2012/02/06/sstable-and-log-structured-storage-leveldb/
- 39. Simple Storage Design - Overall - Writes into RedBlack or AVL trees in memory - memtable => faster writes - When memtable is 64MB, write to disk as SSTable and clean memtable - First read from memtable and most recent segments in- memory sparse index (SSTable) => faster reads - Run a merging and compaction process in the background => lesser storage and faster
- 40. Assignment Let us do the e-commerce design with only key value pair
- 41. Document DBs CouchDB, MongoDB, MarkLogic, DocumentDB, OrientDB, ...
- 43. Query - MongoDB Img-src: http://bicortex.com/introduction-to-mongodb-nosql-database-for-sql-developers-part-3/
- 44. Query - CouchDB // emit the first letter of each pokemon's name var myMapReduceFun = { map: function (doc) { emit(doc.name.charAt(0)); }, reduce: '_count' }; // count the pokemon whose names start with 'P' pouch.query(myMapReduceFun, { key: 'P', reduce: true, group: true }).then(function (result) { // handle result }).catch(function (err) { // handle errors });
- 45. What’s cool? ● Flexible schema. ● Embedded docs come in one read. ●
- 46. Not so cool ● Familiarity ● Needs more space. ● Doesn’t speak SQL.
- 47. Assignment Let us do the e-commerce design with only document database
- 48. Graph DBs Neo4j, TitanDB, OrientDB, ...
- 49. Relationships in relational DBs means joining
- 50. Why? ❏ Modelling and storing relationships in RDBMS is complicated ❏ Performance degrades with number and levels of relationships. ❏ Query complexity grows ❏ Adding new type requires schema redesign
- 51. With neo4j, you can traverse 4M+ relationships per second and core
- 52. Img-src: http://blog.octo.com/wp-content/uploads/2012/07/RequestInSQL.png Squeezing to table structure
- 53. Img-src: http://blog.octo.com/wp-content/uploads/2012/07/RequestInGraph.png Closer to white board model
- 54. Graph consists of ❏ Vertices ❏ Edges
- 55. Cypher Query - Neo4j
- 56. SPARQL - RDF data model
- 57. Column oriented DBs Apache HBase, Cassandra, BigTable, ...
- 63. When is this better? ❏ Huge number of columns, with queries on few columns ❏ Aggregation ❏ Column level update ❏ Column data is uniform; so better compression
- 64. Time Series Data Measurement and Time of measurement done repeatedly Img src: https://www.safaribooksonline.com/library/view/time-series-databases/9781491920909/images/tsdn_0103.png.jpg
- 65. Why - Time Series Data ● Trends
- 66. When - Time Series Data ● Huge amount of data ● Mostly query based on time ● Stock exchange ● Sensor data. E.g: Trucks ● Cell towers for usage patterns
- 67. Time Series Data ● IoT ● Logs
- 68. Replication This is useful when you have a ncie photo or color-black as a background. On this slide only, you can put your elements behind a master element.
- 69. Journals
- 70. Master-Follower
- 75. Masterless
- 77. Sharding Which bucket has your data?
- 78. Master Slave - Master for writes and real-time reads - Slaves for reads
- 79. Table partitioning - By rows or columns for parallelizing reads
- 80. Feature specific DBs - Different DB servers for specific features of application - Can this scale?
- 81. Federated Tables A Federated Table is a table which points to a table in another database instance (mostly on an other server). It can be seen as a view to this remote database table. - Administration overhead - Security - Access over network - Okay for reporting/analytical tasks
- 82. Federated Tables A Federated Table is a table which points to a table in another database instance (mostly on an other server). It can be seen as a view to this remote database table. - Administration overhead - Security - Access over network - Okay for reporting/analytical tasks
- 83. Range based Split data based on range condition. Eg: zip code, region. - Non-uniform distribution
- 84. Hash based Take hash of key and modulo operation, put the data in the server based on reminder value. - Uniform distribution - Range queries may take time
- 85. Co-ordinators - Take request, if key is in the request, talk to correct shard - Co-ordinate across shards to give the result back - Monitor health - Take care of rebalancing - Can be a random node, which will complete the task - Set of co-ordinators
- 86. Take care, while sharding ● Balance your shards, with proper shard key ● Choose correct number of shards. E.g: 12 ● Give time for rebalancing. In case of increasing capacity of server, add nodes faster, and give time move your shards. ● Shard on denormalized data. ● Try to have shard key as part of your queries.
- 87. Where am I?
- 88. THANK YOU For questions or suggestions: Ran.ga.na.than B @ran_than