O'Reilly Webinar: Simplicity Scales - Big Data
Download as PPTX, PDF•0 likes•610 views
The document discusses distributed database systems and properties of the Riak database. It defines distributed systems and discusses key aspects like availability, fault tolerance, and latency. It explains Riak's masterless architecture and how it provides high availability and scalability through horizontal scaling on commodity servers. The document also covers consistency models and how Riak allows tuning availability and consistency based on use cases.
O'Reilly Webinar: Simplicity Scales - Big Data
- 1. SIMPLICITY SCALES Big data application management & operations 1
- 2. WHOAMI? Tyler Hannan Director, Technical Marketing [email protected] @tylerhannan
- 3. SQL
- 4. SQL NoSQL
- 6. CHANGE IN ARCHITECTURAL DESIGN App App App App Virtualization Server App Aggregation Server Server Server Server SMALL APPS BIG SERVERS ONE LOCATION BIG APPS COMMODITY SERVERS MANY LOCATIONS
- 7. In 2014, 20% of enterprise data projects add distributed processes into production
- 8. THE BENEFITS OF RIAK Riak is an operationally friendly database that is: • Fault-tolerant • Highly-available • Scalable • Self-healing
- 9. THE PROPERTIES OF A DISTRIBUTED DB Riak is a multi-model database that is: • Open Source & Commercial • Distributed • Masterless • Eventually Consistent
- 11. This is NOT about Riak. This is about design decisions in distributed systems.
- 12. This IS about Riak. And learning from Basho’s architectural decisions.
- 13. DISTRIBUTED SYSTEMS – A DEFINITION “A distributed system is a software system in which components located on network computers communicate and coordinate their actions by passing messages. The components interact with each other in order to achieve a common goal.” --Wikipedia
- 14. DISTRIBUTED SYSTEMS – A DEFINITION “A distributed system is one in which the failure of a computer you didn't even know existed can render your own computer unusable” --Leslie Lamport
- 15. DISTRIBUTED SYSTEMS – A DEFINITION “Everything works at small scale. Understand failure modalities to understand your realities.” --Tyler Hannan
- 16. THINKING DISTRIBUTED What we consider, when we think distributed: • Availability • Fault Tolerance • Latency
- 17. UPTIME IS A POOR METRIC… Availability
- 18. AVAILABILITY “…widespread underestimation of the specific difficulties of size seems one of the major underlying causes of current software failure.” --Wikipedia
- 19. THE CAP THEOREM
- 20. HARVEST AND YIELD Harvest • a fraction • data available / complete data Yield • a probability • queries completed / queries requested Failure will cause known linear reduction to one of these
- 21. UNDERSTANDING “CONSISTENCY H A R V E S T YIELD Queries Issued Queries Offered Data Available Total Dataset
- 22. HARVEST AND YIELD Traditional design demands 100% HARVEST… but success of modern applications is often measured in YIELD.
- 23. HARVEST AND YIELD: USE CASE
- 24. HARVEST AND YIELD: USE CASE
- 25. RELATIONAL AVAILABILITY primary replica replica replica coordination Write/ Read
- 26. RELATIONAL AVAILABILITY primary replica replica coordination X Write/ Read
- 27. RIAK AVAILABILITY Riak has a masterless architecture in which every node in a cluster is capable of serving read and write requests.
- 28. Each 1/Nth the data 1/Nth the performance = RIAK AVAILABILITY
- 30. RELATIONAL SCALABILITY A - K L - P Q - Z Designed to scale vertically Cost of vertical scaling Sharding
- 31. ADD CAPACITY AS NEEDED Node 3 Node 4 Node 5 Node 0 Node 1 Node 2 Designed for Horizontal Scale Deployed on commodity Hardware Consistent Hashing
- 32. PERFECTION IS UNATTAINABLE… FAILURES WILL HAPPEN Fault Tolerance
- 33. FAULT TOLERANCE How many hosts/replicas do you need to survive “F” failures? • F + 1 – fundamental minimum • 2F + 1 – a majority are alive • 3F + 1 – Byzantine Fault Tolerance
- 34. NAÏVE HASHING NH(Ka) = 0 NH(Kb) = 1 NH(Kc) = 2 NH(Kd) = 0 … Node # = HASH(KEY) % NUM_NODES
- 35. NAÏVE HASHING Node 0 Node 1 Node 2 Ka Kd Kg Kj Km Kp Kb Ke Kh Kk Kn Kq Kc Kf Ki Kl Ko Kr
- 36. NAÏVE HASHING Node 0 Node 1 Node 2 Ka Ke Ki Km Kq Kb Kf Kj Kn Kr Kc Kg Kk Ko Node 4 Kd Kh Kl Kp
- 37. NAÏVE HASHING • K = # of Keys • NN = # of Nodes As NN grows factor essentially becomes 1, thus ALL keys move K * (NN – 1) / NN => K
- 38. CONSISTENT HASHING • # of Partitions remains CONSTANT • Key always maps to the SAME Partition • Node owns Partitions • Partitions contain keys • Extra Level of Indirection Partition # = HASH(KEY) % Partitions
- 39. CONSISTENT HASHING P4 P7 P2 P5 P8 P3 P6 P9P1 Node 0 Node 1 Node 2 Ka Kd Kg Kj Km Kp Kb Ke Kh Kk Kn Kq Kc Kf Ki Kl Ko Kr
- 40. CONSISTENT HASHING P4 P7 P2 P5P8 P3 P6 P9 P1 Node 0 Node 1 Node 2 KaKd Kg KjKm Kp Kb KeKh Kk KnKq Kc Kf Ki Kl Ko Kr Node 4
- 41. CONSISTENT HASHING • K = # of Keys • NN = # of Nodes • Q = # of Partitions As K grows NN becomes constant, thus K/Q keys move NN * K/Q => K/Q
- 42. RIAK AVAILABILITY Node 0 Node 1 Node 2
- 43. BRIEF MILLISECONDS PHOTONS FLYING THROUGH GLASS TIME STOPS FOR NO ONE Latency
- 44. UNDERSTANDING LATENCY 299,792,458 meters/second in a vacuum
- 46. LATENCY: GOOGLE’S BIG TABLE 95th percentile: 24 ms 99.9th percentile: 994 ms
- 47. REDUNDANCY REDUCES LATENCY IF response > 10 ms THEN send 2nd request 5% increase in total requests 99.9th percentile latency = 50 ms
- 48. UNDERSTANDING LATENCY Overall latency is determined by latency of the SLOWEST machine. Get data close to your users.
- 49. MULTI-CLUSTER REPLICATION Replicate data across datacenters or across the world
- 50. UNDERSTANDING PERFORMANCE You get fast read and write performance What does this mean?
- 51. THE PLAN, THE ENVIRONMENT • How do we measure performance? • What do we measure when we measure performance? • basho_bench • Google Cloud
- 53. NODE FAILURE
- 54. THINKING DISTRIBUTED What we consider, when we think distributed: • Availability • Fault Tolerance • Latency
- 55. DISTRIBUTED SYSTEMS – A DEFINITION “Everything works at small scale. Understand failure modalities to understand your realities.” --Tyler Hannan
- 56. CV CV NoSQL Database Unstructured Data No pre-defined Schema Small and Large Data Sets on Commodity HW Many Models: K/V, document store, graph Variety of Query Methods RELATIONAL & NOSQL What’s the difference? Relational Database Structured Data Defined Schema Tables with Rows/Columns Indexed w/ Table Joins SQL
- 57. THE EVOLUTION OF NOSQL Unstructured Data Platforms Multi-Model Solutions Point Solutions
- 58. 42% of database decision makers admit they struggle to manage the NoSQL solutions deployed in their environments” Riak Spark COMPLEX TECHNOLOGY STACK
- 59. Simplify the Complexity Ensure High Availability Scale Horizontally
- 62. Riak KV Client Basho Redis Proxy Client Application Redis Redis Riak KV Riak KV Service Manager Spark Spark Spark Connector Spark Client Riak KV EnsembleBasho Data Platform Read Write Read Write Read Redis Services Read Spark Services Leader Election Query Write Read/Solr MANAGING COMPLEXITY AT SCALE
- 64. MILLIONS OF RECORDS Information requested and amended more than 2.6 BILLION times a year 42 MILLION Summary Care Records 1.3 BILLION prescription messages
- 65. BILLIONS OF MOBILE DEVICES 10 BILLION data transactions a day – 150,000 a second Forecasting 2.8 BILLION locations around the world Generates 4GB OF DATA every second
- 67. learn more at ricon.io
Editor's Notes
- #18: “IF THE SYSTEM IS ‘DOWN’ AND NO ONE MAKES A REQUEST, IS IT REALLY DOWN?” ~ ME
- #20: CAP theorem - Dr. Eric Brewer, Prof UC Berkeley, VP infrastructure google, Basho Board Consistency Always return last written value “eventual consistency” seems scary. Is not – DNS Master/Slave RDMBS log shipping Caching layer Availability Get a response even if portions of system are down Partition Tolerance Not a trade-off google “coda hale partition tolerance”
- #22: Harvest a fraction data available / complete data Yield a probability queries completed / queries requested
- #26: Transaction log shipping hot/hot backup Datacenter failure vs. node failure Master/Replica Architecture Assumption of Transactional Consistency What happens under failure conditions?
- #28: Requests are routed to Nodes using standard load balancing techniques Under the cover keys are actually stored as as combination of the bucket name and user assigned key value. Potentially add proxy/client visual
- #29: Riak’s uniform distribution and equal allocation of vnodes to machines Allows you to think of each machine being responsible for: 1/Nth data & 1/Nth performance
- #31: Hot spots Unevenly spread data and request patterns Resharding is operationally intensive and often manual Designed for Vertical scale Cost Considerations a key element of vertical scaling Sharding
- #34: An approach to fault tolerance is to look at hashing algorithms to distribute data across the F instances. This will be a “ring free” presentation…
- #45: Latency is largely a function of the speed of light, which is 299,792,458 meters/second in vacuum. This would equate to a latency of 3.33 microseconds for every kilometer of path length. The index of refraction of most fibre optic cables is about 1.5, meaning that light travels about 1.5 times as fast in a vacuum as it does in the cable. This works out to about 4.9 microseconds of latency for every kilometer. In shorter metro networks, the latency performance rises a bit more due to building risers and cross-connects and can bring the latency as high as 5 microseconds per kilometer.
- #46: Our understanding of availability translates to the computational systems we build. Some systems use physical temporal clocks and timestamps (even Riak). Time is continuous but we can only represent discrete moments in a computer. Granularity of the discretization is dependent on probable frequency of events. We tend to choose a millisecond which is like plucking a drop of water from a gushing river. To make matters worse, we know for certain that information does not transmit instantaneously, the speed of light is finite. The distance across this nebula is probably many light-years. To put it on something closer to human scale: the distance between San Francisco and New York at the speed of light is 14 milliseconds, which is a long time in a computer! Now add in a global footprint. The latency, round trip, from SF to Amsterdam can be as much as 200ms.
- #47: In-memory BigTable lookups –data replicated in two in-memory tables –issue requests for 1000 keys spread across 100 tablets –measure elapsed time until data for last key arrives
- #48: In-memory BigTable lookups –data replicated in two in-memory tables –issue requests for 1000 keys spread across 100 tablets –measure elapsed time until data for last key arrives
- #51: And…more importantly…when?
- #52: They built a tool called basho_bench that’s designed for testing out distributed, key-value stores. It allows users to specify options such as their key distribution, values, etc. All of our tests used binary representations as integers for keys, and pseudorandom, un-compressible data for values. In our testing, load generation was spread across three nodes, with one node also acting as a test coordinator. There were 256 virtual clients spread across said 3 nodes. All benchmarks were run with {mode, max}, a worst case scenario for load generation. On connection termination, the client was configured to immediately reconnect, and retry the op. The benchmark used the protocol buffers client, which maintains a long-lived connection to the Riak cluster. In order to enable us to reason about an elastic cluster, we added a new driver operation, reconnect. In every one of our tests, 1/10000 operations per client driver was going to result in reconnecting a new node. Each of our instances were of the type n1-standard-16 deployed in the us-central1-f zone. The instances themselves were running the image ‘backports-debian-7-wheezy-v20140904’ All Riak nodes were deployed in the same network. Network load balancing was used to distribute the traffic amongst the Riak nodes, with health checking doing a /ping on the HTTP interface with default healthcheck intervals.
- #53: One of the benefits of distributed, Dynamo databases is that cluster expansion is relatively easy. Often times, unexpected load on traditional databases leaves customers, and engineers in a crux, where the entire system is unavailable. Upgrading, and expanding hardware to said databases is typically a tenous, multi-hour offere operation. In this test, we show the effects of taking a fully loaded cluster of 6 nodes, and growing it by 3 nodes, 180 seconds after we started the test. As you can see in this test, 5 minutes after cluster expansion was initiated, an increased throughput of 27% was realized. On the other hand, update latency took slightly longer to converge, as that was based on rebalancing the Riak_api, and protocol buffers coordination load across the rest of the cluster. During the entire rebalancing period, median latency slightly increased, but handoffs can be throttled, and fewer nodes can be added in a single handoff claimaint handoff quanta, to make this process have fewer side effects.
- #54: Node failure is a normal part of life for operators today. This is really where Riak shines. We started the test, and 90 seconds afterwards, we killed 1 of 9 nodes by prompting a forced, immediate shutdown of Riak through a pkill. After 420 seconds, an operator initiated force-remove was activated, and 2100 seconds later, the cluster had converged. There was only one operation that actually returned in an error, unfortunately that didn’t show up on the graph. The rest of the operations immediately retried, and were successful. Realistically, convergence could have taken up to 2 seconds, given that’s how long Google’s default healthcheck window takes. Node failure is simply a non-event in a properly built, and planned Riak cluster.
- #62: Reduce complexity with integrated NoSQL databases, caching, in-memory analytics, and search components Enhance high availability and fault tolerance across components Integrate real-time analytics with Apache Spark and Riak KV Increase application performance with integrated Redis caching and Riak KV Optimize search with Apache Solr and Riak KV integration
- #64: Xfinity Run app on iOS or Android and program television remotely User Profile - Stored as JSON document Metadata – do you have rights to record? Non-trivial drop in calls to support center ~80%