Riak Use Cases : Dissecting The Solutions To Hard Problems
3 likes2,940 views
This document summarizes a presentation on using Riak to solve hard distributed systems problems. It discusses Riak's key features and tradeoffs in consistency and data model. It then provides examples of how companies like Comcast, Yammer, and Github use Riak for high availability and low latency applications like user metadata storage, notifications, and as a document store. The presentation emphasizes that choosing the right database requires understanding your problem and tradeoffs in consistency, availability and data model. It also highlights how Riak and Erlang help enable high availability, low latency and operational simplicity through features like Bitcask storage, fault tolerance and lightweight threading.
Riak Use Cases : Dissecting The Solutions To Hard Problems
- 1. Riak Use Cases: Dissecting the Solutions to Hard Problems Andy Gross <@argv0> Principal Architect, Basho Technologies QCon SF 2011 Wednesday, November 16, 11
- 2. Riak Dynamo-inspired key value database with full text search, map/reduce, secondary indices, link traversal, commit hooks, HTTP and binary interfaces Written in Erlang (and C/C++) Open Source, Apache 2 licensed Enterprise features (multi-datacenter replication) and support available from Basho Wednesday, November 16, 11
- 3. Choosing a NoSQL Database At small scale, everything works. NoSQL DBs trade off traditional features to better support new and emerging use cases Knowledge of the underlying system is essential NoSQL marketing is... “confusing” Wednesday, November 16, 11
- 4. Tradeoffs If you’re evaluating Mongo vs. Riak, or Couch vs. Cassandra, you don’t understand your problem By choosing Riak, you’ve already made tradeoffs: Sacrificing consistency for availability in failure scenarios A rich data/query model for a simple, scalable one Wednesday, November 16, 11
- 5. Distributed Systems: Desirable Properties Highly Available Fault Tolerant Low Latency Ops-Friendly Scalable Predictable Wednesday, November 16, 11
- 6. Medical Records Store Danish Health Authorities Implemented by Trifork A/S Won the Digitization Prize as one of the “best government IT projects in Denmark” Stores medical prescription history for all Danish Citizens, replicated in 2 data centers. Accessed from pharmacies, hospital, mobile devices Replicated in multiple data centers Wednesday, November 16, 11
- 7. User/Metadata Store Comcast User profile storage for xfinityTV mobile application Storage of metadata on content providers, and content licensing info Strict latency requirements Wednesday, November 16, 11
- 8. Notification Service Yammer Wednesday, November 16, 11
- 9. Session Store Mochi Media First Basho Customer (late 2009) Every hit to a Mochi web property = 1 read, maybe one write to Riak Unavailability, high latency = lost ad revenue Wednesday, November 16, 11
- 10. Document Store Github Pages / Git.io Riak as a web server for Github Pages (in staging) Webmachine is an awesome HTTP server! Git.io URL shortener Wednesday, November 16, 11
- 11. Distributed Systems: Desirable Properties High Availability Fault Tolerance Low Latency Ops-Friendliness Horizontal Scalability Predictability Wednesday, November 16, 11
- 12. High Availability Failure to accept a read/write results in: lost revenue lost users Availability and latency are intertwined Wednesday, November 16, 11
- 13. Low Latency Sometimes late answer is useless or wrong Users perceive slow sites as unavailable SLA violations SOA approaches magnify SLA failures Wednesday, November 16, 11
- 14. Fault Tolerance Everything fails Especially in the cloud When a host/disk/network fails, what is the impact on Availability Latency Operations staff Wednesday, November 16, 11
- 15. Predictability “It’s a piece of plumbing; it has never been a root cause of any of our problems.” Coda Hale, Yammer Wednesday, November 16, 11
- 16. Cost Wednesday, November 16, 11
- 17. Operational Costs Sound familiar? “we chose a bad shard key...” “the failover script did not run as expected...” “the root cause was traced to a configuration error...” Staying up all night fighting your database does not make you a hero. Wednesday, November 16, 11
- 18. High Availability: Erlang Ericcson AXD-301: 99.9999999% uptime (31ms/year) Shared-nothing, immutable, message-passing, functional, concurrent Distributed systems primitives in core language OTP (Open Telecom Platform) Wednesday, November 16, 11
- 19. High Availability: Riak Core Dynamo abstracted: distributed systems toolkit Exhaustively tested In production use at AOL, Yahoo, others Insulates local storage and client API code from the hard problems Wednesday, November 16, 11
- 20. Low Latency: Bitcask Low Latency: All reads = hash lookup + 1 seek After the append completes, an in-memory structure called a ”keydir” is updated. A keydir is simply a hash table that maps every key in a Bitcask to a fixed-size structure giving the file, offset, and size of the most recently written entry for that key. When a write occurs, the keydir is atomically updated with the location of the newest data. The old data is still present on disk, but any new reads will use the latest version available in the keydir. As we’ll see later, the Tradeoff: Index must fit in memory merge process will eventually remove the old value. Reading a value is simple, and doesn’t ever require more than a single disk seek. We look up the key in our keydir, and from there we read the data using the file id, position, and size that are returned from that lookup. In many cases, the operating system’s filesystem read-ahead cache makes this a much faster operation than would be otherwise expected. Wednesday, November 16, 11
- 21. Wednesday, November 16, 11
- 22. Low Latency: Erlang VM Erlang VM was designed for soft-realtime apps Preemptively scheduled lightweight threads GC is per-thread, not stop-the-world Sophisticated scheduler + message passing = effective use of multicore machines. Wednesday, November 16, 11
- 23. Wednesday, November 16, 11