Counters with Riak on Amazon EC2 at Hackover
Download as KEY, PDF0 likes1,353 views
The document discusses using distributed counters with Riak on Amazon EC2. It introduces Riak as a distributed key-value database focused on availability, fault tolerance, simplicity and scalability. It describes using Riak features like consistent hashing, replication and automatic load balancing to implement a REST API for counters with eventual consistency. The demo shows implementing and accessing counters across local machines and EC2 regions to demonstrate the architecture.
Counters with Riak on Amazon EC2 at Hackover
- 1. Distributed Counters with Riak on EC2 Hackover 2012 - hackathon.ro Andrei Savu - [email protected] Twitter: @andreisavu
- 2. Me • Co-Founder of Axemblr.com • Co-Organiser of Bucharest JUG (bjug.ro) • OSS contributor (Apache Whirr, jclouds) • Connect with me on LinkedIn
- 4. ... with Riak on AWS (for fun and profit - real-time analytics)
- 5. Riak • Distributed Database (Key-Value) • Focused on: Availability, Fault-Tolerance, Operation Simplicity, Scalability • Each node is the same (no master) • Erlang (OTP)
- 7. Riak - More Features • Consistent Hashing • Replication (including cross DC) • Automatic load-balancing • Hinted Handoff • Client-side conflict resolution
- 8. Riak Control
- 9. Amazon Cloud • Infrastructure as a Service (IaaS) provider • Services I use: • EC2 (compute) • ELB (load balancing) • Route53 (smart dns) • CloudWatch (monitoring)
- 10. Service API • GET /counters/<id> • POST /counters/<id> <value>
- 11. Eventual Consistency • ... and counters • Client side conflict resolution • counter => {last: 0, step: 0}
- 12. Basic Algorithm • {last: 1, step: 0} • +2 => {last: 1+0, step: 2} • Partition 1 • Partition 2 • +3 => {last: 1+2, step: 3} • +4 => {last: 1+2, step: 4} • Fixed • Fixed • 1+(2 + 3 + 4) = 6 • read: {last: 3, step: 3 + 4}
- 13. Architecture US API ELB Riak w/ cross DC + replication Route53 UE API
- 14. Demo Time Local Machine and on Amazon EC2 (two regions)
- 15. Demo Resources • riak-ec2-launcher https://github.com/andreisavu/riak-ec2- cluster-launcher • riak-dt https://github.com/basho/riak_dt
- 16. Resources #1 • PoC for Convergent Data Types https://github.com/basho/riak_dt • Study on Convergent Data Types http://hal.inria.fr/docs/00/55/55/88/PDF/ techreport.pdf • Gist: https://gist.github.com/882055
- 17. Resources #2 • statebox - state "monad" for automated conflict resolution https://github.com/mochi/statebox • Convergent Replicated Data Types https://github.com/aphyr/meangirls • Strong Eventual Consistency and Conflict-free Replicated Data Types http://research.microsoft.com/apps/video/ default.aspx?id=153540
- 18. Resources #3 • Cassandra Counters http://wiki.apache.org/cassandra/Counters • Convergent Replicated Data Types - SRC Fringe https://speakerdeck.com/bryce/convergent- replicated-data-types-src-fringe • http://amattn.com/2012/09/07/ riaks_two_contentions_and_crdts.html
- 20. Thanks! Questions? Andrei Savu - [email protected] Twitter: @andreisavu