Introducing Riak and Ripple
26 likes2,843 views
The document introduces Riak and Ripple, focusing on scalability and database functionality in application development. It covers key concepts such as horizontal scaling, key-value storage, eventual consistency, and map-reduce processes, highlighting the benefits and costs of scalability. The document also presents comparisons between different database models, like relational databases, document databases, and object-oriented databases.
![Link-walking
GET /riak/bucket/key/[bucket,tag,keep]*
Wednesday, April 14, 2010](https://image.slidesharecdn.com/introducingriakandripple-100414081059-phpapp02/85/Introducing-Riak-and-Ripple-84-320.jpg)
![Link-walking
GET /riak/bucket/key/[bucket,tag,keep]*
bucket = scope to a bucket
Wednesday, April 14, 2010](https://image.slidesharecdn.com/introducingriakandripple-100414081059-phpapp02/85/Introducing-Riak-and-Ripple-85-320.jpg)
![Link-walking
GET /riak/bucket/key/[bucket,tag,keep]*
bucket = scope to a bucket
tag = scope to a tag
Wednesday, April 14, 2010](https://image.slidesharecdn.com/introducingriakandripple-100414081059-phpapp02/85/Introducing-Riak-and-Ripple-86-320.jpg)
![Link-walking
GET /riak/bucket/key/[bucket,tag,keep]*
bucket = scope to a bucket
tag = scope to a tag
keep = return results (last one always)
Wednesday, April 14, 2010](https://image.slidesharecdn.com/introducingriakandripple-100414081059-phpapp02/85/Introducing-Riak-and-Ripple-87-320.jpg)
![Link-walking
GET /riak/bucket/key/[bucket,tag,keep]*
bucket = scope to a bucket
tag = scope to a tag
keep = return results (last one always)
“_” means match anything
Wednesday, April 14, 2010](https://image.slidesharecdn.com/introducingriakandripple-100414081059-phpapp02/85/Introducing-Riak-and-Ripple-88-320.jpg)
![Link-walking
GET /riak/bucket/key/[bucket,tag,keep]*
bucket = scope to a bucket
tag = scope to a tag
keep = return results (last one always)
“_” means match anything
Example:
Wednesday, April 14, 2010](https://image.slidesharecdn.com/introducingriakandripple-100414081059-phpapp02/85/Introducing-Riak-and-Ripple-89-320.jpg)
![Link-walking
GET /riak/bucket/key/[bucket,tag,keep]*
bucket = scope to a bucket
tag = scope to a tag
keep = return results (last one always)
“_” means match anything
Example:
GET /riak/artists/TheBeatles/albums,_,1/tracks,_,1
Wednesday, April 14, 2010](https://image.slidesharecdn.com/introducingriakandripple-100414081059-phpapp02/85/Introducing-Riak-and-Ripple-90-320.jpg)
![A simple M/R job
{“inputs”: “goog”,
“query”: [{“map”:
{“language”:”javascript”,
“name”: “Riak.mapValuesJson”,
“keep”: true}]}
Wednesday, April 14, 2010](https://image.slidesharecdn.com/introducingriakandripple-100414081059-phpapp02/85/Introducing-Riak-and-Ripple-127-320.jpg)
![A simple M/R job
{“inputs”: “goog”,
“query”: [{“map”:
{“language”:”javascript”,
“name”: “Riak.mapValuesJson”,
“keep”: true}]}
Riak::MapReduce.new(c).add(‘goog’).
map(‘Riak.mapValuesJson’, :keep => true).run
Wednesday, April 14, 2010](https://image.slidesharecdn.com/introducingriakandripple-100414081059-phpapp02/85/Introducing-Riak-and-Ripple-128-320.jpg)
![Another M/R Job
{“inputs”: “stocks”,
“query”: [{“map”:{“language”:”javascript”,
“name”: “App.extractTickers”,
“arg”: “GOOG”,
“keep”: false},
{“reduce”:{“language”:”javascript,
“name”: “Riak.reduceMin”,
“keep”: true}]}
Wednesday, April 14, 2010](https://image.slidesharecdn.com/introducingriakandripple-100414081059-phpapp02/85/Introducing-Riak-and-Ripple-129-320.jpg)
![Another M/R Job
{“inputs”: “stocks”,
“query”: [{“map”:{“language”:”javascript”,
“name”: “App.extractTickers”,
“arg”: “GOOG”,
“keep”: false},
{“reduce”:{“language”:”javascript,
“name”: “Riak.reduceMin”,
“keep”: true}]}
Riak::MapReduce.new(c).add(‘stocks’).
map(‘App.extractTickers’, :arg =>”GOOG”).
reduce(“Riak.reduceMin”, :keep => true).run
Wednesday, April 14, 2010](https://image.slidesharecdn.com/introducingriakandripple-100414081059-phpapp02/85/Introducing-Riak-and-Ripple-130-320.jpg)
Introducing Riak and Ripple
- 1. Introducing Riak and Ripple Sean Cribbs Wednesday, April 14, 2010
- 2. system “whoami” Wednesday, April 14, 2010
- 3. system “whoami” BCS, three years as a musician Wednesday, April 14, 2010
- 4. system “whoami” BCS, three years as a musician Started with Ruby in early 2006 Wednesday, April 14, 2010
- 5. system “whoami” BCS, three years as a musician Started with Ruby in early 2006 Rails freelancer 2007-2010 Wednesday, April 14, 2010
- 6. system “whoami” BCS, three years as a musician Started with Ruby in early 2006 Rails freelancer 2007-2010 Radiant CMS Wednesday, April 14, 2010
- 7. system “whoami” BCS, three years as a musician Started with Ruby in early 2006 Rails freelancer 2007-2010 Radiant CMS March 2010 - Developer Advocate basho Wednesday, April 14, 2010
- 8. Wednesday, April 14, 2010
- 9. Wednesday, April 14, 2010
- 10. Rails Can’t Scale Wednesday, April 14, 2010
- 11. Rails Can’t Scale what does it mean? Wednesday, April 14, 2010
- 12. Ruby Rails Can’t Scale what does it mean? Wednesday, April 14, 2010
- 13. MySQL Ruby Rails Can’t Scale what does it mean? Wednesday, April 14, 2010
- 14. insert straw man here MySQL Ruby Rails Can’t Scale what does it mean? Wednesday, April 14, 2010
- 15. Mythbusting “scalability” Wednesday, April 14, 2010
- 16. Mythbusting “scalability” Scalability is not a yes/no question Wednesday, April 14, 2010
- 17. Mythbusting “scalability” Scalability is not a yes/no question It’s a ratio of benefit to cost Wednesday, April 14, 2010
- 18. Mythbusting “scalability” Scalability is not a yes/no question It’s a ratio of benefit to cost Benefits: low latency, throughput, uptime, concurrency, reliability Wednesday, April 14, 2010
- 19. Mythbusting “scalability” Scalability is not a yes/no question It’s a ratio of benefit to cost Benefits: low latency, throughput, uptime, concurrency, reliability Costs: CPU, RAM, disk, bandwidth, power, hw/sw Wednesday, April 14, 2010
- 20. Mythbusting “scalability” Scalability is not a yes/no question It’s a ratio of benefit to cost Benefits: low latency, throughput, uptime, concurrency, reliability Costs: CPU, RAM, disk, bandwidth, power, hw/sw scalability = bang for your buck (ROI) Wednesday, April 14, 2010
- 21. Scaling goes both ways Wednesday, April 14, 2010
- 22. Scaling goes both ways Up - high traffic, “Big Data”, multi-datacenter Wednesday, April 14, 2010
- 23. Scaling goes both ways Up - high traffic, “Big Data”, multi-datacenter Down - laptop, set-top, mobile Wednesday, April 14, 2010
- 24. How do you scale? Wednesday, April 14, 2010
- 25. How do you scale? Vertically - get bigger, expensive machines Wednesday, April 14, 2010
- 26. How do you scale? Vertically - get bigger, expensive machines Horizontally - get more commodity machines Wednesday, April 14, 2010
- 27. What is Riak? (the horizontal scaling bits) Wednesday, April 14, 2010
- 28. What is Riak? (the horizontal scaling bits) Based on Amazon’s Dynamo (2007) Wednesday, April 14, 2010
- 29. What is Riak? (the horizontal scaling bits) Based on Amazon’s Dynamo (2007) key-value storage Wednesday, April 14, 2010
- 30. What is Riak? (the horizontal scaling bits) Based on Amazon’s Dynamo (2007) key-value storage masterless, peer-to-peer replication Wednesday, April 14, 2010
- 31. What is Riak? (the horizontal scaling bits) Based on Amazon’s Dynamo (2007) key-value storage masterless, peer-to-peer replication consistent hashing Wednesday, April 14, 2010
- 32. What is Riak? (the horizontal scaling bits) Based on Amazon’s Dynamo (2007) key-value storage masterless, peer-to-peer replication consistent hashing eventual consistency Wednesday, April 14, 2010
- 33. What is Riak? (the horizontal scaling bits) Based on Amazon’s Dynamo (2007) key-value storage masterless, peer-to-peer replication consistent hashing eventual consistency failover - quorums, hinted handoff Wednesday, April 14, 2010
- 34. “O/RM is the Vietnam of Computer Science.” -- Ted Neward, 2004/6 Wednesday, April 14, 2010
- 35. “O/RM is the Vietnam of Computer Science.” -- Ted Neward, 2004/6 slippery slope Wednesday, April 14, 2010
- 36. “O/RM is the Vietnam of Computer Science.” -- Ted Neward, 2004/6 slippery slope diminishing returns Wednesday, April 14, 2010
- 37. Impedance mismatch Wednesday, April 14, 2010
- 38. Impedance mismatch Relational = predicate logic, truth statements Wednesday, April 14, 2010
- 39. Impedance mismatch Relational = predicate logic, truth statements Object = identity, state, behavior, encapsulation Wednesday, April 14, 2010
- 40. Pick your poison Wednesday, April 14, 2010
- 41. Pick your poison Make the objects fit the database Wednesday, April 14, 2010
- 42. Pick your poison Make the objects fit the database Active Record Wednesday, April 14, 2010
- 43. Pick your poison Make the objects fit the database Active Record Make the database fit the objects Wednesday, April 14, 2010
- 44. Pick your poison Make the objects fit the database Active Record Make the database fit the objects OODBMS, Graph DBs Wednesday, April 14, 2010
- 45. Middle ground Wednesday, April 14, 2010
- 46. Middle ground Document databases (identity and state) Wednesday, April 14, 2010
- 47. Middle ground Document databases (identity and state) No JOINs - denormalized / composed Wednesday, April 14, 2010
- 48. Middle ground Document databases (identity and state) No JOINs - denormalized / composed Loosely structured Wednesday, April 14, 2010
- 49. Middle ground Document databases (identity and state) No JOINs - denormalized / composed Loosely structured Friendly transport formats, e.g. JSON Wednesday, April 14, 2010
- 50. What is Riak? (the document database bits) Wednesday, April 14, 2010
- 51. What is Riak? (the document database bits) Store your objects as JSON (or any format) Wednesday, April 14, 2010
- 52. What is Riak? (the document database bits) Store your objects as JSON (or any format) Link between objects (like hypertext) Wednesday, April 14, 2010
- 53. What is Riak? (the document database bits) Store your objects as JSON (or any format) Link between objects (like hypertext) No SQL - query with Javascript Map-Reduce Wednesday, April 14, 2010
- 54. “DevOps” - agile infrastructure Wednesday, April 14, 2010
- 55. “DevOps” - agile infrastructure programming your infrastructure Wednesday, April 14, 2010
- 56. “DevOps” - agile infrastructure programming your infrastructure rapid “cloud” deployments Wednesday, April 14, 2010
- 57. Befriend your sysadmin Wednesday, April 14, 2010
- 58. Befriend your sysadmin Integrate with existing infrastructure Wednesday, April 14, 2010
- 59. Befriend your sysadmin Integrate with existing infrastructure Reduce the amount of hands-on work Wednesday, April 14, 2010
- 60. Befriend your sysadmin Integrate with existing infrastructure Reduce the amount of hands-on work Have a growth plan Wednesday, April 14, 2010
- 61. What is Riak? (the ops-friendly bits) Wednesday, April 14, 2010
- 62. What is Riak? (the ops-friendly bits) Web-shaped storage Wednesday, April 14, 2010
- 63. What is Riak? (the ops-friendly bits) Web-shaped storage Store data in its original format Wednesday, April 14, 2010
- 64. What is Riak? (the ops-friendly bits) Web-shaped storage Store data in its original format It’s just HTTP - same techniques apply Wednesday, April 14, 2010
- 65. What is Riak? (the ops-friendly bits) Web-shaped storage Store data in its original format It’s just HTTP - same techniques apply load balancing, proxy caches, round-robin DNS Wednesday, April 14, 2010
- 66. What is Riak? (the ops-friendly bits) Web-shaped storage Store data in its original format It’s just HTTP - same techniques apply load balancing, proxy caches, round-robin DNS No node is special - grow horizontally Wednesday, April 14, 2010
- 67. What is Riak? (the ops-friendly bits) Web-shaped storage Store data in its original format It’s just HTTP - same techniques apply load balancing, proxy caches, round-robin DNS No node is special - grow horizontally Get some sleep, fix it in the morning Wednesday, April 14, 2010
- 68. To review, Riak is... Wednesday, April 14, 2010
- 69. To review, Riak is... Magical Horizontally- Scaling Unicorns Wednesday, April 14, 2010
- 70. To review, Riak is... Magical Horizontally- Scaling Unicorns Application-Friendly Rainbows in the Cloud Wednesday, April 14, 2010
- 71. To review, Riak is... Magical Horizontally- Scaling Unicorns Application-Friendly Rainbows in the Cloud Data-Shredding MapReduce Ninjas Wednesday, April 14, 2010
- 72. http://downloads.basho.com/riak/ hg clone http://hg.basho.com/riak/ Wednesday, April 14, 2010
- 73. gem install ripple git clone git://github.com/seancribbs/ripple.git Wednesday, April 14, 2010
- 74. CRUD in Riak Wednesday, April 14, 2010
- 75. CRUD in Riak PUT /riak/bucket/key (C,U) (also POST) Wednesday, April 14, 2010
- 76. CRUD in Riak PUT /riak/bucket/key (C,U) (also POST) GET /riak/bucket/key (R) Wednesday, April 14, 2010
- 77. CRUD in Riak PUT /riak/bucket/key (C,U) (also POST) GET /riak/bucket/key (R) DELETE /riak/bucket/key (D) Wednesday, April 14, 2010
- 78. CRUD Demo Wednesday, April 14, 2010
- 79. Links Wednesday, April 14, 2010
- 80. Links One-way, qualified “pointers” to other objects Wednesday, April 14, 2010
- 81. Links One-way, qualified “pointers” to other objects “Link” header in HTTP Wednesday, April 14, 2010
- 82. Links One-way, qualified “pointers” to other objects “Link” header in HTTP </riak/bucket/key>; riaktag=”tag” Wednesday, April 14, 2010
- 83. Link-walking Wednesday, April 14, 2010
- 84. Link-walking GET /riak/bucket/key/[bucket,tag,keep]* Wednesday, April 14, 2010
- 85. Link-walking GET /riak/bucket/key/[bucket,tag,keep]* bucket = scope to a bucket Wednesday, April 14, 2010
- 86. Link-walking GET /riak/bucket/key/[bucket,tag,keep]* bucket = scope to a bucket tag = scope to a tag Wednesday, April 14, 2010
- 87. Link-walking GET /riak/bucket/key/[bucket,tag,keep]* bucket = scope to a bucket tag = scope to a tag keep = return results (last one always) Wednesday, April 14, 2010
- 88. Link-walking GET /riak/bucket/key/[bucket,tag,keep]* bucket = scope to a bucket tag = scope to a tag keep = return results (last one always) “_” means match anything Wednesday, April 14, 2010
- 89. Link-walking GET /riak/bucket/key/[bucket,tag,keep]* bucket = scope to a bucket tag = scope to a tag keep = return results (last one always) “_” means match anything Example: Wednesday, April 14, 2010
- 90. Link-walking GET /riak/bucket/key/[bucket,tag,keep]* bucket = scope to a bucket tag = scope to a tag keep = return results (last one always) “_” means match anything Example: GET /riak/artists/TheBeatles/albums,_,1/tracks,_,1 Wednesday, April 14, 2010
- 91. Links Demo Wednesday, April 14, 2010
- 92. Riak Map-Reduce I believe I did, Bob. Wednesday, April 14, 2010
- 93. Riak Map-Reduce Wednesday, April 14, 2010
- 94. Riak Map-Reduce Based on Google’s Map-Reduce idea Wednesday, April 14, 2010
- 95. Riak Map-Reduce Based on Google’s Map-Reduce idea Some similarities to Hadoop and CouchDB Wednesday, April 14, 2010
- 96. Riak Map-Reduce Based on Google’s Map-Reduce idea Some similarities to Hadoop and CouchDB High data-locality Wednesday, April 14, 2010
- 97. Riak Map-Reduce Based on Google’s Map-Reduce idea Some similarities to Hadoop and CouchDB High data-locality Phases can be Javascript or Erlang Wednesday, April 14, 2010
- 98. Map-Reduce Terminology Wednesday, April 14, 2010
- 99. Map-Reduce Terminology Job: a query, composed of inputs and phases Wednesday, April 14, 2010
- 100. Map-Reduce Terminology Job: a query, composed of inputs and phases Phase: a single map or reduce function application Wednesday, April 14, 2010
- 101. Map-Reduce Terminology Job: a query, composed of inputs and phases Phase: a single map or reduce function application Map: phase applied to each item - filter, transform Wednesday, April 14, 2010
- 102. Map-Reduce Terminology Job: a query, composed of inputs and phases Phase: a single map or reduce function application Map: phase applied to each item - filter, transform Reduce: phase applied to the collection - collate, aggregate, compute Wednesday, April 14, 2010
- 103. Map Phases Wednesday, April 14, 2010
- 104. Map Phases Inputs are bucket-key pairs (with optional key-specific data) Wednesday, April 14, 2010
- 105. Map Phases Inputs are bucket-key pairs (with optional key-specific data) Must return a list Wednesday, April 14, 2010
- 106. Map Phases Inputs are bucket-key pairs (with optional key-specific data) Must return a list Parallel results are aggregated Wednesday, April 14, 2010
- 107. Map Built-ins Wednesday, April 14, 2010
- 108. Map Built-ins Riak.mapValues Wednesday, April 14, 2010
- 109. Map Built-ins Riak.mapValues Riak.mapValuesJson Wednesday, April 14, 2010
- 110. Reduce Phases Wednesday, April 14, 2010
- 111. Reduce Phases Performed on a single node Wednesday, April 14, 2010
- 112. Reduce Phases Performed on a single node Two processes per reduce phase increase parallelism Wednesday, April 14, 2010
- 113. Reduce Phases Performed on a single node Two processes per reduce phase increase parallelism Must return a list Wednesday, April 14, 2010
- 114. Reduce Built-ins Wednesday, April 14, 2010
- 115. Reduce Built-ins Riak.reduceMin Wednesday, April 14, 2010
- 116. Reduce Built-ins Riak.reduceMin Riak.reduceMax Wednesday, April 14, 2010
- 117. Reduce Built-ins Riak.reduceMin Riak.reduceMax Riak.reduceSort Wednesday, April 14, 2010
- 118. Build a M/R Job Wednesday, April 14, 2010
- 119. Build a M/R Job Specify inputs Wednesday, April 14, 2010
- 120. Build a M/R Job Specify inputs bucket/key, bucket/key/key-specific-arg, bucket Wednesday, April 14, 2010
- 121. Build a M/R Job Specify inputs bucket/key, bucket/key/key-specific-arg, bucket Specify phases Wednesday, April 14, 2010
- 122. Build a M/R Job Specify inputs bucket/key, bucket/key/key-specific-arg, bucket Specify phases Each can receive a static argument Wednesday, April 14, 2010
- 123. Build a M/R Job Specify inputs bucket/key, bucket/key/key-specific-arg, bucket Specify phases Each can receive a static argument Each can return intermediate results Wednesday, April 14, 2010
- 124. Map-Reduce on HTTP Wednesday, April 14, 2010
- 125. Map-Reduce on HTTP Job is a JSON object Wednesday, April 14, 2010
- 126. Map-Reduce on HTTP Job is a JSON object POST /mapred Wednesday, April 14, 2010
- 127. A simple M/R job {“inputs”: “goog”, “query”: [{“map”: {“language”:”javascript”, “name”: “Riak.mapValuesJson”, “keep”: true}]} Wednesday, April 14, 2010
- 128. A simple M/R job {“inputs”: “goog”, “query”: [{“map”: {“language”:”javascript”, “name”: “Riak.mapValuesJson”, “keep”: true}]} Riak::MapReduce.new(c).add(‘goog’). map(‘Riak.mapValuesJson’, :keep => true).run Wednesday, April 14, 2010
- 129. Another M/R Job {“inputs”: “stocks”, “query”: [{“map”:{“language”:”javascript”, “name”: “App.extractTickers”, “arg”: “GOOG”, “keep”: false}, {“reduce”:{“language”:”javascript, “name”: “Riak.reduceMin”, “keep”: true}]} Wednesday, April 14, 2010
- 130. Another M/R Job {“inputs”: “stocks”, “query”: [{“map”:{“language”:”javascript”, “name”: “App.extractTickers”, “arg”: “GOOG”, “keep”: false}, {“reduce”:{“language”:”javascript, “name”: “Riak.reduceMin”, “keep”: true}]} Riak::MapReduce.new(c).add(‘stocks’). map(‘App.extractTickers’, :arg =>”GOOG”). reduce(“Riak.reduceMin”, :keep => true).run Wednesday, April 14, 2010
- 131. Map-Reduce Demo Wednesday, April 14, 2010
- 132. Ripple Wednesday, April 14, 2010
- 133. Ripple Document-style persistence (like MongoMapper) Wednesday, April 14, 2010
- 134. Ripple Document-style persistence (like MongoMapper) ActiveModel niceties Wednesday, April 14, 2010
- 135. Ripple Document-style persistence (like MongoMapper) ActiveModel niceties Callbacks Wednesday, April 14, 2010
- 136. Ripple Document-style persistence (like MongoMapper) ActiveModel niceties Callbacks Validations Wednesday, April 14, 2010
- 137. Ripple Document-style persistence (like MongoMapper) ActiveModel niceties Callbacks Validations ActionPack compatibility Wednesday, April 14, 2010
- 138. Ripple TODO Wednesday, April 14, 2010
- 139. Ripple TODO Associations Wednesday, April 14, 2010
- 140. Ripple TODO Associations Indexes Wednesday, April 14, 2010
- 141. Ripple TODO Associations Indexes Use a property/method as the key Wednesday, April 14, 2010
- 142. Ripple TODO Associations Indexes Use a property/method as the key Dynamic finders Wednesday, April 14, 2010
- 143. Ripple TODO Associations Indexes Use a property/method as the key Dynamic finders Session/cache stores Wednesday, April 14, 2010
- 144. Ripple Demo Wednesday, April 14, 2010
- 145. [email protected] @seancribbs “seancribbs” on Freenode IRC #riak Questions? http://wiki.basho.com [email protected] Wednesday, April 14, 2010