Building a Business on Hadoop, HBase, and Open Source Distributed Computing
94 likes42,204 views
The document discusses the building of a business focused on open-source distributed computing, emphasizing the importance of scalability and the transition from traditional RDBMS to more flexible and efficient data handling solutions. It outlines the challenges and sacrifices associated with various data processing methods, including Hadoop and NoSQL technologies like HBase. The goal is to create a platform capable of analyzing any data with linear costs while addressing issues of latency and structure.
Building a Business on Hadoop, HBase, and Open Source Distributed Computing
- 1. Building a Business on Open Source Distributed Computing company: www.visibletechnologies.com blog: www.roadtofailure.com twitter: @lusciouspear
- 2. Social Media and Scaling
- 3. Social Media and Scaling •Scalability Matters Now.
- 4. Social Media and Scaling •Scalability Matters Now. •SM produces large, complex data
- 5. Social Media and Scaling •Scalability Matters Now. •SM produces large, complex data •Anyone can collect the web
- 6. Social Media and Scaling •Scalability Matters Now. •SM produces large, complex data •Anyone can collect the web •Make a Twitter in a few days
- 7. Social Media and Scaling •Scalability Matters Now. •SM produces large, complex data •Anyone can collect the web •Make a Twitter in a few days •Easy to get TBs of data
- 8. Social Media and Scaling •Scalability Matters Now. •SM produces large, complex data •Anyone can collect the web •Make a Twitter in a few days •Easy to get TBs of data •Big Data enabling new fields for companies
- 10. What Visible Does •BI and Brand Management on Social Media
- 11. What Visible Does •BI and Brand Management on Social Media •Listen, Monitor, Engage
- 16. Old Product: RDBMS •A few MSSQL servers on boxes
- 17. Old Product: RDBMS •A few MSSQL servers on boxes •Lots of ETL
- 18. Old Product: RDBMS •A few MSSQL servers on boxes •Lots of ETL •Several TB, inserts slow, deletes impossible, random fail
- 19. Why RDBMS Bad
- 20. Why RDBMS Bad •Nonlinear scale cost
- 21. Why RDBMS Bad •Nonlinear scale cost •Used as a storage abstraction
- 22. Why RDBMS Bad •Nonlinear scale cost •Used as a storage abstraction •Mainly Select, Join, Group, Count
- 23. Why RDBMS Bad •Nonlinear scale cost •Used as a storage abstraction •Mainly Select, Join, Group, Count •Specialized Scale-Out ones ‘meh’
- 24. Why RDBMS Bad •Nonlinear scale cost •Used as a storage abstraction •Mainly Select, Join, Group, Count •Specialized Scale-Out ones ‘meh’ •Impedance Mismatch - Try to be High- Throughput, Low-Latency
- 25. Why RDBMS Bad •Nonlinear scale cost •Used as a storage abstraction •Mainly Select, Join, Group, Count •Specialized Scale-Out ones ‘meh’ •Impedance Mismatch - Try to be High- Throughput, Low-Latency •Swiss-army knife, unstable, transactions, advanced SQL, tuning
- 26. Why OSS?
- 27. Why OSS? •Previously all MS
- 28. Why OSS? •Previously all MS •It exists!
- 29. Why OSS? •Previously all MS •It exists! •Scaling + Licensing = No
- 30. Why OSS? •Previously all MS •It exists! •Scaling + Licensing = No •Can’t build a platform without source
- 31. Why OSS? •Previously all MS •It exists! •Scaling + Licensing = No •Can’t build a platform without source •It’s Enterprise Now!
- 32. Goals for New Platform
- 33. Goals for New Platform •“Golden Timeline”
- 34. Goals for New Platform •“Golden Timeline” •Search/Analyze *any* data
- 35. Goals for New Platform •“Golden Timeline” •Search/Analyze *any* data •Linear Cost
- 36. Goals for New Platform •“Golden Timeline” •Search/Analyze *any* data •Linear Cost •Not Hacked Together
- 37. Goals for New Platform •“Golden Timeline” •Search/Analyze *any* data •Linear Cost •Not Hacked Together •“Collect the Social Internet”
- 38. HOW TO SCALE
- 39. HOW TO SCALE •What makes you special?
- 40. HOW TO SCALE •What makes you special? •What are you willing to sacrifice?
- 41. HOW TO SCALE •What makes you special? •What are you willing to sacrifice? •How will you structure the data?
- 43. Avoiding Impedance Mismatch •Most problems can be divided into High or Low latency
- 44. Avoiding Impedance Mismatch •Most problems can be divided into High or Low latency •Get a lot of data eventually, or a little now
- 45. Avoiding Impedance Mismatch •Most problems can be divided into High or Low latency •Get a lot of data eventually, or a little now •MapReduce vs. Sharding/Indexing
- 46. Ecosystem Compiled Pig Cascading Hive Processing Katta / Applications Raw Zookeeper MapReduce Processing Structured HBase Storage Unstructured Hadoop DFS Storage
- 47. Simple Workflow Semantic Unstructured Hadoop Collect Analysis Analysis Structured Analysis Hadoop + Store in HBase HBase Store in Indexing Hadoop Lucene+ Load/ Pull Solr+ Replicate Indexes Katta Shards Search
- 48. Unstructured Processing Cluster Semantic Unstructured Structured Internet Collect Store Analysis Analysis HBase HTML XML Records
- 49. Hadoop + MR
- 50. Hadoop + MR •Special: Crunch web-scale data fast
- 51. Hadoop + MR •Special: Crunch web-scale data fast •Sacrifice: Low-Latency, Transactions, Random Access, Updates
- 52. Hadoop + MR •Special: Crunch web-scale data fast •Sacrifice: Low-Latency, Transactions, Random Access, Updates •Structure: Chunked flat files
- 53. Structured Processing Cluster Enriched Data Structured Analysis Unstructured Store in Cluster HBase Store in Search Indexing Hadoop Cluster HBase Records Sharded Lucene Index Lucene Index
- 54. Document Structure ContentID: 00BAC189 Title: Iron Maiden Rules Body: I think Janick Gers is an amazing guitarist blah blah PostDT: 20090718 ParentID: 0FDEADBEEF Permalink: www.roadtofailure.com/post?=20
- 55. HBase
- 56. HBase •Special: Scalable random/sequential access almost as fast as RDBMS
- 57. HBase •Special: Scalable random/sequential access almost as fast as RDBMS •Sacrifice: Joins, Secondary Indexes, Transactions (kind of)
- 58. HBase •Special: Scalable random/sequential access almost as fast as RDBMS •Sacrifice: Joins, Secondary Indexes, Transactions (kind of) •Structure: BigTable - column oriented
- 59. Search Cluster Lucene Load/ Pull Indexes from Replicate Indexes HDFS Shards Search Lucene Lucene Indexes Indexes
- 60. Search
- 61. Katta + Solr
- 62. Katta + Solr •Special: Sharded search
- 63. Katta + Solr •Special: Sharded search •Sacrifice: Consistency, high-throughput
- 64. Katta + Solr •Special: Sharded search •Sacrifice: Consistency, high-throughput •Structure: Reverse index
- 65. BI
- 66. BI •Group, Sort, Filter, Count, Sum
- 67. BI •Group, Sort, Filter, Count, Sum •Semi-additive (Avg) rare but not hard
- 68. BI •Group, Sort, Filter, Count, Sum •Semi-additive (Avg) rare but not hard •MapReduce Jobs
- 69. BI •Group, Sort, Filter, Count, Sum •Semi-additive (Avg) rare but not hard •MapReduce Jobs •Faceted Search
- 70. Examples
- 73. Challenges
- 76. Challenges •Scaling Search •Understanding Latency •What do we need ‘now’? Can customers wait for big data?
- 77. Challenges •Scaling Search •Understanding Latency •What do we need ‘now’? Can customers wait for big data? •Monitoring
- 78. Recap: Rules for Scaling
- 79. Recap: Rules for Scaling •RDBMS is not a Swiss-Army Knife
- 80. Recap: Rules for Scaling •RDBMS is not a Swiss-Army Knife •Know your sacrifices
- 81. Recap: Rules for Scaling •RDBMS is not a Swiss-Army Knife •Know your sacrifices •Know your specialness
- 82. Recap: Rules for Scaling •RDBMS is not a Swiss-Army Knife •Know your sacrifices •Know your specialness •Know your data structure
- 83. Recap: Rules for Scaling •RDBMS is not a Swiss-Army Knife •Know your sacrifices •Know your specialness •Know your data structure •Ponder Latency
- 84. What Next?
- 86. What Next? •HBase Analytics? •“What would make a bank trust it”
- 87. What Next? •HBase Analytics? •“What would make a bank trust it” •Teach people to think about data
- 88. ...
- 89. The End company: www.visibletechnologies.com blog: www.roadtofailure.com twitter: @lusciouspear [email protected]