Big data for the rest of us
Download as KEY, PDF9 likes3,595 views
The document discusses the evolution and significance of big data, highlighting its exponential growth and the need for custom software in handling it. It emphasizes the roles of technologies like MongoDB and Hadoop in storage and processing, exemplified through real-world applications such as Orbitz's hotel data collection. The future of big data is portrayed as a constantly evolving landscape, necessitating innovative tools and solutions.
Big data for the rest of us
- 1. Not Just Hadoop NoSQL in the Enterprise
- 2. Talking about What is BIG Data BIG Data & you Real world examples The future of Big Data
- 3. @spf13 AKA Steve Francia 16+ years building the internet Father, husband, skateboarder Chief Evangelist @ responsible for drivers, integrations, web & writing
- 5. 2000 Google Inc Today announced it has released the largest search engine on the Internet. Google’s new index, comprising more than 1 billion URLs
- 6. 2008 Our indexing system for processing links indicates that we now count 1 trillion unique URLs (and the number of individual web pages out there is growing by several billion pages per day).
- 7. An unprecedented amount of data is being created and is accessible
- 8. Data Growth 1,000 1000 750 500 500 250 250 120 55 4 10 24 1 0 2000 2001 2002 2003 2004 2005 2006 2007 2008 Millions of URLs
- 9. Truly Exponential Growth Is hard for people to grasp A BBC reporter recently: "Your current PC is more powerful than the computer they had on board the first flight to the moon".
- 10. Moore’s Law Applies to more than just CPUs Boiled down it is that things double at regular intervals It’s exponential growth.. and applies to big data
- 11. How BIG is it?
- 12. How BIG is it? 2008
- 13. How BIG is it? 2007 2008 2005 2006 2003 2004 2001 2002
- 14. We’ve had BIG Data needs for a long time In 1998 Google won the search race through custom software & infrastructure
- 15. We’ve had BIG Data needs for a long time In 2002 Amazon again wrote custom & proprietary software to handle their BIG Data needs
- 16. We’ve had BIG Data needs for a long time In 2006 Facebook started with off the shelf software, but quickly turned to developing their own custom built solutions
- 17. Ability to handle big data is one of the largest factors in determining winners vs losers.
- 18. For over a decade BIG Data = custom software
- 19. Why all this talk about BIG Data now?
- 20. In the past few years open source software emerged enabling ‘us’ to handle BIG Data
- 21. The Big Data Story
- 23. Doers & Tellers talking about different things http://www.slideshare.net/siliconangle/trendconnect-big-data-report-september
- 24. Tellers
- 25. Doers
- 26. Doers talk a lot more about actual solutions
- 27. They know it’s a two sided story Storage Processing
- 28. Take aways MongoDB and Hadoop MongoDB for storage & operations Hadoop for processing & analytics
- 29. How MongoDB enables big data •Flexible schema • Horizontal scale built in & free •Operates at near speed of memory • Optimized for modern apps
- 30. MongoDB @ Orbitz Rob Lancaster October 23 | 2012
- 31. Use Cases • Hotel Data Collection • Hotel Rate Feed: • Supply hotel rates to Google for their Hotel Finder • Uses MongoDB: - Maintain state of data sent to Google - Identify changes in rates as they occur • Makes use of complex querying, secondary indexing • EasyLoader: • Feature allowing suppliers to easily load inventory to Orbitz • Uses MongoDB to persist all changes for auditing purposes 29
- 32. Hotel Data Collection • Goals: • Better understand performance of our hotel path • Optimize our hotel rate cache • Methods: • Capture every action performed by our hotel search engine. • Persist this data for long periods. • Challenges: • Need high performance capture. • Scalable, inexpensive storage. 30
- 33. Requirements Collection Storage & Processing • High write throughput • High data volume • 500 servers • ~500 GB/day • > 100 million documents/day • 7 TB/month compressed • Flexibility • Scalable • Complex extendable documents • Inexpensive • No forced schema • Proximity with other data • Scalability • Simplicity 31
- 34. The Solution • Utilize MongoDB as a collector: • ~ 500 clients • Utilize unsafe writes for high throughput • Heterogeneous documents • New collection for each hour • HDFS for storage & processing: • Data moved via M/R job: - One job per collection - One mapper per MongoDB instance • Additional processing and analysis by other jobs 32
- 35. Challenges & Conclusions •Challenges? None really. •Achieved a robust and simple solution •MongoDB has been entirely worry free • Very high write throughput • Reads (well, full collection dumps across the wire) are slower 33
- 36. The Futureof BIG data
- 37. What is BIG? BIG today is normal tomorrow
- 38. Data Growth 9,000 9000 6750 4,400 4500 2,150 2250 1,000 500 55 120 250 1 4 10 24 0 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 Millions of URLs
- 39. Data Growth 9,000 9000 6750 4,400 4500 2,150 2250 1,000 500 55 120 250 1 4 10 24 0 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 Millions of URLs
- 40. How BIG is it?
- 41. How BIG is it? 2012
- 42. How BIG is it? 2011 2012 2009 2010 2007 2008 2005 2006
- 43. 2012 Generating over 250 Millions of tweets per day
- 44. MongoDB enables us to scale with the redefinition of BIG. Tools like Hadoop are enabling us to process the new BIG.
- 45. MongoDB is committed to working with best data tools including Hadoop, Storm, Disco, Spark & more
- 46. http://spf13.com http://github.com/s @spf13 Question download at mongodb.org We’re hiring!! Contact us at [email protected]
Editor's Notes
- #2: \n
- #3: \n
- #4: 10\n15\n10\n5\n
- #5: \n
- #6: \n
- #7: \n
- #8: \n
- #9: \n
- #10: \n
- #11: \n
- #12: \n
- #13: \n
- #14: \n
- #15: \n
- #16: \n
- #17: \n
- #18: \n
- #19: \n
- #20: \n
- #21: \n
- #22: \n
- #23: \n
- #24: \n
- #25: \n
- #26: \n
- #27: \n
- #28: \n
- #29: \n
- #30: http://www.slideshare.net/siliconangle/trendconnect-big-data-report-september\n
- #31: \n
- #32: \n
- #33: \n
- #34: \n
- #35: \n
- #36: \n
- #37: \n
- #38: \n
- #39: \n
- #40: \n
- #41: \n
- #42: \n
- #43: \n
- #44: \n
- #45: \n
- #46: \n
- #47: \n
- #48: \n
- #49: \n
- #50: \n
- #51: \n
- #52: \n
- #53: \n
- #54: \n
- #55: \n
- #56: One site is generating nearly as many URLs as the entire internet 6 years ago.\n
- #57: \n
- #58: \n
- #59: \n
- #60: \n