Budapest Big Data Meetup Real-time stream processing
- 2. • Who we are and what we are doing • Big Data era • BSP (Bulk synchronous parallel) • Apache Giraph • Storm • Our use case • Conclusion Topics today Starschema Experience and Innovation
- 3. Continuous growth 25 FTE plus external resources, over $1.5million EBIT Open source projects Share the knowledge with the public. Open source project in ETL and data warehousing fields. Founded in 2006 Company was founded by private owners with decade of BI and data warehouse background R&D Cooperation with Obuda University, NKE, EU co-founded technology research and development COMPANY Data Facts about Starschema Starschema Experience and Innovation
- 4. Big Data eraThe rise of Hadoop Starschema Experience and Innovation Google Year of WP Apache Year GFS 2003 HDFS 2007 MapReduce 2004 Hadoop MR 2007 BigTable 2006 HBase 2007 Chubby Lock Service 2006 ZooKeeper 2007 Pregel 2009 Giraph 2011 Dremel 2010 Drill 2012 ? Which is next? (Curator, Falcon, MRQL, etc.)
- 5. • Leslie Valiant - article in nov. 1990 • Supersteps • Data stored in local memory • Asynchronous data processing • Barrier sync • Optimal load balacing (more logical processes than physcal processors, random allocation of processes) • Solution differences (procotols, buffer management, routing strategies) • No deadlock or any other race conditions (since no circular dependency) • Use cases BSP (Bulk synchronous parallel)What is it? What is it good for? Starschema Experience and Innovation
- 6. Storm Apache Giraph Starschema Experience and Innovation
- 7. Apache Giraph Starschema Experience and Innovation • A loose implementation of Pregel • Avery Chink: We can't use it at Yahoo, that's too bad • Developed at Yahoo • Runs on existing MapReduce infrastructure • Netty based comm. instead of Hadoop RPC • In-memory • Fault tolerant • Internal state is saved at user-defined intervals • Master/slave architecture What is it?
- 8. Storm Starschema Experience and Innovation • Storm is a free and open source distributed real time computation system • Developed at BackType, open-sourced by Twitter in 2011 • Guaranteed data processing • Horizontal scalability • Fault tolerance • ZeroMQ for message passing • Processing unbounded sequence of tuples • Groupings What is it?
- 9. Storm Starschema Experience and Innovation What is it for? • Analyze, clean, normalize • Real-time calculation • Real-time ETL • Failure detection from log files • Machine data analysis • IT early-warning systems, security and fraud detection • Traffic information, DOS attack • Stream processing - Continous computation - Distributed RPC
- 10. Our use case Starschema Experience and Innovation • Real-time calculation • Error detection • Horizontal scalability • Fast implementation • High-availability • Error prediction POC: Processing machine data from sensors Requirements
- 11. Our use case part 2 Starschema Experience and Innovation • Choosen tool: Storm • One spout for each sensor • Dynamic add and remove of spouts • Error detection based on statistical calculations • ~ 200 lines • HA capability of Storm POC: Processing machine data from sensors Solution:
- 12. Conclusion • Extend existing infrastructure • Answer to new questions • Re-think old problems • New solutions, new features • Happy customers/users • $$$ Starschema Experience and Innovation
- 13. Starschema Experience and Innovation What else to use? • Yahoo S4 (Apache Incubator project) • Apache Hama (Top level Apache project) • GoldenOrb • Signal/Collect
- 14. QUESTIONS & ANSWERS Q…A Starschema Experience and Innovation [email protected] www.starschema.net