Big data and computing grid
0 likes333 views
This document discusses big data concepts and distributed computing frameworks. It explains that big data refers to large and complex datasets that are difficult to process using traditional tools. Examples are given of enterprises generating terabytes and petabytes of data daily from various sources. Common big data uses like recommendation engines and fraud detection are listed. Apache Hadoop and Apache Spark are introduced as open-source frameworks for distributed processing of large datasets across clusters of computers. Key characteristics of each are outlined, with Spark noted as building on Hadoop MapReduce but extending the model to support more types of computations in a faster, more efficient manner.
Big data and computing grid
- 2. 1 2 3 4 5 6 Starting our complex problem Awesome Distribute file system Lightning-Fast Cluster Computing 2
- 3. 3 About goal This workshop will help in understanding compute grid, it’s model and it’s implementation. We have tried my best to explain the concepts in detail. The programming language used for demo in here is Scala. And we then apply this model to settle some problems we can see where this is simple and useful for us.
- 4. 1
- 5. • Lost of Data (Terabytes or Petabytes) • Big data is the term of collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing application. The challenges include capture, curation, storage, search, sharing, transfer, analysis and visualization. • Systems/ Enterprises generate huge amount of data form Terabytes to and even Petabytes of information. NYSE generates about one terabyte of new trade data/day to perform stock trading analytics to determine trends for optimal trades.
- 7. 7 • 2,500 exabytes of new information in 2013 with internet as primary driver • Digital universe grew by 62% last year to 800K petabytes and will grow to 1.2 zettabytes this years
- 8. 8
- 9. 9 Enterprises are awash with ever-growing data of all types, easily amassing terabytes—even petabytes—of information. Sometimes 2 minutes is too late. For time-sensitive processes such as catching fraud, big data must be used as it streams into your enterprise in order to maximize its value. Big data is any type of data - structured and unstructured data such as text, sensor data, audio, video, click streams, log files and more. New insights are found when analyzing these data types together.
- 10. 1 2 3 4 10 Recommendation engines Ad targeting Search quality Abuse and Click fraud detection
- 11. 1 2 3 4 11 Customer churn prevention Network performance optimization Calling data record analysis Analyzing network to predict failure
- 12. 1 2 3 4 12 Health information exchange Gene sequencing Healthcare service quality improvements Drug safety
- 13. 1 2 3 4 13 Modeling true risk Threat analysis Fraud detection Credit scoring and analysis
- 17. 2
- 19. Apache Hadoop is a framework that allows for distributed processing of large data sets across clusters of commodity computers using a simple programming model. It is an Open-source Data Management with scale-out storage & distributed processing. 19
- 20. 1 2 3 4 20
- 21. 21
- 22. 22 • Splits a task across processors • “Near” the data & assembles results • Self-healing, high bandwidth • Clustered storage • JobTracker manages the TaskTrackers • Distributed across nodes • Natively redundant • NameNode tracks locations.
- 23. 23
- 24. 3
- 25. Apache Spark is an open-source cluster-computing framework for real time processing developed by the Apache Software Foundation. Spark provides an interface for programming entire clusters with implicit data parallelism and fault-tolerance. It was built on top of Hadoop MapReduce and it extends the MapReduce model to efficiently use more types of computations. 25
- 27. 27
- 28. 28
- 29. 29 Map, flatMap, Filter, … Collect, Take, count, … Resilient Distributed Datasets (RDD): Collection that can be operated in parallel.
- 30. 4
- 31. 31 Hadoop Mapreduce Spark Time to sort 100TB data
- 32. 32 Most active open source community in big data 200+ developers, 50+ companies contributing
- 33. 33
- 34. 34
- 35. 4