Case Studies on Big-Data Processing and Streaming - Iranian Java User Group
6 likes1,638 views
The document presents a comprehensive overview of big-data processing and data streaming, showcasing various case studies and technologies such as Apache Hadoop and NoSQL databases. It discusses the characteristics of big data, its significance in modern data management, and various solutions for efficient processing, including scalable data warehouses and real-time search engines. Furthermore, it explores the applications of machine learning and stream processing for tasks like fraud detection and log management.
Case Studies on Big-Data Processing and Streaming - Iranian Java User Group
- 1. Case Studies on Big-Data Processing and Data Streaming By: Amir Sedighi LinkedIn: http://linkedin.com/in/amirsedighi Twitter: @amirsedighi
- 2. JUG - A.Sedighi - 2015 2 / 48 Background ● BS and MS degrees in Software Engineering ● Senior Software Engineer – +20 Years of Programming Experience ● Cross-platform Software Development – +4 Years of Big-Data Processing and Machine-Learning Experience ● Log Management and Forensic ● Big-Data Visualization ● Data Warehouse using Big-Data Technologies ● Recommender Systems ● Analytical Real-Time Search Engines ● Integrating Fedora Digital Library with HDFS ● Next Generation Event Processing ● Online Resume – http://linkedin.com/in/amirsedighi
- 3. JUG - A.Sedighi - 2015 3 / 48 Outline ● An Introduction to Big-Data Processing ● Big-Data and Processing and Data Streaming – Data Processing 1. +TB Scale Data Warehouse 2. Analytical Real-Time Search Solution and BI 3. Scaleable Recommender System 4. Integrating Fedora Digital Library with HDFS – Stream and Event Processing 1. Super Fast Scaleable Log Management, Forensic and BI 2. Super Fast Scaleable Fraud Detection
- 4. JUG - A.Sedighi - 2015 4 / 48 What Big-Data Is?
- 5. JUG - A.Sedighi - 2015 5 / 48 ● Every 2 Days Human Create As Much Information As We Did Up To 2003 - Eric Schmidt
- 6. JUG - A.Sedighi - 2015 6 / 48 Big-Data Characteristics ● Volume ● Variety ● Velocity
- 7. JUG - A.Sedighi - 2015 7 / 48 You're a Part of It Every Day ● We've have the ability to store anything ● Companies and people are generating data like never before in history – Social Networks – Online Web Portals – Log Writers - Our Digital Footprint!
- 8. JUG - A.Sedighi - 2015 8 / 48 You're a Part of It Every Day ● Big-Data is whatever people do in the digital world, including the foot print of what people, companies, devices and services do (Logs), including traditional tabular data stores.
- 9. JUG - A.Sedighi - 2015 9 / 48 As a Manager still You're a Part of It ● “Over half of the business leaders today, realize they don't have access to the insights they need to do their job.” - IBM
- 10. JUG - A.Sedighi - 2015 10 / 48 Vertical or Horizontal?
- 11. JUG - A.Sedighi - 2015 11 / 48 Scale Up or Scale Out
- 12. JUG - A.Sedighi - 2015 12 / 48 Linear Scalability
- 13. JUG - A.Sedighi - 2015 13 / 48 Big-Data Processing Solutions
- 14. JUG - A.Sedighi - 2015 14 / 48 Q: How To Be Linear Scaleable on Commodity Machines? A: MapReduce
- 15. JUG - A.Sedighi - 2015 15 / 48 Q: How to store big data on commodity machines? A: Distributed File System
- 16. JUG - A.Sedighi - 2015 16 / 48 Replication → Fault Tolerant Replication → Data Locality → Utilization
- 17. JUG - A.Sedighi - 2015 17 / 48 Big-Data Processing, Most Popular Technologies ● Apache Hadoop Ecosystem ● NoSQL Databases – HBase – Cassandra – MongoDB – Neo4j ● Elasticsearch – Lucene – SolR ● Java
- 18. JUG - A.Sedighi - 2015 18 / 48 +TB Scale Data Warehouse 1
- 19. JUG - A.Sedighi - 2015 19 / 48 DW Solution ● SQL ● ETL – RDBMS – NoSQL – File System ● REST API
- 20. JUG - A.Sedighi - 2015 20 / 48 REST Admin Panel
- 21. JUG - A.Sedighi - 2015 21 / 48 Features ● Extendable Capacity for Data Warehousing ● Making Very Big Integrated Databases Based on Different Technologies/Schemas – DB2, Oracle, MS-SQL … – Different Schemas Such as HRMS, Banking, Sales... – Making Small Dense Integrated RDBMSs ● SQL Language Interface ● Linear Scalability
- 22. JUG - A.Sedighi - 2015 22 / 48 Main Technologies and Frameworks ● Apache Hadoop – Sqoop – YARN/HDFS – Hive or Drill or Impala ● Microservices Architecture – Java 1.7 – Spring Boot
- 23. JUG - A.Sedighi - 2015 23 / 48 Analytical Real-Time Scalable Search Solution and BI 2
- 24. JUG - A.Sedighi - 2015 24 / 48 +TB Scale RT Searching ● Indexing Incoming Data on-the-fly ● Highly Scaleable and Reliable ● Simple or Complex Queries ● REST API ● Schema Agnostic ● Customizable GUI and BI
- 25. JUG - A.Sedighi - 2015 25 / 48 Business Intelligence
- 26. JUG - A.Sedighi - 2015 26 / 48 Rich GUI
- 27. JUG - A.Sedighi - 2015 27 / 48 Main Technologies and Frameworks ● Elasticsearch – Apache Lucene – REST ● Kibana
- 28. JUG - A.Sedighi - 2015 28 / 48 Scalable Recommender System 3
- 29. JUG - A.Sedighi - 2015 29 / 48 Recommender System ● Value-added Service (Loyalty Services) ● Machine-Learning – Clustering Throw Thousands of Nodes ● Apache Mahout ● Super Fast
- 30. JUG - A.Sedighi - 2015 30 / 48 How It Works?
- 31. JUG - A.Sedighi - 2015 31 / 48 Technologies and Frameworks ● Microservices Architecture ● Java 1.6 ● Apache Mahout ● Redis
- 32. Fedora Digital Library and HDFS Integration 4
- 33. Migrating from Expensive Servers to Commodity Machines ● Making HDFS as Fedora Digital Library Storage – Research and Development – Hadoop 1.2, Later Hadoop YARN 2.2 – Integrating with SolR over HDFS ● Java 1.7 ● Fedora – Islandora – GSearch
- 34. JUG - A.Sedighi - 2015 34 / 48 Data Streaming
- 35. JUG - A.Sedighi - 2015 35 / 48 Big-Data Streaming, Most Popular Technologies ● Piping and Messaging – Kafka, Flume, FluentD and ZeroMQ ● Stream Processing – Storm, Samza and Spark ● Machine Learning – Machine Learning: MLLib and Mahout ● Persisting – NoSQL DBs – HDFS
- 36. JUG - A.Sedighi - 2015 36 / 48 Log Management, Forensic and BI 1
- 37. JUG - A.Sedighi - 2015 37 / 48 Log Management, Forensic and BI ● Every Digital Stuff Writes Things Into Log Files – Log Files Are Streams of Data – Log Files Are Messy – Log Files Come Very Fast, in an Un-Predictable Manner – Log Files Are About Everything within Your Business ● Log Files Are Full of Insight – Who Can Hold Them For a Reasonable Period of Time – Who Can Search Them Rapidly – Who Can Visualize Them Easily (BI)
- 38. JUG - A.Sedighi - 2015 38 / 48 Network Topology LB Masters Data
- 39. JUG - A.Sedighi - 2015 39 / 48 Main Technologies and Frameworks ● LogStash – Flume ● Elasticsearch ● Kibana
- 40. JUG - A.Sedighi - 2015 40 / 48 Snapshot
- 41. JUG - A.Sedighi - 2015 41 / 48 Fraud Detection 2
- 42. JUG - A.Sedighi - 2015 42 / 48 Inputs & Outputs ● Inputs: One or multiple sources generate data continuously, in real time – Sensor Networks – Transaction Logs – Text Streams such as News – Network Traffic Analysis ● Outputs: Up-to-date Answers generated continuously or periodically
- 43. JUG - A.Sedighi - 2015 43 / 48 Data Processing Transient Query – Issued once, then forgotten Persistent Data Stored until deleted by user or apps
- 44. JUG - A.Sedighi - 2015 44 / 48 Stream Processing Transient Data – Deleted as Window Slides Forward Generated up-to-date answers as time goes on Persistent Queries TimeBased CountBased
- 45. JUG - A.Sedighi - 2015 45 / 48 Features ● Scalability ● Real-Timing, (Only 1 Second delay at most) ● Super Fast Decision Making ● Implementing Complex Fraud Scenarios Aa Easy as Defining Queries ● Uniform Api For Processing Old or Early Events
- 46. JUG - A.Sedighi - 2015 46 / 48 Main Technologies and Frameworks ● Java 1.7, Scala 2.11 ● Apache Flume ● Apache Kafka ● Apache Spark
- 47. Where To Start? ● You need Big Amount of Data ● You need to change your mind – Rack Space and Number of Servers, IO and Process Limitations ● You need To Understand Fundamentals – Linux (Bash Script) – Java is a Most, Python works and Scala is an advantage – SQL and ETL – MapReduce, Resource Management and Serialization Frameworks – Apache Hadoop Ecosystem and Successors
- 48. JUG - A.Sedighi - 2015 48 / 48 Thank You!, Question? http://slideshare.net/amirsedighi