Introduction to Hadoop and Big Data Processing
0 likes35 views
The document presents an introduction to Hadoop and big data processing, highlighting the rapid growth of IoT units and data generation in industries like automotive. It covers the history of Hadoop, its architecture (HDFS and MapReduce), and security considerations including confidentiality and vulnerabilities. The document also discusses the Hadoop ecosystem, including related tools and the importance of resilience in Hadoop clusters.
Introduction to Hadoop and Big Data Processing
- 1. Introduction to Hadoop and Big Data Processing Presented by Sam Ng Date: 7 Deptember 2018
- 2. 2 • The number of IoT Units installed in 2018 is doubled comparing with the number of installed Units in 2016. Two years later, the number of IoT Units is expected to be doubled again. • That means sensor data will increase rapidly due to highly adoption of IoT devices. Introduction
- 3. 3 • Around a Terabyte of Sound Data will be generated if a car manufacturer records sound files for a single product line such to control quality in a year. • File size of 30 seconds of sound = 5.046980702MB, A car manufacturer produces 200,000 cars for a single model per year, If a file is recorded for each car, The total size of recorded files will be 985.7384183GB. However, they may record more than a file for each car. Introduction
- 4. 4 • An example solution for automobile manufacturers Introduction
- 5. 5 • A Brief History of Hadoop • What is HDFS and how to use it • What is Map Reduce • Advanced Map Reduce • Namenode Resilience • Directed Acyclic Graph • Hadoop Ecosystem • How to configure security for a Hadoop Cluster Agenda
- 6. 6 • In 2003, Google published a paper “The Google File System” about a scalable distributed file system that they were using. http://static.googleusercontent.com/media/research.google.com/ en//archive/gfs-sosp2003.pdf • That paper inspired Doug Cutting, an employee of Yahoo!, to create an open-source framework Hadoop based on the core concept “MapReduce” borrowed from Google. • The name Hadoop doesn’t have any meaning at all. The kid of Doug Cutting drew a yellow elephant for this project. A Brief History of Hadoop
- 7. 7 • Projects related to Hadoop trends to use animal names or animal logos, such as pig and hive. Those descriptive components build up a Hadoop ecosystem. • The configuration management tool in the Hadoop ecosystem is called “ZooKeeper”. A Brief History of Hadoop
- 8. 8 • Name Nodes: To record where the files go, and log what is being created and modified • Data Nodes: To store data. The default block size is 128MB. (The block size varies depending on file systems, can be 512 bytes, 4kB, 8kB, 16kB, 32kB etc. The block size in my Macbook is 512 Bytes. ) • Client Nodes: To store client’s applications • Please note that HDFS only refers to the file system. To operate it, a resource manager “YARN” is required. What is HDFS
- 9. 9 • UI (Ambari, Hue) • CLI, similar to cd, ls • HTTP / HTTPS Proxies • Java interface • NFS Gateway (To remove or mount a file system into a server) How to use HDFS
- 10. 10 • Map data: transform data to another structure for solving, associate the data with some Key Values • Reduce data: aggregate data together (what you like to do with each piece of data, eg count, maximum) What is Map Reduce?
- 11. 11 What is Map Reduce? Magic Happened!
- 12. 12 What is Map Reduce? Shuffle and sort
- 13. 13 What is Map Reduce? (Advanced)
- 14. 14 • The single point of failure in a Hadoop cluster is the NameNode. While the loss of any other machine (intermittently or permanently) does not result in data loss, NameNode loss results in cluster unavailability. The permanent loss of NameNode data would render the cluster's HDFS inoperable. Namenode Resilience
- 15. 15 • Backup metadata (data node route table and edit logs) • Secondary namenode (Maintain a copy) • HDFS Federation(Have a separated namenode for each namenode volume) -> Only lose a portion of data when a namenode is down • HDFS High Availability (Use shared edit log based on reliable file system) -> Use Zookeeper keeps track of the active namenode Namenode Resilience
- 16. 16 • Instead of Map Reduce, find out the fastest way to calculate the result depending on scenarios. • Using DAG, Sparks claimed that it is 100 times fastest than Hadoop. Directed Acyclic Graph
- 18. 18 • H2O • Spark ML or mllib • Mahout • Spark • Pig Data analysis tools in Hadoop Machine Learning Tools: Database Tools: • Hive • HBase
- 19. Hadoop Security Security Concern: - Confidentiality - Integrity - Availability - Authentication - Authorization - Accounting Threats: - Unauthorized access - Insider threat - DoS - Data threat Vulnerabilities: - Password truncated - Ping of death Consideration: - Network Security - Data Flow - Client Access - Admin Traffic Defense in Depth “Any single security measure is not likely to mitigate all threats”
- 20. 20 Hadoop Security - Kerberos
- 21. Thank you!