Hadoop - An Introduction
Download as PPTX, PDF0 likes859 views
This document provides an introduction and overview of Hadoop. It discusses how businesses have been collecting large amounts of data but face challenges in analyzing it due to application complexities, data growth, infrastructure limitations, and economic factors. Hadoop is presented as a solution that can handle high-volume data, perform complex operations at scale, is robust and fault tolerant. Key components of Hadoop like HDFS, MapReduce, and the Hadoop ecosystem are described at a high level.
Hadoop - An Introduction
- 1. Shankar RadhakrishnanHCL TechnologiesHadoop – An Introduction
- 2. State of the DataWhat is HadoopHadoop EcosystemReferencesAgenda
- 3. Data driven businessesBusinesses have been collecting information all the timeMine more == Collect more (and vice-versa)ChallengesApplication ComplexitiesData growthInfrastructureEconomicsNeed of the dayState of the data
- 4. Data driven businessBusinesses have been collecting informationall the timeMine more == Collect more (and vice-versa)ChallengesApplication ComplexitiesData growthInfrastructureEconomicsState of the data
- 5. ApplicationsSearches, Message posts, Comments, Emails,Blogs, Photos, Video Clips, Product ListingsERP, CRM, Databases, Internal Applications, Customer/Consumer facing productsMobileContextWeb, Customers, Products, Business Systems,Processes, ServicesSupport SystemsCRM, SOA, Recommendation Systems/processes,Data warehouses, Business Intelligence, BPMData driven business
- 6. Data driven businessesBusinesses have been collecting informationall the timeMine more == Collect more (and vice-versa)ChallengesApplication ComplexitiesData growthInfrastructureEconomicsState of the data
- 7. DriversROICustomer RetentionProduct AffinityMarket TrendsResearch AnalysisCustomer/Consumer AnalyticsProcessClusteringClassificationBuild RelationshipsRegressionTypesStructuredSemi-structuredUnstructuredMine more
- 8. Data driven businessesBusinesses have been collecting informationall the timeMine more == Collect more (and vice-versa)ChallengesApplication ComplexitiesData growthInfrastructureEconomicsState of the data
- 9. Complex ApplicationsData integration is a good but complex problem to solveData GrowthGrowth is exponentialInfrastructureAvailabilityUnscalablehardwareEconomicsManaging high data volume comes at a priceFailures are very costlyChallenges
- 10. System that can handle high volume dataSystem that can perform complex operationsScalableRobustHighly AvailableFault TolerantCheapNeed of the day
- 11. Top level Apache projectOpen sourceInspired by Google’s white papers onMap/Reduce (MR), Google File System (GFS)Originally developed to support Apache Nutch Search EngineSoftware Framework - JavaDesignedFor sophisticated analysisTo deal with structured and unstructured complex data
- 12. Runs on commodity hardwareShared-nothing architectureScale hardware when ever you wantSystem compensates for hardware scalingand issues (if any)Run large-scale, high volume data processesScales well with complex analysis jobsHandles failuresIdeal to consolidate data from both new and legacy data sourcesValue to the businessWhy Hadoop?
- 13. Hadoop in an enterprise - Example
- 14. HDFS Hadoop Distributed File SystemMap/Reduce Software framework for Clustered, Distributed data processingZooKeeper SchedulerAvro Data SerializationChukwa Data Collection System to monitor Distributed SystemsHBase Data storage for distributed large tablesHive Data warehousing infrastructurePig High-Level Query LanguageHadoop Ecosystem
- 15. Master/Slave ArchitectureRuns on commodity hardwareFault TolerantHandle large volumes of dataProvides High ThroughputStreaming data-accessSimple file coherency modelPortable to heterogeneous hardware and softwareRobustHandles disk failures, replication (& re-replication)Performs cluster rebalancing, data integrity checksHDFS – Hadoop Distributed File System
- 16. HDFS – ExampleName nodeFile system operations
- 17. Maps data-nodesData nodeProcess read/write
- 19. ReplicationTagged by a jobSplits input data-set into separate chunk’sProcessed by map tasks, in parallelSorts the output of the mapsProcessed by reduce tasks, in parallelTypically stored and processed in a file systemFramework takes care ofScheduling tasksMonitoringRe-executing failed tasksHadoop Map/Reduce
- 20. Example : Mapper Function
- 21. Example : Reduce Function
- 22. Who runs Hadoop?