Big data- HDFS(2nd presentation)
Download as PPTX, PDF0 likes977 views
Hadoop Distributed File System (HDFS) is a distributed file system that stores large datasets across commodity hardware. It is highly fault tolerant, provides high throughput, and is suitable for applications with large datasets. HDFS uses a master/slave architecture where a NameNode manages the file system namespace and DataNodes store data blocks. The NameNode ensures data replication across DataNodes for reliability. HDFS is optimized for batch processing workloads where computations are moved to nodes storing data blocks.
Big data- HDFS(2nd presentation)
- 1. Presentation on Big Data- HADOOP DISTRIBUTED FILE SYSTEM(H.D.F.S) Presented by – TAKRIM UL ISLAM LASKAR(120103006)
- 2. Who uses Hadoop? • Amazon/A9 • Facebook • Google • IBM : Blue Cloud? • Joost • Last.fm • New York Times • PowerSet • Veoh • Yahoo!
- 3. HADOOP DISTRIBUTED FILE SYSTEM(HDFS) Highly fault tolerant High throughput Suitable for application with large data sets Streaming access to file system data Can be built out of commodity hardware
- 4. Commodity Hardware Typically in 2 level architecture – Nodes are commodity PCs – Uplink from rack is 3-4 gigabit – Rack-internal is 1 gigabit
- 5. Why DFS? 1 MACHINE • 4 I/O channels • Each channel- 100mb/s 10 MACHINES • 4 I/O channels • Each channel- 100mb/s …... 45 Minutes 4.5 Minutes READ 1TB DATA
- 6. Goals of HDFS • Very Large Distributed File System – 10K nodes, 100 million files, 10 PB • Assumes Commodity Hardware – Files are replicated to handle hardware failure – Detect failures and recovers from them • Optimized for Batch Processing – Data locations exposed so that computations can move to where data resides – Provides very high aggregate bandwidth
- 8. Functions of a NameNode • Manages File System Namespace – Maps a file name to a set of blocks – Maps a block to the DataNodes where it resides • Cluster Configuration Management • Replication Engine for Blocks
- 9. NameNode Metadata • Meta-data in Memory – The entire metadata is in main memory (RAM) • Types of Metadata – List of files – List of Blocks for each file – List of DataNodes for each block – File attributes, e.g creation time, replication factor • A Transaction Log – Records file creations, file deletions. etc
- 10. DataNode • A Block Server – Stores data in the local file system (e.g. ext3) – Stores meta-data of a block – Serves data and meta-data to Clients • Block Report – Periodically sends a report of all existing blocks to the NameNode • Facilitates Pipelining of Data – Forwards data to other specified DataNodes
- 11. Block Placement • Current Strategy -- One replica on local node -- Second replica on a remote rack -- Third replica on same remote rack -- Additional replicas are randomly placed • Clients read from nearest replica.
- 12. Heartbeats • DataNodes send heartbeat to the NameNode – Once every 3 seconds • NameNode used heartbeats to detect DataNode failure
- 13. Replication Engine • NameNode detects DataNode failures – Chooses new DataNodes for new replicas – Balances disk usage – Balances communication traffic to DataNodes
- 14. HDFS- 2nd generation NameNode- Active NameNode Passive NameNode
- 15. Data Pipelining • Client retrieves a list of DataNodes on which to place replicas of a block • Client writes block to the first DataNode • The first DataNode forwards the data to the next DataNode in the Pipeline • When all replicas are written, the Client moves on to write the next block in file
- 16. Secondary NameNode • Copies FsImage and Transaction Log from NameNode to a temporary directory • Merges FSImage and Transaction Log into a new FSImage in temporary directory • Uploads new FSImage to the NameNode – Transaction Log on NameNode is purged
- 17. User Interface • Command for HDFS User: – hadoop dfs -mkdir /foodir – hadoop dfs -cat /foodir/myfile.txt – hadoop dfs -rm /foodir myfile.txt • Command for HDFS Administrator – hadoop dfsadmin -report – hadoop dfsadmin -decommission datanodename • Web Interface – http://host:port/dfshealth.jsp
- 18. Hadoop Map/Reduce • The Map-Reduce programming model – Framework for distributed processing of large data sets – Pluggable user code runs in generic framework • Common design pattern in data processing cat * | grep | sort | unique -c | cat > file input | map | shuffle | reduce | output • Natural for: – Log processing – Web search indexing – Ad-hoc queries
- 19. References : 1. Youtube Lecture video on chennal ‘ Training on Big Data and Hadoop ’ By User ‘Edureka’. 2. ‘White Book Of Big Data’ By ‘Fujistu’ . 3. ‘Big Data For Dummies’ by ‘A Wiley Brand’ . 4. Research paper by ‘Kalapriya Kannan’ in ‘IBM Research Labs’. 5. Collected data from “slideshare.com”
- 20. Useful Links • HDFS Design: – http://hadoop.apache.org/core/docs/current/hdfs_design.html • Hadoop Information – http://hadoop.apache.org/ • Hadoop API: • – http://hadoop.apache.org/core/docs/current/api/
- 21. -THANK YOU. I appreciate your patience.