Hdfs internals
1 like108 views
Hadoop uses large 64MB blocks by default to store file data in HDFS for improved performance. The namenode manages file metadata and knows which datanodes store each block. Datanodes store and retrieve blocks as requested by clients. The secondary namenode helps manage the namenode metadata but cannot replace it in case of failure. Writing files involves breaking them into blocks and storing replicas across datanodes, while reading locates blocks and retrieves their data.
Hdfs internals
- 3. What are Blocks? ● A physical storage disk has a block size - minimum amount of data it can read or write. Normally 512 bytes. ● File systems for a single disk also deal with data in blocks. Normally few kilo bytes (4 kb). ● Hadoop has a much larger block size. By default it is 64 mb. ● Files in HDFS are broken down into block sized chunks and are stored as independent units. ● However, files smaller than a block size do not occupy the entire block. ○ Should I care?
- 4. Why so large blocks? ● Minimize disk seek times ● Assuming 10 ms of seek time, and 100 MB/s as disk transfer rate, if block size if 100 MB, then seek time is 1% of transfer time which is small enough to ignore. ● Hence default is 64 MB while many production environments also use 128 MB.
- 5. HDFS Architecture Image Source: https://hadoop.apache.org
- 6. Namenode and Datanode ● Master - Namenode ○ Manages file system namespace ○ File system tree and metadata for all files and directories ○ Stores this info in - ■ Namespace image ■ Edit log ○ Knows for a given file which datanodes has the corresponding blocks. Reconstructed at startup ● Worker - Datanode ○ Store and retrieve blocks as requested by clients ○ Periodically report back to the namenode on the list of blocks they are storing
- 7. HDFS Storage Image Source: https://developer.yahoo.com/hadoop/tutorial/module2.html
- 8. Secondary Namenode Image Source: http://www.quickmeme.com/meme/35ke38
- 9. Secondary Namenode ● Not a backup namenode ● Periodically merge the namespace image with the edit log, if edit log becomes too large ● Usually runs on a different machine than the namenode ● The secondary however always lags behind primary and hence the merged copy cannot be used in case of primary failure ● In event of primary failure, copy the primary namespace image to the secondary and run it as the new primary.
- 10. Writing a File in HDFS Image Source: Hadoop The definitive guide, 4th edition
- 11. Reading a file in HDFS Image Source: Hadoop The definitive guide, 4th edition
- 12. HDFS Block Placement Image Source: Hadoop The definitive guide, 4th edition
- 13. Small File Problem? Each file occupies namespace irrespective of file size!! Image Source: http://www.bodhtree.com/blog/2012/09/28/hadoop-how-to-manage-huge-numbers-of-small-files-in-hdfs/
- 14. Further Reading HDFS Comics :-) https://docs.google.com/open?id=0B-zw6KHOtbT4MmRkZWJjYzEtYjI3Ni00NTFjLW E0OGItYTU5OGMxYjc0N2M1
- 15. Sample:
- 16. Thank You!! Please send your questions at: [email protected]