Apache Hadoop for System Administrators
2 likes4,358 views
This document provides an introduction to Apache Hadoop for system administrators. It discusses the core components of Hadoop including MapReduce, HDFS, and YARN. It emphasizes the operational challenges of Hadoop and provides best practices for monitoring, performance tuning, security, and disaster recovery. Key recommendations include using health checks, setting resource controls, implementing quotas, partitioning disks, enabling Kerberos, and being proactive rather than reactive in operations.
![ Map (“transform”)
– Perl:
@items=(1,2,3,4,5);
sub sqr {return $_**2);
print join(‘,’,map(sqr,@items));
1,4,9,16,25
– Python:
items = [1,2,3,4,5]
def sqr(x) : return x**2
print list(map(sqr,items))
[1, 4, 9, 16, 25]](https://image.slidesharecdn.com/2013-11-08-lisa-131108112603-phpapp01/85/Apache-Hadoop-for-System-Administrators-8-320.jpg)
![ Reduce (“compress” or “fold”)
– Perl
use List::Util qw/reduce/;
@items=(1,4,9,16,25);
print reduce {$a>$b ? $a:$b} @items;
25
– Python
from functools import reduce
items = [1,4,9,16,25]
print reduce ((lambda x,y: x if
(x>y) else y), items)
25](https://image.slidesharecdn.com/2013-11-08-lisa-131108112603-phpapp01/85/Apache-Hadoop-for-System-Administrators-9-320.jpg)
Apache Hadoop for System Administrators
- 1. Apache Hadoop for System Administrators Allen Wittenauer
- 5. Needed some place to sit before lunch?
- 6. An Extremely Quick & Incomplete Intro to Hadoop
- 8. Map (“transform”) – Perl: @items=(1,2,3,4,5); sub sqr {return $_**2); print join(‘,’,map(sqr,@items)); 1,4,9,16,25 – Python: items = [1,2,3,4,5] def sqr(x) : return x**2 print list(map(sqr,items)) [1, 4, 9, 16, 25]
- 9. Reduce (“compress” or “fold”) – Perl use List::Util qw/reduce/; @items=(1,4,9,16,25); print reduce {$a>$b ? $a:$b} @items; 25 – Python from functools import reduce items = [1,4,9,16,25] print reduce ((lambda x,y: x if (x>y) else y), items) 25
- 18. Name Node DN DN DN DN DN D D D D M M R R D D D D M M R R D D D D M M R R D D D D M M R R D D D D M M R R TT TT TT TT TT Job Tracker
- 19. Hadoop isn’t designed for system administrators and/or support staff.
- 20. “Hadoop is not a developer problem; it’s an operations problem.” -- Hadoop vendor ex-employee
- 23. tail’ing the logs won’t tell you the whole story.
- 24. %
- 30. LinkedIn’s Configuration – 30+ Health Checks per Grid Masters, canary report, daily fsck, etc Nagios – 10+ Health Checks per DC LDAP, Kerberos, etc ... – Cross-DC Nagios Server Checks ZK Warn: 5% down nodes Panic: 30% down HDFS: 20% Free Space Gateway home dir: 10% free space ... VD NN JT Compute Nodes AZ GW
- 31. Health Check Script – “OK” - good status – “ERROR (message)” - bad status mapred.healthChecker.script.path Consider checking ... – critical software – ownership & permissions – network connection speed – drive count – file system space – RO file systems – IO errors – missing memory
- 33. Use the tools most of your user’s code is written in! Pig – testfile: 100 – Code: A = load 'testfile' using PigStorage(',') as (i: int); B = foreach C generate i; C = distinct B; dump C; – Output: (100)
- 34. Reactive
- 35. Proactive
- 37. JobTracker Memory Resource Controls – Limit jobs stored in JT heap: mapred.jobtracker.completeuserjobs.maximum – Limit total # of job tasks: mapred.jobtracker.maxtasks.per.job Job Memory Resource Controls – Scheduler-level: mapred.cluster.*.memory.mb – TT-level: auto-calculated based upon MR slot counts & scheduler level settings – MR Job-level: mapred.job.*.memory.mb – Linux only: /proc memory calculator and task killer
- 38. “I set the heap to 1G but my process ran out of memory?”
- 39. Treat HDFS like any other multi-tenant FS
- 40. Quota everything – Yes, including /tmp – No “show me all quotas” functionality dfsadmin -setQuota dfsadmin -setSpaceQuota Be consistent: – /user/* all get same quota Be flexible: – Make another dir for user’s to store big projects (e.g., /project) Be smart: – Have a policy that content in /tmp gets deleted after X days. Automate this! – Build reporting that shows files that are replicated less than 3 times
- 41. Compute Node Disk Partitioning as Protective Measure
- 42. root partitioning 20 GB /, ... 200 GB task space (rest) HDFS non-root partinioning 5 GB swap 200 GB task space (rest) HDFS
- 43. Security!
- 44. Queue Level ACLs mapred-queue-acls.xml – users – groups – netgroups Service Level ACLs – hosts – users – groups – netgroups hadoop-policy.xml – Limitation: Web services are all or nothing! :( – Be aware: Hadoop uses ephemeral ports all over the place! :(
- 45. Kerberos!
- 47. Corp IT Active Directory @CORP krbtgt/GRID@CORP krbtgt/host@GRID krbtgt/service@GRID Password Client Node Grid Realm @GRID krbtgt/user@CORP krbtgt/GRID@CORP Hadoop Services
- 51. http://data.linkedin.com/opensource/white-elephant
- 52. Fonzi: http://www.flickr.com/photos/elzey/7224689810 Captain Obvious by artist Stuart McGhee. http://stuartmcghee.com/ Ant on flower: http://www.flickr.com/photos/bolonski/6116358907 Ant Colony: http://www.flickr.com/photos/klearchos/2821230516 Ant Queen: http://commons.wikimedia.org/wiki/ File:Camponotus_crispulus_queen_ant.jpg Canary: http://www.flickr.com/photos/nathan_and_jenny/2454127424 Mona Lisa: Leonardo Da Vinci White Elephant: http://data.linkedin.com/opensource/white-elephant Ecce Homo: – Elías García Martínez (original) – Cecilia Giménez (restored)
- 53. Thanks! Contact: Twitter: @_a__w_ Email: [email protected] More info: Quora: www.quora.com/user/allenwittenauer SlideShare: www.slideshare.net/allenwittenauer