Troubleshooting Kerberos in Hadoop: Taming the Beast
Download as PPTX, PDF6 likes2,434 views
The document provides a comprehensive overview of troubleshooting Kerberos in Hadoop, highlighting its significance as a universal authentication mechanism and its role across the Hadoop stack. It covers topics such as the workings of Kerberos, realms, principals, tools available in Hadoop, a systematic approach to diagnosing issues, and common error messages. The author, Vipin Rathor, emphasizes the importance of proper configuration and the integration of Kerberos with enterprise user management for secure Hadoop operations.
Troubleshooting Kerberos in Hadoop: Taming the Beast
- 1. 1 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Troubleshooting Kerberos In Hadoop : Taming the Beast DataWorks Summit Sept 2017
- 2. 2 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Author Profile Vipin Rathor Sr. Product Specialist (HDP Security) Contributed to Kerberos, Apache Zeppelin, Apache Atlas [email protected] / @VipinRathor46
- 3. 3 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Agenda • Why Kerberos? • Where is Kerberos used across the Hadoop Stack? • What is Kerberos & how does it work • Realms, Principals and Keytabs • Systematic Approach to Kerberos Nirvana • Tools available in Hadoop • Native Kerberos Tools / Debug Options • Kerberos Checklist • Most Common Kerberos Error Messages (& their meanings)
- 4. 4 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Why Kerberos? • Universal Authentication mechanism for Hadoop stack • Integrates with enterprise user management (e.g. Active Directory) Solves: • How can parts of a cluster trust each other (NameNodes, DataNodes, YARN, HBase, ZooKeeper...) • How can users trust the system? • How can the system trust users? • Foundation for: how can users delegate rights to applications? • Without Kerberos: your cluster has NO security Hadoop clusters are some of the largest Kerberos systems ever!!
- 5. 5 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Where is Kerberos used across the Hadoop Stack? • Ubiquitous End-User / Hadoop Service Authentication mechanism • Hadoop DelegationToken (Delegated authentication to NameNode) • != Kerberos Tickets • Bootstrapped with Kerberos authentication token • HTTP Authentication • Using SPNEGO (RFC 4559) • Via Browsers / cURL (curl --negotiate) • RPC Authentication • Using Simple Authentication & Security Layer aka SASL (!= SSL) • Java API Based Kerberos login • Using JGSS / JAAS • GSS-API (RFC 2743)
- 6. 6 © Hortonworks Inc. 2011 – 2017. All Rights Reserved What is Kerberos • Open source, Developed by MIT • Password is NEVER transmitted over wire • Central trusted authority – Key Distribution Center (KDC) • Symmetric key (common shared key) • Flavors: • MIT Kerberos • Active Directory • Heimdal Kerberos (OS X)
- 7. 7 © Hortonworks Inc. 2011 – 2017. All Rights Reserved How does Kerberos work End User - Does kinit (1 & 2) - Runs HDFS command (3 - 6) Hadoop NameNode - Starts up with nn.service.keytab - Verifies user and gives access to HDFS KDC - Provisions user keys and service keytabs (e.g. nn.service.keytab) - Provides TGT and TGS
- 8. 8 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Realms, Principals and Keytabs • Realm • User Principal • E.g. [email protected] • ken/[email protected] • ken/[email protected] • Service Principal • E.g. HTTP/[email protected] • nn/[email protected] • dn/[email protected] • dn/[email protected] • Keytabs • Service keytabs (for service) • Headless keytabs (for user)
- 9. 9 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Systematic Approach to Kerberos Nirvana • Identify the involved parties (user, service, keytabs, nodes) • Identify the stage where Kerberos is failing • Based on stage & error message, narrow down between client-side or service- side issue • Check & verify configurations for correctness using the appropriate tools • Repeat as necessary
- 10. 10 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Kerberos Tools Available in Hadoop • Kdiag • Runs a series of diagnostic checks & gives suggestions • hadoop org.apache.hadoop.security.KDiag
- 11. 11 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Kerberos Tools Available in Hadoop (cntd..) • HadoopKerberosName • Checks Auth_to_local rules (Kerberos Principal to Unix user name conversion) • hadoop org.apache.hadoop.security.HadoopKerberosName nn/[email protected]
- 12. 12 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Native Kerberos Tools / Debug Options • via command line • kinit • klist -eaf / klist –kte • kvno • kdestroy • export KRB5_TRACE=/tmp/krb5-curl.out curl -ivL --negotiate -u: "http://namenode-host:50070/webhdfs/v1/?op=LISTSTATUS" • via debug messages • export HADOOP_JAAS_DEBUG=true • export HADOOP_ROOT_LOGGER=DEBUG,console • via Java library • -Dsun.security.krb5.debug=true • -Dsun.security.spnego.debug=true • export OPTS=“$OPTS -Dsun.security.krb5.debug=true”
- 13. 13 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Kerberos Checklist • FQDN • Name Resolution • If DNS is configured, then check reverse lookup • Date/Time sync (< 5 minutes) • Configuration file - /etc/krb5.conf • Principal Names • Stale Keytabs (via kvno) • Credential Cache location (JAAS config) • Which Java suite, JCE policy • KDC log file - /var/log/kerberos/krb5kdc.log
- 14. 14 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Most Common Kerberos Error Messages (& their meaning) • <unknown client> for <unknown service> • Decrypt Integrity Check Failed • AES256 EncType not supported • Clock skew too great • Kerberos service principal not found in the database • Client not found in the database • No valid initial credential found
- 15. 15 © Hortonworks Inc. 2011 – 2017. All Rights Reserved References • http://web.mit.edu/kerberos/ • http://www.kerberos.org/software/tutorial.html • https://github.com/steveloughran/kerberos_and_hadoop
- 16. 16 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Thank you !
Editor's Notes
- #9: Realms = Domain in Active Directory KDC makes no differentiation between user principals and service principals. Goes same for the keytabs too.