Hadoop and Kerberos: the Madness Beyond the Gate
Download as PPTX, PDF•9 likes•2,341 views
The document provides a comprehensive overview of integrating Kerberos with Hadoop security, detailing key concepts such as authentication, service principals, and delegation tokens. It discusses the architectural elements necessary for securing Hadoop clusters using Kerberos, user group information, and the importance of proper configuration. Additionally, it addresses error handling and debugging techniques related to Kerberos in the context of Hadoop applications.
![© Hortonworks Inc
Hadoop RPC
@KerberosInfo(serverPrincipal = "my.kerberos.principal")
public interface MyRpc extends VersionedProtocol { … }
public class MyRpcPolicyProvider extends PolicyProvider {
public Service[] getServices() {
return new Service[] {
new Service("my.protocol.acl", MyRpc.class)
};
}
}
public class MyRpcSecurityInfo extends SecurityInfo { … }
META-INF/services/org.apache.hadoop.security.SecurityInfo
org.example.rpc.MyRpcSecurityInfo
Page 25](https://image.slidesharecdn.com/2015-09-kerberos-the-madness-150930100106-lva1-app6892/85/Hadoop-and-Kerberos-the-Madness-Beyond-the-Gate-25-320.jpg)
Hadoop and Kerberos: the Madness Beyond the Gate
- 1. © Hortonworks Inc. 2015 Hadoop and Kerberos: The madness beyond the gate Steve Loughran [email protected] @steveloughran 2015
- 2. Page 2 Me: Before Kerberos
- 3. © Hortonworks Inc. Page 3 Me: After Kerberos
- 4. © Hortonworks Inc. Page 4 HP Lovecraft Kerberos Evil lurking in New England MIT Project Athena Ancient, inhuman deities Kerberos Domain Controller Manuscripts to drive the reader insane IETF RFC 4120 Entities never spoken of aloud UserGroupInformation Doomed explorers of darkness You
- 5. © Hortonworks Inc. 2015 Leave now if you want to retain your life of naïve innocence Page 5
- 7. © Hortonworks Inc. 2015 export HADOOP_USER="root" Page 7
- 8. © Hortonworks Inc. 2015 Modern Hadoop clusters are locked down through Kerberos Page 8
- 9. © Hortonworks Inc. 2015 Discover Kerberos before Kerberos discovers you Page 9
- 10. © Hortonworks Inc. 2015 Kerberos: the gateway to hell Page 10
- 11. © Hortonworks Inc. This is not a metaphor Art: Andrés Álvarez Iglesias
- 12. © Hortonworks Inc. 2015 KP Kerberos is the gateway Page 12 Authentication Service Ticket Granting Service Principal user@REALM user/hostname@REALM (P, TGS, n1) {KP.TGS, n1}KP, {ticket(P,TGS)} KTGS Ticket(P, TGS) = (TGS, P, tstart, tend, KPT) KP {KP.S, n2}KP, {ticket(P,S)} KS {auth(P)}KP.TGS,{ticket(P,TGS)}KTGS,S,n2 KTGS Kerberos Domain ControllerClient auth(P)KP.TGS = {P, time)}KP.TGS
- 13. © Hortonworks Inc Every service is a principal alice@REALM bob@REALM oozie/ooziehost@REALM namenode/nn1@REALM hdfs/_HOST@REALM hdfs/r04s12@REALM hdfs/r04s13@REALM yarn/_HOST@REALM yarn/r04s12@REALM HTTP/_HOST@REALM Page 13 short names: alice bob oozie namenode hdfs yarn HTTP
- 14. © Hortonworks Inc. Page 14 Entering the darkness
- 16. © Hortonworks Inc. Page 16
- 17. © Hortonworks Inc. 2015 HDFS Bootstrap: Kerberos Login Page 17 shared keytab in /etc/hadoop log in to kerberos datanode/_HOST@REALM tickets for TGS namenode/nn@REALM
- 18. © Hortonworks Inc. 2015 HDFS Bootstrap: DNs register with NN Page 18 shared keytab in /etc/hadoop DN registration Ticket for namenode/nn@REALM ExportedBlockKeys Request ticket for namenode/nn@REALM namenode/nn@REALM datanode/_HOST@REALM
- 19. © Hortonworks Inc. Hadoop Tokens • Issued and tracked by individual services (HDFS, WebHDFS, Timeline Server, YARN RM, …) • Grant some form of access: Block tokens, Delegation Tokens • Can be passed on to other processes • Renewable via service APIs (RPC, HTTP) • Revocable in server via service APIs Page 19 read: O'Malley 2009, Hadoop Security Architecture
- 20. © Hortonworks Inc. 2015 HDFS IO: Block Tokens Page 20 alice@REALM Obtain ticket for namenode/nn@REALM BlockToken BlockToken BlockToken: userId, (BlockPoolId, BlockId), keyId, expiryDate, access-modes namenode/nn@REALM open("file")
- 21. © Hortonworks Inc. 2015 service/host@REALM Delegation Tokens delegate access Page 21 alice@REALM BlockToken HDFS Delegation Token BlockToken HDFS Delegation Token HDFS Delegation Token namenode/nn@REALM Token Obtain ticket for namenode/nn@REALM Request delegation token
- 22. © Hortonworks Inc. 2015 Launch Context YARN app launch Page 22 alice@REALM HDFS Delegation Token HDFS resourcemanager/rm@REALM nodemanager/_HOST@REALMalice Launch Context AM/RM HDFS AM/RM HDFS HDFS HDFS AM/RM namenode/nn@REALM Obtain ticket for resourcemanager/rm@REALM Request delegation token AM/RM Token Obtain tickvet for namenode/nn@REALM AM/RM' AM/RM' AM/RM' Refresh AM/RM
- 23. © Hortonworks Inc That which must not be named: UGI if(!UserGroupInformation.isSecurityEnabled()) { stayInALifeOfNaiveInnocence(); } else { sufferTheEnternalPainOfKerberos(); } UserGroupInformation.checkTGTAndReloginFromKeytab(); UserGroupInformation.getLoginUser() // principal logged in as UserGroupInformation.getCurrentUser() // principal acting as Page 23
- 24. © Hortonworks Inc UGI.doAs() UserGroupInformation bob = UserGroupInformation.createProxyUser("bob", UserGroupInformation.getLoginUser()); FileSystem userFS = bob.doAs( new PrivilegedExceptionAction<FileSystem>() { public FileSystem run() throws Exception { return FileSystem.get(FileSystem.getDefaultUri(), conf); } }); Page 24
- 25. © Hortonworks Inc Hadoop RPC @KerberosInfo(serverPrincipal = "my.kerberos.principal") public interface MyRpc extends VersionedProtocol { … } public class MyRpcPolicyProvider extends PolicyProvider { public Service[] getServices() { return new Service[] { new Service("my.protocol.acl", MyRpc.class) }; } } public class MyRpcSecurityInfo extends SecurityInfo { … } META-INF/services/org.apache.hadoop.security.SecurityInfo org.example.rpc.MyRpcSecurityInfo Page 25
- 26. © Hortonworks Inc IPC Server: get the current user identity Messages.KillResponse killContainer(Messages.KillRequest request) { UserGroupInformation callerUGI; try { callerUGI = UserGroupInformation.getCurrentUser(); } catch (IOException ie) { LOG.info("Error getting UGI ", ie); AuditLogger.logFailure("UNKNOWN", "Error getting UGI"); throw RPCUtil.getRemoteException(ie); } … Page 26
- 27. © Hortonworks Inc IPC Server: Authorize String user = callerUGI.getShortUserName(); if (!checkAccess(callerUGI, MODIFY)) { AuditLog.unauthorized(user, KILL_CONTAINER_REQUEST, "User doesn't have permissions to " + MODIFY); throw RPCUtil.getRemoteException( new AccessControlException( + user + " lacks access " + MODIFY_APP.name())); } AuditLog.authorized(user, KILL_CONTAINER_REQUEST) Page 27
- 28. © Hortonworks Inc. 2015 SASL: RFC4422 Page 28
- 29. © Hortonworks Inc. REST: SPNEGO (+ Delegation tokens) Page 29 • Jersey + java.net • httpclient? “if lucky it'll work” HADOOP-11825: Move timeline client Jersey+Kerberos+UGI support into a public implementation
- 31. © Hortonworks Inc. Error messages to fear Art: Andrés Álvarez Iglesias Failure unspecified at GSS-API level (Checksum failed) No valid credentials provided (Failed to find any Kerberos tgt) Server not found in Kerberos database Clock skew too great Principal not found No valid credentials provided (Illegal key size)
- 32. © Hortonworks Inc. Topics Avoided Not Covered • Zookeeper • JAAS • Trying to use HTTPS in a YARN application • Trying to use Full REST in a YARN application • System properties to debug Kerberos & SPNEGO • Group management • HADOOP_PROXY_USER Page 32
- 33. © Hortonworks Inc. gitbook.com/@steveloughran Questions? Art: Andrés Álvarez Iglesias
- 34. © Hortonworks Inc. Zookeeper • SASL to negotiate security: System.setProperty("zookeeper.sasl.client", "true"); • Permissions are not transitive down the tree Page 34 List<ACL> perms = new ArrayList<>(); if (UserGroupInformation.isSecurityEnabled()) { perms(new ACL(ZooDefs.Perms.ALL, ZooDefs.Ids.AUTH_IDS)); perms.add(new ACL(ZooDefs.Perms.READ,ZooDefs.Ids.ANYONE_ID_UNSAFE)); } else { perms.add(new ACL(ZooDefs.Perms.ALL, ZooDefs.Ids.ANYONE_ID_UNSAFE)); } zk.createPath(path, null, perms, CreateMode.PERSISTENT);
- 35. © Hortonworks Inc System Properties for debugging -Dsun.security.krb5.debug=true -Dsun.security.spnego.debug=true export HADOOP_JAAS_DEBUG=true Page 35
- 36. © Hortonworks Inc. Services • RPC authentication via annotations & metadata in JAR • YARN Web UIs: rely on RM proxy for authentication • Authentication != Authorization • Add audit logs on service endpoints • YARN services: come up with a token refresh strategy: keytab everywhere; keytab in AM; update from client Page 36
- 37. © Hortonworks Inc. JAAS • Java Authentication and Authorization Service • Core Kerberos classes and types (Principal) • Text files to configure –Different for different JVMs –Need to double escape for windows paths • UGI handles setting up a JAAS context & logging in Page 37
- 38. © Hortonworks Inc. Glossary • Simple Authentication and Security Layer (SASL) • GSSAPI Generic Security Service Application Program Interface (RFC-2743+ others) • JAAS: Java Authentication and Authorization Service • Simple and Protected GSSAPI Negotiation Mechanism (SPNEGO) Page 38
Editor's Notes
- #4: Th
- #7: Enough people like Dunkin Donut's decaf coffee that you can buy it for home use —and supermarkets will stock it next to the MacDonalds coffee.
- #8: This is your get out clause. Turn off encryption. Users are who they claim to be; the environment variable HADOOP_USER can change it on a whim.
- #9: ..which is why production clusters are all locked down with kerberos. Callout: this doesn't cover authorization/access control (exception: Hadoop IPC acls), wire encryption, HTTPS or data encryption.
- #10: So you can't ignore Kerberos. You only get a choice about when to encounter it -early on in your coding and testing -during final integration tests -in late night support calls.
- #12: Photo: https://www.flickr.com/photos/doctorserone/4635167170/ Andrés Álvarez Iglesias
- #13: The KDC is managed by the enterprise security team. They are either paranoid about security, or your organisation is 0wned by everyone from Anonymous to North Korea. They don't trust you, they don't trust Hadoop, and make the rest of the network ops people seem welcoming. You will need to work with these people.
- #30: AuthenticatedURL DelegationTokenAuthenticatedURL org.apache.hadoop.hdfs.web.URLConnectionFactory org/apache/spark/deploy/history/yarn/rest in SPARK-1537
- #31: There is a mini KDC, "MiniKDC" in the Hadoop codebase. I've used this in the YARN-913 registry work; its good for verifying that you got through the permissions logic, and for learning various acronyms. And at the end of the run you get tests that Jenkins can run every build. But I've embraced testing against kerberized VMs, where you do the work of creating keytabs, filling in the configuration files, requiring SPENGO authed web browsers, having your command line account kinit in regularly, services having tokens expire, etc. etc. Why? Because its what the real world is like. L
- #32: Error messages with UGI are usually a sign of trouble Photo: https://www.flickr.com/photos/doctorserone/4635167170/ Andrés Álvarez Iglesias
- #34: Photo: https://www.flickr.com/photos/doctorserone/4635167170/ Andrés Álvarez Iglesias