SlideShare a Scribd company logo

Deep Dive into Keystone Tokens and Lessons Learned

Download as PPTX, PDF
Priti Desai
Priti Desai

This document provides an in-depth exploration of keystone token formats in OpenStack, detailing UUID, PKI, PKIZ, and Fernet formats, including their generation and validation workflows. Each section outlines the configuration settings, pros and cons, and specific examples of token usage and attributes. The document aims to guide users in selecting the appropriate token format based on their deployment requirements.

Engineering
1 of 62
Downloaded 202 times
1
2
3
4
5
6
7
8
9
10
Most read
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
Deep Dive into Keystone Tokens and Lessons Learned Priti Desai & Brad Pokorny
Who are we? Priti Desai Advisory Software Engineer, IBM Brad Pokorny Principal Software Engineer, Symantec Deep Dive into Keystone Tokens and Lessons Learned
What token format should we configure in our OpenStack Deployment?
Token Formats Deep Dive into Keystone Tokens and Lessons Learned UUID PKI PKIZ Fernet
UUID Deep Dive into Keystone Tokens and Lessons Learned
UUID • Simplest and Most Light Weight • Version 4 UUID • Configuration in keystone.conf : [token] provider = keystone.token.providers.uuid.Provider Deep Dive into Keystone Tokens and Lessons Learned
UUID – Token Generation Workflow Keystone KVS Request Token with: • User Name • Password • Project Name Identity Resources Assignment Catalog User Validation Retrieves User ID Token Project Validation Retrieves Project ID and Domain ID Retrieves Roles for this User on the Project or Domain Returns Failure if the User does not have any Role Retrieves Services and Endpoints for all the services Bundles Identity, Resource, Assignment, and Catalog information into Token Payload Creates Token ID : uuid.uuid4().hex Store them in SQL/KVS: • Token ID • Expiration • Valid • User ID • Extra Token Generation Workflow
Sample UUID Token in SQL Backend Deep Dive into Keystone Tokens and Lessons Learned id: f10700e71ff045cbb850072a0bd6a4e6 expires: 2015-10-08 21:18:43 extra: {"token_data": {"token": {"methods": ["password"], "roles": [{"id": "1688449cf1df44839b10a41e3d9b09dd", "name": "admin"}], "expires_at": "2015-10- 08T21:18:43.995255Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "423d45cddec84170be365e0b31a1b15f", "name": "admin"}, "extras": {}, "user": {"domain": {"id": "default", "name": "Default"}, "id": "1334f3ed7eb2483b91b8192ba043b580", "name": "admin"}, "audit_ids": ["bI1EMzqUQM2sqFimOtIPpQ"], "issued_at": "2015-10-08T20:18:43.995284Z"}}, "user": {"domain": {"id": "default", "name": "Default"}, "id": "1334f3ed7eb2483b91b8192ba043b580", "name": "admin"}, "key": "f10700e71ff045cbb850072a0bd6a4e6", "token_version": "v3.0", "tenant": {"domain": {"id": "default", "name": "Default"}, "id": "423d45cddec84170be365e0b31a1b15f", "name": "admin"}, "metadata": {"roles": ["1688449cf1df44839b10a41e3d9b09dd"]}} valid: 1 trust_id: NULL user_id: 1334f3ed7eb2483b91b8192ba043b580
UUID – Keystone Token Validation Workflow Parse Token and Retrieve Metadata Validate Token with: GET v3/auth/tokens • X-Subject-Token • X-Auth-Token Retrieves Token payload from token backend KVS/SQL Read cached token reference and parse: • User ID • Project ID • Audit ID • Token Expiry Token KVS Valid? Current Time < Expiry Time Token Not Found Token Not Found Is Revoked? Token Not Found HTTP/1.1 200 OK Yes No No No Yes Yes Check if a token matches any revocation events Check if a token is expired, current time is calculated in UTC Token Validation Workflow
UUID – Keystone Token Revocation Workflow Revoke by Audit ID Revoke Token with: DELETE v3/auth/tokens • X-Subject-Token • X-Auth-Token Before revoking a token, validate it based on Token Validation Workflow Validate X-Subject-Token Audit ID? Revoke by Token Expiry Create Revoke Event with: Audit ID Revoke At Issued Before Prune Expired Events Set valid to False Yes No Filter existing revocation events based on Revoke At Sample Revocation Event: { "audit_id": "HVvI0d-cTD21yatAfQc4IQ", "issued_before”: "2015-10-24T21:20:45.000000Z" }, Token Revocation Workflow Create Revoke Event with: User ID Project ID Revoke At Issued Before Token Expiry Token KVS
UUID Across Multiple Data Centers Users Groups Domains Projects Roles Catalog Assignments Users Groups Domains Projects Roles Catalog Assignments MySQL Replication (Database is always in sync) LDAP Replication (Directory Tree is always in sync) Tokens KVS UUID Tokens Tokens KVS UUID Tokens Keystone UUID - Multiple Data Centers Nova Keystone Middleware Keystone Nova Keystone Middleware US-EASTUS-WEST Request Token UUID Token Token Validation VM Instance Token Validation Token Not Found Token Found Token Not Found
Pros and cons • Pros – Simplest and Smallest Token Format – Recommended for Simple OpenStack Deployment • Cons – Persistent Token Format – Token validation can only be done by Identity service – Not feasible for multiple OpenStack deployments Deep Dive into Keystone Tokens and Lessons Learned
PKI/PKIZ Deep Dive into Keystone Tokens and Lessons Learned
PKI • Cryptographically Encrypted Signed Document using X509 Standards • CMS • Converted to custom URL- Safe format • Compressed PKI • Prefixed with “PKIZ” Deep Dive into Keystone Tokens and Lessons Learned PKIZ
PKI/PKIZ Configuration - Certificates • Signing Key (signing_key.pem) : • Generate private key in PEM format • Signing Certificate (signing_cert.pem) : • Generate CSR using Signing Key • Submit CSR to CA • Receive Certificate from CA • Certificate Authority Certificate (ca.pem) Deep Dive into Keystone Tokens and Lessons Learned
PKI/PKIZ Configuration • Configuration in keystone.conf : [token] provider = keystone.token.providers.[pki|pkiz].Provider [signing] certfile = /etc/keystone/ssl/certs/signing_cert.pem keyfile = /etc/keystone/ssl/private/signing_key.pem ca_certs = /etc/keystone/ssl/certs/ca.pem Deep Dive into Keystone Tokens and Lessons Learned
PKI/PKIZ – Token Generation Workflow Validate Identity, Resource, and Assignment Request Token with: • User Name • Password • Project Name Token Generation Workflow Create JSON Token Payload Sign JSON Payload with Signing Key and Signing Certificate openssl cms –sign –outform PEM Convert it to UTF-8 Convert CMS Signed Token in PEM format to custom URL Safe format: • “/” replaced with “-” • Deleted: “n”, “----BEGIN CMS----”,“----END CMS---- ” Compress using zlib Convert it to UTF-8 Base64 URL Safe Append Prefix PKIZ PKI PKIZ Store Token into SQL/KVS
Sample PKI Token in SQL Backend Deep Dive into Keystone Tokens and Lessons Learned id: b460fec2efcd0d803e2baf48d3bcd72b expires: 2015-10-09 20:07:36 extra: {"token_data": {"token": {"methods": ["password"], "roles": [{"id": "1688449cf1df44839b10a41e3d9b09dd", "name": "admin"}], "expires_at": "2015-10-09T20:07:36.656431Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "423d45cddec84170be365e0b31a1b15f", "name": "admin"}, "extras": {}, "user": {"domain": {"id": "default", "name": "Default"}, "id": "1334f3ed7eb2483b91b8192ba043b580", "name": "admin"}, "audit_ids": ["8dh07HudSh6rHoU1G9bs-Q"], "issued_at": "2015-10-09T19:07:36.656460Z"}}, "user": {"domain": {"id": "default", "name": "Default"}, "id": "1334f3ed7eb2483b91b8192ba043b580", "name": "admin"}, "key": "MIIDiwYJKoZIhvcNAQcCoIIDfDCCA3gCAQExDTALBglghkgBZQMEAgEwggHZBgkqhkiG9w0BBwGgggHKBIIBxnsidG9rZW4iOnsib WV0aG9kcyI6WyJwYXNzd29yZCJdLCJyb2xlcyI6W3siaWQiOiIxNjg4NDQ5Y2YxZGY0NDgzOWIxMGE0MWUzZDliMDlkZCIsIm5hb WUiOiJhZG1pbiJ9XSwiZXhwaXJlc19hdCI6IjIwMTUtMTAtMDlUMjA6MDc6MzYuNjU2NDMxWiIsInByb2plY3QiOnsiZG9tYWluIjp7I mlkIjoiZGVmYXVsdCIsIm5hbWUiOiJEZWZhdWx0In0sImlkIjoiNDIzZDQ1Y2RkZWM4NDE3MGJlMzY1ZTBiMzFhMWIxNWYiLCJuY W1lIjo…", "token_version": "v3.0", "tenant": {"domain": {"id": "default", "name": "Default"}, "id": "423d45cddec84170be365e0b31a1b15f", "name": "admin"}, "metadata": {"roles": ["1688449cf1df44839b10a41e3d9b09dd"]}} valid: 1 trust_id: NULL user_id: 1334f3ed7eb2483b91b8192ba043b580
Sample PKIZ Token in SQL Backend Deep Dive into Keystone Tokens and Lessons Learned id: c48321ac51a903b07c264ac3e80809c6 expires: 2015-10-12 18:45:23 extra: {"token_data": {"token": {"methods": ["password"], "roles": [{"id": "1688449cf1df44839b10a41e3d9b09dd", "name": "admin"}], "expires_at": "2015-10-12T18:45:23.806229Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "423d45cddec84170be365e0b31a1b15f", "name": "admin"}, "extras": {}, "user": {"domain": {"id": "default", "name": "Default"}, "id": "1334f3ed7eb2483b91b8192ba043b580", "name": "admin"}, "audit_ids": ["kKmQzTuxSnCN9vo3bzxErw"], "issued_at": "2015-10-12T17:45:23.806257Z"}}, "user": {"domain": {"id": "default", "name": "Default"}, "id": "1334f3ed7eb2483b91b8192ba043b580", "name": "admin"}, "key": "PKIZ_eJxtlMtyqzgQhvc8xexTqcPFdsLiLCQEWCSCgAGBdgZscbVxDOHy9CMnc6mpGlWpSmqpW39_Uuv5WTRo2tj9wy CHxiN35dqjqybi9eb6DuE7ZLd7_WxtAd6MtR1wP7PT5PxJE2F7U53WYH5D5qZbc53OSkeWPoo3hdrU7VQwhe5JBReo 71GWv72WT2vLPRk62_XuDmt_T9sZku-veT-xPfUaEk…", "token_version": "v3.0", "tenant": {"domain": {"id": "default", "name": "Default"}, "id": "423d45cddec84170be365e0b31a1b15f", "name": "admin"}, "metadata": {"roles": ["1688449cf1df44839b10a41e3d9b09dd"]}} valid: 1 trust_id: NULL user_id: 1334f3ed7eb2483b91b8192ba043b580
PKI/PKIZ – Token Validation Workflow Parse Token and Retrieve Metadata Validate Token with: GET v3/auth/tokens • X-Subject-Token • X-Auth-Token Retrieves Token reference from token backend KVS/SQL Read cached token reference and parse: • User ID • Project ID • Audit ID • Token Expiry Token KVS Valid? Current Time < Expiry Time Token Not Found Token Not Found Is Revoked? Token Not Found HTTP/1.1 200 OK Yes No No No Yes Yes Check if a token matches any revocation events Check if a token is expired, current time is calculated in UTC Token Validation Workflow Unique ID of X-Subject-Token Hash PKI Token with the pre- configured hashing algorithm
PKI/PKIZ – Keystone Token Revocation Workflow Revoke by Audit ID Revoke Token with: DELETE v3/auth/tokens • X-Subject-Token • X-Auth-Token Before revoking a token, validate it based on Token Validation Workflow Validate X-Subject-Token Audit ID? Revoke by Token Expiry Create Revoke Event with: Audit ID Revoke At Issued Before Prune Expired Events Set valid to False Yes No Filter existing revocation events based on Revoke At Sample Revocation Event: { "audit_id": "HVvI0d-cTD21yatAfQc4IQ", "issued_before”: "2015-10-24T21:20:45.000000Z" }, Token Revocation Workflow Create Revoke Event with: User ID Project ID Revoke At Issued Before Token Expiry Token KVS
PKI/PKIZ - Across Multiple Data Centers Users Groups Domains Projects Roles Catalog Assignments Users Groups Domains Projects Roles Catalog Assignments MySQL Replication (Database is always in sync) LDAP Replication (Directory Tree is always in sync) Tokens KVS PKI/PKIZ Tokens Tokens KVS PKI/PKIZ Tokens Keystone PKI/PKIZ - Multiple Data Centers Nova Keystone Middleware Keystone Nova Keystone Middleware US-EASTUS-WEST Request Token PKI/PKIZ Token Token Validation VM Instance Token Validation VM Instance
Pros and Cons PKI • Pros – Token validation without a request to Keystone • Cons – Larger than standard HTTP Header Size – Complex configuration – base64 –d <pki_token – Not truly feasible for multiple OpenStack Deployments PKIZ • Pros – Token validation without a request to Keystone • Cons – Still Larger than standard HTTP Header Size – Similar to PKI Deep Dive into Keystone Tokens and Lessons Learned
FERNET Deep Dive into Keystone Tokens and Lessons Learned
Fernet • Cryptographic Authentication Method – Fernet • Symmetric Key Encryption • Fernet Keys stored in /etc/keystone/fernet-keys/ – Encrypted with Primary Fernet Key – Decrypted with a list of Fernet Keys Deep Dive into Keystone Tokens and Lessons Learned
• Configuration in keystone.conf : [token] provider = keystone.token.providers.fernet.Provider [fernet_tokens] key_repository = /etc/keystone/fernet-keys/ max_active_keys = <number of keys> # default is 3 Deep Dive into Keystone Tokens and Lessons Learned Fernet Configuration
Fernet Keys • Fernet Key File - 256 bits 83b4sCF0Q4pb3aNWJYtSdtdaH8PMA_5dlN7OswXKbvE= xf3vxf8xb0!tCx8a[xddxa3V%x8bRvxd7Zx1fxc3xccx03xfe]x94xdexcexb3x05xcanxf1 Deep Dive into Keystone Tokens and Lessons Learned SHA256 HMAC Signing Key (128 bits) AES Encrypting Key (128 bits)
Fernet Keys • Fernet Key File Name - Integers starting from 0 • ls /etc/keystone/fernet-keys => 0 1 2 3 4 • Type 1: Primary Key – Encrypt and Decrypt – Key file named with the highest index • Type 2: Secondary Key – Only Decrypt – Lowest Index < Secondary Key File Name < Highest Index • Type 3: Staged Key – Decrypt and Next In Line to become Primary Key – Key file named with lowest index (of 0) Deep Dive into Keystone Tokens and Lessons Learned
Fernet Key Rotation 0 1 Primary KeyStaged Key No Secondary Key 2 Primary Key0Staged Key 1 Secondary Key 3 Primary Key21 Secondary Key 0Staged Key Secondary Key Rotate Rotate
Fernet – Token Generation Workflow Token Generation Workflow HMACFernet Token Version Current Timestamp IV Cipher Text Token Payload: Version User ID Methods Project ID Expiry Time Audit ID Padding Encrypted using Encrypting Key Signed using Signing Key
Sample Fernet Token in SQL Backend Deep Dive into Keystone Tokens and Lessons Learned gAAAAABWLUzy0dxSNo2--K- 3trDutnX7LpUpv3us0crQIl8BDHLLd3lR3F243VwnYpNJHIaUiPEE2roYJJNA- SwBe1swDcr6MYaFR1t9ZYcYF4GRqDm3N9_1EGgXgICbzE_GuUVidG4gky0Cv8 f1nwD7XM26NRh59VEnt2iVTAxlnvAICJDeK5k
Fernet – Keystone Token Validation Workflow Determine the Version from the Token Payload Validate Token with: GET v3/auth/tokens • X-Subject-Token • X-Auth-Token Re-inflate token with “=” and return token with correct padding Version: Fixed Versioning by Keystone: • Unscoped Payload : 0 • Domain Scoped Payload : 1 • Project Scoped Payload : 2 Restore Padding Is Revoked? Token Not Found HTTP/1.1 200 OK No No Yes Yes Check if a token matches any revocation events Check if a token is expired, current time is calculated in UTC Token Validation Workflow Decrypt using Fernet Keys to retrieve Token Payload Disassemble payload to determine validation fields For Project Scoped Token: • User ID Project ID • Methods Token Expiry • Audit ID Current Time < Expiry Time Token Not Found No
Fernet – Keystone Token Revocation Workflow Revoke by Audit ID Revoke Token with: DELETE v3/auth/tokens • X-Subject-Token • X-Auth-Token Before revoking a token, validate it based on Token Validation Workflow Validate X-Subject-Token Audit ID? Revoke by Token Expiry Create Revoke Event with: Audit ID Revoke At Issued Before Prune Expired Events Set valid to False Yes No Filter existing revocation events based on Revoke At Sample Revocation Event: { "audit_id": "HVvI0d-cTD21yatAfQc4IQ", "issued_before”: "2015-10-24T21:20:45.000000Z" }, Token Revocation Workflow Create Revoke Event with: User ID Project ID Revoke At Issued Before Token Expiry Token KVS
Fernet - Across Multiple Data Centers Users Groups Domains Projects Roles Catalog Assignments Users Groups Domains Projects Roles Catalog Assignments MySQL Replication (Database is always in sync) LDAP Replication (Directory Tree is always in sync) Keystone Fernet - Multiple Data Centers Nova Keystone Middleware Keystone Nova Keystone Middleware US-EASTUS-WEST Request Token Fernet Token Token Validation VM Instance Token Validation VM Instance Validate Fernet Token Validate Fernet Token
Pros and cons • Pros – No persistence – Reasonable Token Size – Multiple Data Center • Cons – Token validation impacted by the number of revocation events Deep Dive into Keystone Tokens and Lessons Learned
Fernet Token Validation 11.17 46.406 83.654 124.974 163.529 234.398 376.604 510.058 0 100 200 300 400 500 600 0 100 200 300 400 500 600 700 800 900 1000 1100 1200 1300 1400 1500 2000 3000 4000 Time(ms) Revocation Events Revocation Events Impact on Validation Time Response Time (ms) Deep Dive into Keystone Tokens and Lessons Learned
Fernet Token Validation 89.46 21.55 11.95 8 5.77 1.96 0 10 20 30 40 50 60 70 80 90 100 0 100 200 300 400 500 600 700 800 900 1000 1100 1200 1300 1400 1500 2000 3000 4000 ValidationsPerSecond Revocation Events Revocation Events Impact on Validation Requests Token Validation Requests Deep Dive into Keystone Tokens and Lessons Learned
What token format should we configure in our OpenStack Deployment? Fernet for Multiple OpenStack Deployments with minimal Revocation Events Deep Dive into Keystone Tokens and Lessons Learned
HORIZON AND TOKENS Deep Dive into Keystone Tokens and Lessons Learned
How horizon uses tokens Deep Dive into Keystone Tokens and Lessons Learned • Tokens for each logged in user • Unscoped token and project scoped token • Token reuse • Reduced transaction load on Keystone • Stored in the session • Configurable token storage methods • Local memory cache • Cookie backend • Memcache • Database • Cached Database
Cookie backend Deep Dive into Keystone Tokens and Lessons Learned • Currently the devstack default • Token stored in browser cookie • Secure cookies in production, use https • CSRF_COOKIE_SECURE = True • SESSION_COOKIE_SECURE = True • http://docs.openstack.org/developer/horizon/topics/settings.html • http://docs.openstack.org/security-guide/dashboard/cookies.html • Highly scalable • The dreaded boot back to login
Cookie backend Deep Dive into Keystone Tokens and Lessons Learned • The dreaded boot back to login • Now sign in…
Cookie backend Deep Dive into Keystone Tokens and Lessons Learned • And you see… • Cookie overflow!
memcache backend Deep Dive into Keystone Tokens and Lessons Learned • Allows storage of larger token sizes • Tokens stored on server side • Requires memcached • Can be used with backing DB • http://docs.openstack.org/developer/horizon/topics/deployment.html
Token hashing Deep Dive into Keystone Tokens and Lessons Learned • Hashed in Django OpenStack Auth (DOA) • Keeps stored token data small • Currently not working for PKI tokens • New config in Liberty to disable • OPENSTACK_TOKEN_HASH_ENABLED • PKI - Will increase memcache storage requirement
Multiregion and tokens Deep Dive into Keystone Tokens and Lessons Learned • Service regions vs. Authentication regions • Service regions in Keystone catalog • Auth regions specified in AVAILABLE_REGIONS • UUID, PKI, and PKIZ Tokens don’t work across auth regions • Token replication is infeasible • But Fernet tokens work between Authentication regions! Service Region Authentication Region
Horizon and Fernet Deep Dive into Keystone Tokens and Lessons Learned • Yes, Fernet tokens work with Horizon • Liberty and beyond – No patches necessary • Kilo – Needs a patch for DOA • https://review.openstack.org/#/c/169994/
V3 domains Domain Scoped Token Project Scoped Token Deep Dive into Keystone Tokens and Lessons Learned "auth": { "identity": { }, “scope”: { ”domain": { “name”: “Default” } } } "auth": { "identity": { }, “scope”: { “project”: { ”domain": { “name”: “Default” }, “name”: “ProjectA” } } } • Extra token for Horizon
V3 domains Deep Dive into Keystone Tokens and Lessons Learned • Requires changes in Django OpenStack Auth and Horizon • Planned for Mitaka • Info on usage (a bit out of date): • http://www.symantec.com/connect/blogs/how-use-horizon-keystone-v3 • Domains patches: • https://review.openstack.org/#/c/148082/ • https://review.openstack.org/#/c/141153/ • https://review.openstack.org/#/c/196328/
Will fernet tokens solve all our problems? Deep Dive into Keystone Tokens and Lessons Learned • Smaller token size • No persistence for tokens • Seamless authentication across regions • Performance issues with token revocation
Thank You !!! Questions ?
References • Token: https://clubpenguincheatscitya4.files.wordpress.com/2011/08/1_token.jpg • Key to Cloud: https://www.hc1.com/wp- content/uploads/2013/10/14916002_cloud_computing_and_storage_security_concept_blue_glossy_cloud_icon_with_with_golden_key_in_keyhole _.jpg • User Icon: http://findicons.com/files/icons/1075/scrap/300/user_2.png • Password: http://icons.iconarchive.com/icons/sbstnblnd/plateau/512/Apps-password-icon.png • UUID: https://plugins.qgis.org/static/cache/21/c0/21c0d3fedb5bf42ff8a6a11712595124.png • PKI: http://www.zaheerspeaks.com/wp-content/uploads/2009/10/PKI-Certificate.gif • PKIZ: http://i571.photobucket.com/albums/ss153/rijal_abror/pun170-winzip-file-compress-icon59.gif • Identity: https://www.innopay.com/assets/Uploads/icon-digitalidentity-232x232.png Deep Dive into Keystone Tokens and Lessons Learned
EXTRA Deep Dive into Keystone Tokens and Lessons Learned
What is an Openstack Token? Deep Dive into Keystone Tokens and Lessons Learned Key to OpenStack Cloud
How can I generate a token? Deep Dive into Keystone Tokens and Lessons Learned Keystone curl -s POST https://keystone.com/v3/auth/tokens "auth": { "identity": { "methods": [ "password" ], "password": { "user": { ”domain": { ”name": ”MyDomain.com” }, ”name": ”PritiDesai", "password": "secretsecret” }
How can I generate a token? Deep Dive into Keystone Tokens and Lessons Learned Keystone curl -s POST https://keystone.com/v3/auth/tokens "auth": { "identity": { "methods": [ ”token" ], ”token": { ”id": ”e8079ab…” } }
Token generated using password Deep Dive into Keystone Tokens and Lessons Learned Header: X-Subject-Token: a740dcd6f3fc404aaaf556b9cbd2f994 Body: { "token": { "methods": [ "password" ], "expires_at": "2015-10-05T20:25:03.180741Z", "extras": {}, "user": { "domain": { "id": "default", "name": "Default" }, "id": "1334f3ed7eb2483b91b8192ba043b580", "name": ”smith" }, "auth": { "identity": { "methods": [ "password" ], "password": { "user": { ”domain": { ”name": ”Default” }, ”name": ”Smith", "password": "secretsecret” }
Token generated using token Deep Dive into Keystone Tokens and Lessons Learned Header: X-Subject-Token: 3fb7b3b0a0a8489882f07fdb9cd2a990 Body: { "token": { "issued_at": "2015-10-05T19:40:38.943250Z", "audit_ids": [ "4vNgmP5cQk6sMpPiw7EnCg", "HFwMKdDrSCOq-MAtkXKTlw" ], "user": { "name": ”smith", "id": "1334f3ed7eb2483b91b8192ba043b580", "domain": { "name": "Default", "id": "default" } }, "extras": {}, "auth": { "identity": { "methods": [ ”token" ], ”token": { “id”: “a7409b” } } }
Identity Token Path User OpenStack Service Step 1: Obtain unscoped token with credentials POST v3/auth/tokens Keystone Step 2: Discover projects you have access to GET v3/users/<user_id>/projects Step 3: Obtain project scoped token either with your credentials or unscoped token from step 1. Step 4: Invoke the target service by sending requests to endpoints in token from step 3 Step 5: Validate roles and access metadata in token with Keystone service or Keystone Middleware Step 6: Serve API request Step 7: Return response
Token Creation 0 10 20 30 40 50 60 70 80 90 UUID PKI PKIZ Fernet Time Per Request Time Per Request Deep Dive into Keystone Tokens and Lessons Learned 0 5 10 15 20 UUID PKI PKIZ Fernet Requests Per Sec Requests Per Sec
Token Validation 0 2 4 6 8 10 12 14 UUID PKI PKIZ Fernet Time Per Request Time Per Request Deep Dive into Keystone Tokens and Lessons Learned 0 20 40 60 80 100 UUID PKI PKIZ Fernet Requests Per Sec Requests Per Sec
Configurable token hashing https://github.com/openstack/django_openstack_auth/blob/master/openstack_auth/user.py Deep Dive into Keystone Tokens and Lessons Learned

More Related Content

What's hot (20)

PDF
Prometheus on EKS
Jo Hoon
 
PDF
Community Openstack 구축 사례
Open Source Consulting
 
PDF
Hands-on Helm
Docker, Inc.
 
PDF
Service Function Chaining in Openstack Neutron
Michelle Holley
 
PDF
OpenStack 개요 및 활용 사례 @ Community Open Camp with Microsoft
Ian Choi
 
PDF
オンライン物理バックアップの排他モードと非排他モードについて（第15回PostgreSQLアンカンファレンス@オンライン 発表資料）
NTT DATA Technology & Innovation
 
PPTX
Vault
Jean-Philippe Bélanger
 
PPTX
OVN - Basics and deep dive
Trinath Somanchi
 
PDF
Kubernetes monitoring using prometheus stack
Juraj Hantak
 
PPTX
Building secure applications with keycloak
Abhishek Koserwal
 
PPTX
Adopting OpenTelemetry
Vincent Behar
 
PPTX
Learn nginx in 90mins
Larry Cai
 
PDF
멀티클라우드 Service Mesh
Jeong-Ho Na
 
PDF
Harbor RegistryのReplication機能
Masanori Nara
 
PDF
30分でRHEL6 High Availability Add-Onを超絶的に理解しよう!
Etsuji Nakai
 
PDF
Prometheus + Grafana = Awesome Monitoring
Henrique Galafassi Dalssaso
 
PDF
[OpenInfra Days Korea 2018] (Track 2) Neutron LBaaS 어디까지 왔니? - Octavia 소개
OpenStack Korea Community
 
PDF
Monitoring Kubernetes with Prometheus
Grafana Labs
 
PPTX
[OpenStack 하반기 스터디] Interoperability with ML2: LinuxBridge, OVS and SDN
OpenStack Korea Community
 
PDF
Kubernetes
erialc_w
 
Prometheus on EKS
Jo Hoon
 
Community Openstack 구축 사례
Open Source Consulting
 
Hands-on Helm
Docker, Inc.
 
Service Function Chaining in Openstack Neutron
Michelle Holley
 
OpenStack 개요 및 활용 사례 @ Community Open Camp with Microsoft
Ian Choi
 
オンライン物理バックアップの排他モードと非排他モードについて（第15回PostgreSQLアンカンファレンス@オンライン 発表資料）
NTT DATA Technology & Innovation
 
Vault
Jean-Philippe Bélanger
 
OVN - Basics and deep dive
Trinath Somanchi
 
Kubernetes monitoring using prometheus stack
Juraj Hantak
 
Building secure applications with keycloak
Abhishek Koserwal
 
Adopting OpenTelemetry
Vincent Behar
 
Learn nginx in 90mins
Larry Cai
 
멀티클라우드 Service Mesh
Jeong-Ho Na
 
Harbor RegistryのReplication機能
Masanori Nara
 
30分でRHEL6 High Availability Add-Onを超絶的に理解しよう!
Etsuji Nakai
 
Prometheus + Grafana = Awesome Monitoring
Henrique Galafassi Dalssaso
 
[OpenInfra Days Korea 2018] (Track 2) Neutron LBaaS 어디까지 왔니? - Octavia 소개
OpenStack Korea Community
 
Monitoring Kubernetes with Prometheus
Grafana Labs
 
[OpenStack 하반기 스터디] Interoperability with ML2: LinuxBridge, OVS and SDN
OpenStack Korea Community
 
Kubernetes
erialc_w
 

Viewers also liked (20)

PDF
OpenStack keystone identity service
openstackindia
 
PDF
Keystone fernet token
Yuki Nishiwaki
 
PDF
OpenStack Identity - Keystone (liberty) by Lorenzo Carnevale and Silvio Tavilla
Lorenzo Carnevale
 
PDF
Openstack Keystone
Kamesh Pemmaraju
 
PPTX
Keystone - Openstack Identity Service
Prasad Mukhedkar
 
PPTX
OpenStack Keystone
Deepti Ramakrishna
 
PDF
Keystone: Federated
jamielennox
 
ODP
OpenStack keystone identity service
openstackindia
 
PDF
Making Glance tasks work for you - OpenStack Summit May 2015 Vancouver
Brian Rosmaita
 
PPTX
Couch to open_stack_keystone
ProfessionalVMware
 
PDF
8 Key Facts about the Keystone Pipeline
U.S. Chamber of Commerce
 
PDF
Open stack networking_101_part-1
yfauser
 
PPTX
OpenStack Storage Overview
Bharat Kumar Kobagana
 
PDF
Identity as a Service: a missing gap for moving enterprise applications in In...
Hoang Tri Vo
 
PDF
Stateless authentication with OAuth 2 and JWT - JavaZone 2015
Alvaro Sanchez-Mariscal
 
PPTX
IDaaS. The Now Big Thing
Nishant Kaushik
 
PDF
Diffie-Hellman key exchange
hughpearse
 
PPTX
Building IAM for OpenStack
Steve Martinelli
 
PPTX
IdM vs. IDaaS
Drew Koenig
 
PPTX
AWS IAM and security
Erik Paulsson
 
OpenStack keystone identity service
openstackindia
 
Keystone fernet token
Yuki Nishiwaki
 
OpenStack Identity - Keystone (liberty) by Lorenzo Carnevale and Silvio Tavilla
Lorenzo Carnevale
 
Openstack Keystone
Kamesh Pemmaraju
 
Keystone - Openstack Identity Service
Prasad Mukhedkar
 
OpenStack Keystone
Deepti Ramakrishna
 
Keystone: Federated
jamielennox
 
OpenStack keystone identity service
openstackindia
 
Making Glance tasks work for you - OpenStack Summit May 2015 Vancouver
Brian Rosmaita
 
Couch to open_stack_keystone
ProfessionalVMware
 
8 Key Facts about the Keystone Pipeline
U.S. Chamber of Commerce
 
Open stack networking_101_part-1
yfauser
 
OpenStack Storage Overview
Bharat Kumar Kobagana
 
Identity as a Service: a missing gap for moving enterprise applications in In...
Hoang Tri Vo
 
Stateless authentication with OAuth 2 and JWT - JavaZone 2015
Alvaro Sanchez-Mariscal
 
IDaaS. The Now Big Thing
Nishant Kaushik
 
Diffie-Hellman key exchange
hughpearse
 
Building IAM for OpenStack
Steve Martinelli
 
IdM vs. IDaaS
Drew Koenig
 
AWS IAM and security
Erik Paulsson
 
Ad

Similar to Deep Dive into Keystone Tokens and Lessons Learned (20)

PPTX
OpenStack Toronto Meetup - Keystone 101
Steve Martinelli
 
PDF
.NET Fest 2019. Eran Stiller. Create Your Own Serverless PKI with .NET & Azur...
NETFest
 
PDF
SSL Everywhere!
Simon Haslam
 
PDF
WebLogic in Practice: SSL Configuration
Simon Haslam
 
PDF
Issue certificates with PyOpenSSL
Pau Freixes
 
PPTX
IBM Spectrum Scale Authentication For Object - Deep Dive
Smita Raut
 
PPTX
The Dynamic Duo of Puppet and Vault tame SSL Certificates - Puppet Camps Cent...
Nick Maludy
 
PDF
The Dynamic Duo of Puppet and Vault tame SSL Certificates, Nick Maludy
Puppet
 
PPTX
Kerberos Survival Guide: Columbus 2015
J.D. Wade
 
PPTX
Kerberos Survival Guide: SharePointalooza
J.D. Wade
 
PPTX
Client certificate validation in windows 8
Ashish Agrawal
 
PDF
Kerberos survival guide
J.D. Wade
 
PPT
Java Cert Pki
phanleson
 
PPTX
Create Your Own Serverless PKI with .NET & Azure Key Vault
Eran Stiller
 
PDF
2016 pycontw web api authentication
Micron Technology
 
PDF
#MoreCrypto : Introduction to TLS
Olle E Johansson
 
PDF
ID連携入門 (実習編) - Security Camp 2016
Nov Matake
 
PDF
Kerberos Survival Guide - St. Louis Day of .Net
J.D. Wade
 
PPTX
Hashitalks 2021 - How the Dynamic Duo of Vault and Puppet Tame SSL Certificates
Nick Maludy
 
PPTX
SPS Ozarks 2012: Kerberos Survival Guide
J.D. Wade
 
OpenStack Toronto Meetup - Keystone 101
Steve Martinelli
 
.NET Fest 2019. Eran Stiller. Create Your Own Serverless PKI with .NET & Azur...
NETFest
 
SSL Everywhere!
Simon Haslam
 
WebLogic in Practice: SSL Configuration
Simon Haslam
 
Issue certificates with PyOpenSSL
Pau Freixes
 
IBM Spectrum Scale Authentication For Object - Deep Dive
Smita Raut
 
The Dynamic Duo of Puppet and Vault tame SSL Certificates - Puppet Camps Cent...
Nick Maludy
 
The Dynamic Duo of Puppet and Vault tame SSL Certificates, Nick Maludy
Puppet
 
Kerberos Survival Guide: Columbus 2015
J.D. Wade
 
Kerberos Survival Guide: SharePointalooza
J.D. Wade
 
Client certificate validation in windows 8
Ashish Agrawal
 
Kerberos survival guide
J.D. Wade
 
Java Cert Pki
phanleson
 
Create Your Own Serverless PKI with .NET & Azure Key Vault
Eran Stiller
 
2016 pycontw web api authentication
Micron Technology
 
#MoreCrypto : Introduction to TLS
Olle E Johansson
 
ID連携入門 (実習編) - Security Camp 2016
Nov Matake
 
Kerberos Survival Guide - St. Louis Day of .Net
J.D. Wade
 
Hashitalks 2021 - How the Dynamic Duo of Vault and Puppet Tame SSL Certificates
Nick Maludy
 
SPS Ozarks 2012: Kerberos Survival Guide
J.D. Wade
 
Ad

Recently uploaded (20)

PPTX
Shinkawa Proposal to meet Vibration API670.pptx
AchmadBashori2
 
PPTX
Lecture 1 Shell and Tube Heat exchanger-1.pptx
mailforillegalwork
 
PPTX
MobileComputingMANET2023 MobileComputingMANET2023.pptx
masterfake98765
 
PPTX
Server Side Web Development Unit 1 of Nodejs.pptx
sneha852132
 
PDF
Basic_Concepts_in_Clinical_Biochemistry_2018كيمياء_عملي.pdf
AdelLoin
 
PPTX
VITEEE 2026 Exam Details , Important Dates
SonaliSingh127098
 
PDF
GTU Civil Engineering All Semester Syllabus.pdf
Vimal Bhojani
 
PDF
Biomechanics of Gait: Engineering Solutions for Rehabilitation (www.kiu.ac.ug)
publication11
 
DOC
MRRS Strength and Durability of Concrete
CivilMythili
 
PPTX
What is Shot Peening | Shot Peening is a Surface Treatment Process
Vibra Finish
 
PPTX
Damage of stability of a ship and how its change .pptx
ehamadulhaque
 
PDF
Viol_Alessandro_Presentazione_prelaurea.pdf
dsecqyvhbowrzxshhf
 
PPTX
fatigue in aircraft structures-221113192308-0ad6dc8c.pptx
aviatecofficial
 
PDF
Set Relation Function Practice session 24.05.2025.pdf
DrStephenStrange4
 
PPTX
Arduino Based Gas Leakage Detector Project
CircuitDigest
 
PPTX
Green Building & Energy Conservation ppt
Sagar Sarangi
 
PDF
Ethics and Trustworthy AI in Healthcare – Governing Sensitive Data, Profiling...
AlqualsaDIResearchGr
 
PPTX
The Role of Information Technology in Environmental Protectio....pptx
nallamillisriram
 
PDF
PORTFOLIO Golam Kibria Khan — architect with a passion for thoughtful design...
MasumKhan59
 
PDF
International Journal of Information Technology Convergence and services (IJI...
ijitcsjournal4
 
Shinkawa Proposal to meet Vibration API670.pptx
AchmadBashori2
 
Lecture 1 Shell and Tube Heat exchanger-1.pptx
mailforillegalwork
 
MobileComputingMANET2023 MobileComputingMANET2023.pptx
masterfake98765
 
Server Side Web Development Unit 1 of Nodejs.pptx
sneha852132
 
Basic_Concepts_in_Clinical_Biochemistry_2018كيمياء_عملي.pdf
AdelLoin
 
VITEEE 2026 Exam Details , Important Dates
SonaliSingh127098
 
GTU Civil Engineering All Semester Syllabus.pdf
Vimal Bhojani
 
Biomechanics of Gait: Engineering Solutions for Rehabilitation (www.kiu.ac.ug)
publication11
 
MRRS Strength and Durability of Concrete
CivilMythili
 
What is Shot Peening | Shot Peening is a Surface Treatment Process
Vibra Finish
 
Damage of stability of a ship and how its change .pptx
ehamadulhaque
 
Viol_Alessandro_Presentazione_prelaurea.pdf
dsecqyvhbowrzxshhf
 
fatigue in aircraft structures-221113192308-0ad6dc8c.pptx
aviatecofficial
 
Set Relation Function Practice session 24.05.2025.pdf
DrStephenStrange4
 
Arduino Based Gas Leakage Detector Project
CircuitDigest
 
Green Building & Energy Conservation ppt
Sagar Sarangi
 
Ethics and Trustworthy AI in Healthcare – Governing Sensitive Data, Profiling...
AlqualsaDIResearchGr
 
The Role of Information Technology in Environmental Protectio....pptx
nallamillisriram
 
PORTFOLIO Golam Kibria Khan — architect with a passion for thoughtful design...
MasumKhan59
 
International Journal of Information Technology Convergence and services (IJI...
ijitcsjournal4
 

Deep Dive into Keystone Tokens and Lessons Learned

  • 1. Deep Dive into Keystone Tokens and Lessons Learned Priti Desai & Brad Pokorny
  • 2. Who are we? Priti Desai Advisory Software Engineer, IBM Brad Pokorny Principal Software Engineer, Symantec Deep Dive into Keystone Tokens and Lessons Learned
  • 3. What token format should we configure in our OpenStack Deployment?
  • 4. Token Formats Deep Dive into Keystone Tokens and Lessons Learned UUID PKI PKIZ Fernet
  • 5. UUID Deep Dive into Keystone Tokens and Lessons Learned
  • 6. UUID • Simplest and Most Light Weight • Version 4 UUID • Configuration in keystone.conf : [token] provider = keystone.token.providers.uuid.Provider Deep Dive into Keystone Tokens and Lessons Learned
  • 7. UUID – Token Generation Workflow Keystone KVS Request Token with: • User Name • Password • Project Name Identity Resources Assignment Catalog User Validation Retrieves User ID Token Project Validation Retrieves Project ID and Domain ID Retrieves Roles for this User on the Project or Domain Returns Failure if the User does not have any Role Retrieves Services and Endpoints for all the services Bundles Identity, Resource, Assignment, and Catalog information into Token Payload Creates Token ID : uuid.uuid4().hex Store them in SQL/KVS: • Token ID • Expiration • Valid • User ID • Extra Token Generation Workflow
  • 8. Sample UUID Token in SQL Backend Deep Dive into Keystone Tokens and Lessons Learned id: f10700e71ff045cbb850072a0bd6a4e6 expires: 2015-10-08 21:18:43 extra: {"token_data": {"token": {"methods": ["password"], "roles": [{"id": "1688449cf1df44839b10a41e3d9b09dd", "name": "admin"}], "expires_at": "2015-10- 08T21:18:43.995255Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "423d45cddec84170be365e0b31a1b15f", "name": "admin"}, "extras": {}, "user": {"domain": {"id": "default", "name": "Default"}, "id": "1334f3ed7eb2483b91b8192ba043b580", "name": "admin"}, "audit_ids": ["bI1EMzqUQM2sqFimOtIPpQ"], "issued_at": "2015-10-08T20:18:43.995284Z"}}, "user": {"domain": {"id": "default", "name": "Default"}, "id": "1334f3ed7eb2483b91b8192ba043b580", "name": "admin"}, "key": "f10700e71ff045cbb850072a0bd6a4e6", "token_version": "v3.0", "tenant": {"domain": {"id": "default", "name": "Default"}, "id": "423d45cddec84170be365e0b31a1b15f", "name": "admin"}, "metadata": {"roles": ["1688449cf1df44839b10a41e3d9b09dd"]}} valid: 1 trust_id: NULL user_id: 1334f3ed7eb2483b91b8192ba043b580
  • 9. UUID – Keystone Token Validation Workflow Parse Token and Retrieve Metadata Validate Token with: GET v3/auth/tokens • X-Subject-Token • X-Auth-Token Retrieves Token payload from token backend KVS/SQL Read cached token reference and parse: • User ID • Project ID • Audit ID • Token Expiry Token KVS Valid? Current Time < Expiry Time Token Not Found Token Not Found Is Revoked? Token Not Found HTTP/1.1 200 OK Yes No No No Yes Yes Check if a token matches any revocation events Check if a token is expired, current time is calculated in UTC Token Validation Workflow
  • 10. UUID – Keystone Token Revocation Workflow Revoke by Audit ID Revoke Token with: DELETE v3/auth/tokens • X-Subject-Token • X-Auth-Token Before revoking a token, validate it based on Token Validation Workflow Validate X-Subject-Token Audit ID? Revoke by Token Expiry Create Revoke Event with: Audit ID Revoke At Issued Before Prune Expired Events Set valid to False Yes No Filter existing revocation events based on Revoke At Sample Revocation Event: { "audit_id": "HVvI0d-cTD21yatAfQc4IQ", "issued_before”: "2015-10-24T21:20:45.000000Z" }, Token Revocation Workflow Create Revoke Event with: User ID Project ID Revoke At Issued Before Token Expiry Token KVS
  • 11. UUID Across Multiple Data Centers Users Groups Domains Projects Roles Catalog Assignments Users Groups Domains Projects Roles Catalog Assignments MySQL Replication (Database is always in sync) LDAP Replication (Directory Tree is always in sync) Tokens KVS UUID Tokens Tokens KVS UUID Tokens Keystone UUID - Multiple Data Centers Nova Keystone Middleware Keystone Nova Keystone Middleware US-EASTUS-WEST Request Token UUID Token Token Validation VM Instance Token Validation Token Not Found Token Found Token Not Found
  • 12. Pros and cons • Pros – Simplest and Smallest Token Format – Recommended for Simple OpenStack Deployment • Cons – Persistent Token Format – Token validation can only be done by Identity service – Not feasible for multiple OpenStack deployments Deep Dive into Keystone Tokens and Lessons Learned
  • 13. PKI/PKIZ Deep Dive into Keystone Tokens and Lessons Learned
  • 14. PKI • Cryptographically Encrypted Signed Document using X509 Standards • CMS • Converted to custom URL- Safe format • Compressed PKI • Prefixed with “PKIZ” Deep Dive into Keystone Tokens and Lessons Learned PKIZ
  • 15. PKI/PKIZ Configuration - Certificates • Signing Key (signing_key.pem) : • Generate private key in PEM format • Signing Certificate (signing_cert.pem) : • Generate CSR using Signing Key • Submit CSR to CA • Receive Certificate from CA • Certificate Authority Certificate (ca.pem) Deep Dive into Keystone Tokens and Lessons Learned
  • 16. PKI/PKIZ Configuration • Configuration in keystone.conf : [token] provider = keystone.token.providers.[pki|pkiz].Provider [signing] certfile = /etc/keystone/ssl/certs/signing_cert.pem keyfile = /etc/keystone/ssl/private/signing_key.pem ca_certs = /etc/keystone/ssl/certs/ca.pem Deep Dive into Keystone Tokens and Lessons Learned
  • 17. PKI/PKIZ – Token Generation Workflow Validate Identity, Resource, and Assignment Request Token with: • User Name • Password • Project Name Token Generation Workflow Create JSON Token Payload Sign JSON Payload with Signing Key and Signing Certificate openssl cms –sign –outform PEM Convert it to UTF-8 Convert CMS Signed Token in PEM format to custom URL Safe format: • “/” replaced with “-” • Deleted: “n”, “----BEGIN CMS----”,“----END CMS---- ” Compress using zlib Convert it to UTF-8 Base64 URL Safe Append Prefix PKIZ PKI PKIZ Store Token into SQL/KVS
  • 18. Sample PKI Token in SQL Backend Deep Dive into Keystone Tokens and Lessons Learned id: b460fec2efcd0d803e2baf48d3bcd72b expires: 2015-10-09 20:07:36 extra: {"token_data": {"token": {"methods": ["password"], "roles": [{"id": "1688449cf1df44839b10a41e3d9b09dd", "name": "admin"}], "expires_at": "2015-10-09T20:07:36.656431Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "423d45cddec84170be365e0b31a1b15f", "name": "admin"}, "extras": {}, "user": {"domain": {"id": "default", "name": "Default"}, "id": "1334f3ed7eb2483b91b8192ba043b580", "name": "admin"}, "audit_ids": ["8dh07HudSh6rHoU1G9bs-Q"], "issued_at": "2015-10-09T19:07:36.656460Z"}}, "user": {"domain": {"id": "default", "name": "Default"}, "id": "1334f3ed7eb2483b91b8192ba043b580", "name": "admin"}, "key": "MIIDiwYJKoZIhvcNAQcCoIIDfDCCA3gCAQExDTALBglghkgBZQMEAgEwggHZBgkqhkiG9w0BBwGgggHKBIIBxnsidG9rZW4iOnsib WV0aG9kcyI6WyJwYXNzd29yZCJdLCJyb2xlcyI6W3siaWQiOiIxNjg4NDQ5Y2YxZGY0NDgzOWIxMGE0MWUzZDliMDlkZCIsIm5hb WUiOiJhZG1pbiJ9XSwiZXhwaXJlc19hdCI6IjIwMTUtMTAtMDlUMjA6MDc6MzYuNjU2NDMxWiIsInByb2plY3QiOnsiZG9tYWluIjp7I mlkIjoiZGVmYXVsdCIsIm5hbWUiOiJEZWZhdWx0In0sImlkIjoiNDIzZDQ1Y2RkZWM4NDE3MGJlMzY1ZTBiMzFhMWIxNWYiLCJuY W1lIjo…", "token_version": "v3.0", "tenant": {"domain": {"id": "default", "name": "Default"}, "id": "423d45cddec84170be365e0b31a1b15f", "name": "admin"}, "metadata": {"roles": ["1688449cf1df44839b10a41e3d9b09dd"]}} valid: 1 trust_id: NULL user_id: 1334f3ed7eb2483b91b8192ba043b580
  • 19. Sample PKIZ Token in SQL Backend Deep Dive into Keystone Tokens and Lessons Learned id: c48321ac51a903b07c264ac3e80809c6 expires: 2015-10-12 18:45:23 extra: {"token_data": {"token": {"methods": ["password"], "roles": [{"id": "1688449cf1df44839b10a41e3d9b09dd", "name": "admin"}], "expires_at": "2015-10-12T18:45:23.806229Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "423d45cddec84170be365e0b31a1b15f", "name": "admin"}, "extras": {}, "user": {"domain": {"id": "default", "name": "Default"}, "id": "1334f3ed7eb2483b91b8192ba043b580", "name": "admin"}, "audit_ids": ["kKmQzTuxSnCN9vo3bzxErw"], "issued_at": "2015-10-12T17:45:23.806257Z"}}, "user": {"domain": {"id": "default", "name": "Default"}, "id": "1334f3ed7eb2483b91b8192ba043b580", "name": "admin"}, "key": "PKIZ_eJxtlMtyqzgQhvc8xexTqcPFdsLiLCQEWCSCgAGBdgZscbVxDOHy9CMnc6mpGlWpSmqpW39_Uuv5WTRo2tj9wy CHxiN35dqjqybi9eb6DuE7ZLd7_WxtAd6MtR1wP7PT5PxJE2F7U53WYH5D5qZbc53OSkeWPoo3hdrU7VQwhe5JBReo 71GWv72WT2vLPRk62_XuDmt_T9sZku-veT-xPfUaEk…", "token_version": "v3.0", "tenant": {"domain": {"id": "default", "name": "Default"}, "id": "423d45cddec84170be365e0b31a1b15f", "name": "admin"}, "metadata": {"roles": ["1688449cf1df44839b10a41e3d9b09dd"]}} valid: 1 trust_id: NULL user_id: 1334f3ed7eb2483b91b8192ba043b580
  • 20. PKI/PKIZ – Token Validation Workflow Parse Token and Retrieve Metadata Validate Token with: GET v3/auth/tokens • X-Subject-Token • X-Auth-Token Retrieves Token reference from token backend KVS/SQL Read cached token reference and parse: • User ID • Project ID • Audit ID • Token Expiry Token KVS Valid? Current Time < Expiry Time Token Not Found Token Not Found Is Revoked? Token Not Found HTTP/1.1 200 OK Yes No No No Yes Yes Check if a token matches any revocation events Check if a token is expired, current time is calculated in UTC Token Validation Workflow Unique ID of X-Subject-Token Hash PKI Token with the pre- configured hashing algorithm
  • 21. PKI/PKIZ – Keystone Token Revocation Workflow Revoke by Audit ID Revoke Token with: DELETE v3/auth/tokens • X-Subject-Token • X-Auth-Token Before revoking a token, validate it based on Token Validation Workflow Validate X-Subject-Token Audit ID? Revoke by Token Expiry Create Revoke Event with: Audit ID Revoke At Issued Before Prune Expired Events Set valid to False Yes No Filter existing revocation events based on Revoke At Sample Revocation Event: { "audit_id": "HVvI0d-cTD21yatAfQc4IQ", "issued_before”: "2015-10-24T21:20:45.000000Z" }, Token Revocation Workflow Create Revoke Event with: User ID Project ID Revoke At Issued Before Token Expiry Token KVS
  • 22. PKI/PKIZ - Across Multiple Data Centers Users Groups Domains Projects Roles Catalog Assignments Users Groups Domains Projects Roles Catalog Assignments MySQL Replication (Database is always in sync) LDAP Replication (Directory Tree is always in sync) Tokens KVS PKI/PKIZ Tokens Tokens KVS PKI/PKIZ Tokens Keystone PKI/PKIZ - Multiple Data Centers Nova Keystone Middleware Keystone Nova Keystone Middleware US-EASTUS-WEST Request Token PKI/PKIZ Token Token Validation VM Instance Token Validation VM Instance
  • 23. Pros and Cons PKI • Pros – Token validation without a request to Keystone • Cons – Larger than standard HTTP Header Size – Complex configuration – base64 –d <pki_token – Not truly feasible for multiple OpenStack Deployments PKIZ • Pros – Token validation without a request to Keystone • Cons – Still Larger than standard HTTP Header Size – Similar to PKI Deep Dive into Keystone Tokens and Lessons Learned
  • 24. FERNET Deep Dive into Keystone Tokens and Lessons Learned
  • 25. Fernet • Cryptographic Authentication Method – Fernet • Symmetric Key Encryption • Fernet Keys stored in /etc/keystone/fernet-keys/ – Encrypted with Primary Fernet Key – Decrypted with a list of Fernet Keys Deep Dive into Keystone Tokens and Lessons Learned
  • 26. • Configuration in keystone.conf : [token] provider = keystone.token.providers.fernet.Provider [fernet_tokens] key_repository = /etc/keystone/fernet-keys/ max_active_keys = <number of keys> # default is 3 Deep Dive into Keystone Tokens and Lessons Learned Fernet Configuration
  • 27. Fernet Keys • Fernet Key File - 256 bits 83b4sCF0Q4pb3aNWJYtSdtdaH8PMA_5dlN7OswXKbvE= xf3vxf8xb0!tCx8a[xddxa3V%x8bRvxd7Zx1fxc3xccx03xfe]x94xdexcexb3x05xcanxf1 Deep Dive into Keystone Tokens and Lessons Learned SHA256 HMAC Signing Key (128 bits) AES Encrypting Key (128 bits)
  • 28. Fernet Keys • Fernet Key File Name - Integers starting from 0 • ls /etc/keystone/fernet-keys => 0 1 2 3 4 • Type 1: Primary Key – Encrypt and Decrypt – Key file named with the highest index • Type 2: Secondary Key – Only Decrypt – Lowest Index < Secondary Key File Name < Highest Index • Type 3: Staged Key – Decrypt and Next In Line to become Primary Key – Key file named with lowest index (of 0) Deep Dive into Keystone Tokens and Lessons Learned
  • 29. Fernet Key Rotation 0 1 Primary KeyStaged Key No Secondary Key 2 Primary Key0Staged Key 1 Secondary Key 3 Primary Key21 Secondary Key 0Staged Key Secondary Key Rotate Rotate
  • 30. Fernet – Token Generation Workflow Token Generation Workflow HMACFernet Token Version Current Timestamp IV Cipher Text Token Payload: Version User ID Methods Project ID Expiry Time Audit ID Padding Encrypted using Encrypting Key Signed using Signing Key
  • 31. Sample Fernet Token in SQL Backend Deep Dive into Keystone Tokens and Lessons Learned gAAAAABWLUzy0dxSNo2--K- 3trDutnX7LpUpv3us0crQIl8BDHLLd3lR3F243VwnYpNJHIaUiPEE2roYJJNA- SwBe1swDcr6MYaFR1t9ZYcYF4GRqDm3N9_1EGgXgICbzE_GuUVidG4gky0Cv8 f1nwD7XM26NRh59VEnt2iVTAxlnvAICJDeK5k
  • 32. Fernet – Keystone Token Validation Workflow Determine the Version from the Token Payload Validate Token with: GET v3/auth/tokens • X-Subject-Token • X-Auth-Token Re-inflate token with “=” and return token with correct padding Version: Fixed Versioning by Keystone: • Unscoped Payload : 0 • Domain Scoped Payload : 1 • Project Scoped Payload : 2 Restore Padding Is Revoked? Token Not Found HTTP/1.1 200 OK No No Yes Yes Check if a token matches any revocation events Check if a token is expired, current time is calculated in UTC Token Validation Workflow Decrypt using Fernet Keys to retrieve Token Payload Disassemble payload to determine validation fields For Project Scoped Token: • User ID Project ID • Methods Token Expiry • Audit ID Current Time < Expiry Time Token Not Found No
  • 33. Fernet – Keystone Token Revocation Workflow Revoke by Audit ID Revoke Token with: DELETE v3/auth/tokens • X-Subject-Token • X-Auth-Token Before revoking a token, validate it based on Token Validation Workflow Validate X-Subject-Token Audit ID? Revoke by Token Expiry Create Revoke Event with: Audit ID Revoke At Issued Before Prune Expired Events Set valid to False Yes No Filter existing revocation events based on Revoke At Sample Revocation Event: { "audit_id": "HVvI0d-cTD21yatAfQc4IQ", "issued_before”: "2015-10-24T21:20:45.000000Z" }, Token Revocation Workflow Create Revoke Event with: User ID Project ID Revoke At Issued Before Token Expiry Token KVS
  • 34. Fernet - Across Multiple Data Centers Users Groups Domains Projects Roles Catalog Assignments Users Groups Domains Projects Roles Catalog Assignments MySQL Replication (Database is always in sync) LDAP Replication (Directory Tree is always in sync) Keystone Fernet - Multiple Data Centers Nova Keystone Middleware Keystone Nova Keystone Middleware US-EASTUS-WEST Request Token Fernet Token Token Validation VM Instance Token Validation VM Instance Validate Fernet Token Validate Fernet Token
  • 35. Pros and cons • Pros – No persistence – Reasonable Token Size – Multiple Data Center • Cons – Token validation impacted by the number of revocation events Deep Dive into Keystone Tokens and Lessons Learned
  • 37. Fernet Token Validation 89.46 21.55 11.95 8 5.77 1.96 0 10 20 30 40 50 60 70 80 90 100 0 100 200 300 400 500 600 700 800 900 1000 1100 1200 1300 1400 1500 2000 3000 4000 ValidationsPerSecond Revocation Events Revocation Events Impact on Validation Requests Token Validation Requests Deep Dive into Keystone Tokens and Lessons Learned
  • 38. What token format should we configure in our OpenStack Deployment? Fernet for Multiple OpenStack Deployments with minimal Revocation Events Deep Dive into Keystone Tokens and Lessons Learned
  • 39. HORIZON AND TOKENS Deep Dive into Keystone Tokens and Lessons Learned
  • 40. How horizon uses tokens Deep Dive into Keystone Tokens and Lessons Learned • Tokens for each logged in user • Unscoped token and project scoped token • Token reuse • Reduced transaction load on Keystone • Stored in the session • Configurable token storage methods • Local memory cache • Cookie backend • Memcache • Database • Cached Database
  • 41. Cookie backend Deep Dive into Keystone Tokens and Lessons Learned • Currently the devstack default • Token stored in browser cookie • Secure cookies in production, use https • CSRF_COOKIE_SECURE = True • SESSION_COOKIE_SECURE = True • http://docs.openstack.org/developer/horizon/topics/settings.html • http://docs.openstack.org/security-guide/dashboard/cookies.html • Highly scalable • The dreaded boot back to login
  • 42. Cookie backend Deep Dive into Keystone Tokens and Lessons Learned • The dreaded boot back to login • Now sign in…
  • 43. Cookie backend Deep Dive into Keystone Tokens and Lessons Learned • And you see… • Cookie overflow!
  • 44. memcache backend Deep Dive into Keystone Tokens and Lessons Learned • Allows storage of larger token sizes • Tokens stored on server side • Requires memcached • Can be used with backing DB • http://docs.openstack.org/developer/horizon/topics/deployment.html
  • 45. Token hashing Deep Dive into Keystone Tokens and Lessons Learned • Hashed in Django OpenStack Auth (DOA) • Keeps stored token data small • Currently not working for PKI tokens • New config in Liberty to disable • OPENSTACK_TOKEN_HASH_ENABLED • PKI - Will increase memcache storage requirement
  • 46. Multiregion and tokens Deep Dive into Keystone Tokens and Lessons Learned • Service regions vs. Authentication regions • Service regions in Keystone catalog • Auth regions specified in AVAILABLE_REGIONS • UUID, PKI, and PKIZ Tokens don’t work across auth regions • Token replication is infeasible • But Fernet tokens work between Authentication regions! Service Region Authentication Region
  • 47. Horizon and Fernet Deep Dive into Keystone Tokens and Lessons Learned • Yes, Fernet tokens work with Horizon • Liberty and beyond – No patches necessary • Kilo – Needs a patch for DOA • https://review.openstack.org/#/c/169994/
  • 48. V3 domains Domain Scoped Token Project Scoped Token Deep Dive into Keystone Tokens and Lessons Learned "auth": { "identity": { }, “scope”: { ”domain": { “name”: “Default” } } } "auth": { "identity": { }, “scope”: { “project”: { ”domain": { “name”: “Default” }, “name”: “ProjectA” } } } • Extra token for Horizon
  • 49. V3 domains Deep Dive into Keystone Tokens and Lessons Learned • Requires changes in Django OpenStack Auth and Horizon • Planned for Mitaka • Info on usage (a bit out of date): • http://www.symantec.com/connect/blogs/how-use-horizon-keystone-v3 • Domains patches: • https://review.openstack.org/#/c/148082/ • https://review.openstack.org/#/c/141153/ • https://review.openstack.org/#/c/196328/
  • 50. Will fernet tokens solve all our problems? Deep Dive into Keystone Tokens and Lessons Learned • Smaller token size • No persistence for tokens • Seamless authentication across regions • Performance issues with token revocation
  • 52. References • Token: https://clubpenguincheatscitya4.files.wordpress.com/2011/08/1_token.jpg • Key to Cloud: https://www.hc1.com/wp- content/uploads/2013/10/14916002_cloud_computing_and_storage_security_concept_blue_glossy_cloud_icon_with_with_golden_key_in_keyhole _.jpg • User Icon: http://findicons.com/files/icons/1075/scrap/300/user_2.png • Password: http://icons.iconarchive.com/icons/sbstnblnd/plateau/512/Apps-password-icon.png • UUID: https://plugins.qgis.org/static/cache/21/c0/21c0d3fedb5bf42ff8a6a11712595124.png • PKI: http://www.zaheerspeaks.com/wp-content/uploads/2009/10/PKI-Certificate.gif • PKIZ: http://i571.photobucket.com/albums/ss153/rijal_abror/pun170-winzip-file-compress-icon59.gif • Identity: https://www.innopay.com/assets/Uploads/icon-digitalidentity-232x232.png Deep Dive into Keystone Tokens and Lessons Learned
  • 53. EXTRA Deep Dive into Keystone Tokens and Lessons Learned
  • 54. What is an Openstack Token? Deep Dive into Keystone Tokens and Lessons Learned Key to OpenStack Cloud
  • 55. How can I generate a token? Deep Dive into Keystone Tokens and Lessons Learned Keystone curl -s POST https://keystone.com/v3/auth/tokens "auth": { "identity": { "methods": [ "password" ], "password": { "user": { ”domain": { ”name": ”MyDomain.com” }, ”name": ”PritiDesai", "password": "secretsecret” }
  • 56. How can I generate a token? Deep Dive into Keystone Tokens and Lessons Learned Keystone curl -s POST https://keystone.com/v3/auth/tokens "auth": { "identity": { "methods": [ ”token" ], ”token": { ”id": ”e8079ab…” } }
  • 57. Token generated using password Deep Dive into Keystone Tokens and Lessons Learned Header: X-Subject-Token: a740dcd6f3fc404aaaf556b9cbd2f994 Body: { "token": { "methods": [ "password" ], "expires_at": "2015-10-05T20:25:03.180741Z", "extras": {}, "user": { "domain": { "id": "default", "name": "Default" }, "id": "1334f3ed7eb2483b91b8192ba043b580", "name": ”smith" }, "auth": { "identity": { "methods": [ "password" ], "password": { "user": { ”domain": { ”name": ”Default” }, ”name": ”Smith", "password": "secretsecret” }
  • 58. Token generated using token Deep Dive into Keystone Tokens and Lessons Learned Header: X-Subject-Token: 3fb7b3b0a0a8489882f07fdb9cd2a990 Body: { "token": { "issued_at": "2015-10-05T19:40:38.943250Z", "audit_ids": [ "4vNgmP5cQk6sMpPiw7EnCg", "HFwMKdDrSCOq-MAtkXKTlw" ], "user": { "name": ”smith", "id": "1334f3ed7eb2483b91b8192ba043b580", "domain": { "name": "Default", "id": "default" } }, "extras": {}, "auth": { "identity": { "methods": [ ”token" ], ”token": { “id”: “a7409b” } } }
  • 59. Identity Token Path User OpenStack Service Step 1: Obtain unscoped token with credentials POST v3/auth/tokens Keystone Step 2: Discover projects you have access to GET v3/users/<user_id>/projects Step 3: Obtain project scoped token either with your credentials or unscoped token from step 1. Step 4: Invoke the target service by sending requests to endpoints in token from step 3 Step 5: Validate roles and access metadata in token with Keystone service or Keystone Middleware Step 6: Serve API request Step 7: Return response
  • 60. Token Creation 0 10 20 30 40 50 60 70 80 90 UUID PKI PKIZ Fernet Time Per Request Time Per Request Deep Dive into Keystone Tokens and Lessons Learned 0 5 10 15 20 UUID PKI PKIZ Fernet Requests Per Sec Requests Per Sec
  • 61. Token Validation 0 2 4 6 8 10 12 14 UUID PKI PKIZ Fernet Time Per Request Time Per Request Deep Dive into Keystone Tokens and Lessons Learned 0 20 40 60 80 100 UUID PKI PKIZ Fernet Requests Per Sec Requests Per Sec

Editor's Notes

  • #8: Identity: Checks if User exist in User Domain Check if User is enabled Retrieves User ID Matches Password Resource: Checks if Domain or Project exist Check if Domain or Project is enabled Retrieves Project ID and Domain ID Catalog: Retrieves Services associated with User’s Project Retrieves the list of endpoints for all the services
  • #18: Identity: Checks if User exist in User Domain Check if User is enabled Retrieves User ID Matches Password Resource: Checks if Domain or Project exist Check if Domain or Project is enabled Retrieves Project ID and Domain ID Catalog: Retrieves Services associated with User’s Project Retrieves the list of endpoints for all the services
  • #31: Identity: Checks if User exist in User Domain Check if User is enabled Retrieves User ID Matches Password Resource: Checks if Domain or Project exist Check if Domain or Project is enabled Retrieves Project ID and Domain ID Catalog: Retrieves Services associated with User’s Project Retrieves the list of endpoints for all the services
  • #40: I’ll next take us through how Horizon uses tokens. We’ve used Horizon for most of the time we’ve been using OpenStack, and we’ve found some interesting behaviors with the way Horizon manages them.
  • #41: When logging in, Horizon gets a keystone token using the user’s credentials. This is an important aspect of security in Horizon, as Horizon doesn’t require a service credential, minimizing the impact of an attacker compromising Horizon itself. First get an unscoped token, which is used to get scoped tokens to projects when the user switches projects. The storage method for tokens is configurable and can have a large impact depending on the complexity of your cloud. The configurable methods are: We are using the memcache backend. I’ll talk more just about the cookie backend and Memcache backend.
  • #42: The cookie backend has some very strong advantages, and it’s currently the devstack default. In this case, tokens are stored in a browser cookie on the client side. If using the cookie backend in production, it’s important to configure https connections to Horizon and also configure security for the tokens. Otherwise, someone could recover a token while sniffing the network. The cookie backend is highly scalable, as token storage is all done on the client side. However, the cookie backend can’t be used if you have many endpoints in the keystone catalog.
  • #43: Cookie sizes for most browsers are about 4KB. When the token takes up a lot of that space, you’ll log into Horizon, sign in…
  • #44: ..And see this!
  • #45: Using the memcache backend resolves cookie overflow issues. We currently use it at Symantec.
  • #46: Token Hashing has been used in the past to reduce the impact of large token sizes. We currently use token hashing with Kilo Horizon, which works fine with Hashing, but you could have issues if using the master branch.
  • #47: For UUID, PKI, and PKIZ, Tokens won’t work across auth regions. This will be a benefit of Fernet tokens, as they will allow authentication between multiple keystone instances.
  • #48: Fernet tokens work with Horizon
  • #49: This is somewhat out of scope for the different token types.
  • #54: I’ll next take us through how Horizon uses tokens. We’ve used Horizon for most of the time we’ve been using OpenStack, and we’ve found some interesting behaviors with the way Horizon manages them.
  • #63: This is just a code example for the Liberty code that disables token hashing.