Protect your Private Data in your Hadoop Clusters with ORC Column Encryption

Fine Grained Access Control for Big
Data: ORC Column Encryption
Owen O’Malley
owen@cloudera.com
@owen_omalley
March 2019
Srikanth Venkat
svenkat@cloudera.com
@srikvenk

2 © Hortonworks Inc. 2011 – 2019. All Rights Reserved
Who Are We?
• Owen
• Worked on Hadoop since Jan 2006
• MapReduce, Security, Hive, and ORC
• Founder & Technical Fellow
• Srikanth
• Senior Director, Product Management (Security &
Governance portfolio)
• Apache Ranger, Apache Knox, Apache Atlas, ODPi
• Security, Data Stewardship, Metadata, Governance areas

Security & Data Protection in Hadoop

Example Data Lake Scenario
Marketing
Demographics
Electronic
medical records
CRM
POS
(Structured)(Structured) (Structured) (Structured) (Structured)
Cluster 1: Dublin Cluster 2: San Francisco
(Unstructured)(Unstructured)(Unstructured)
Cluster 3: Prague
(Structured)
On Premise Data Lakes
(Unstructured)(Structured) (Unstructured) (Structured)
Cloud Data Lakes
Social
Weblogs & Feeds
Transactional
Mobile
IoT
Personal Data

What’s different about the Big Data context?
Ã Breaking down silos: fantastic for analytics, but leads to increased security
challenges
– Centralized data lake with multi-tenancy requires secure (and easy) authentication and fine-
grained authorization
Ã Data democratization and the Data Scientist role (often a data superuser
with elevated privileges)
Ã Data is maintained over a long duration
Ã Cloud and Hybrid architectures spanning data center and (multiple) public
clouds further broaden the attack surface area and present novel
authentication and authorization challenges
Ã Along with adherence to security fundamentals and defense in-depth, a
data-centric approach to security becomes critical

Watch Towers
Limited Entry Points
Moat
Kerberos
Securing your data lake
High Hard Walls
Check Identity
Inner Walls
Firewall
Encryption, TLS, Key
Trustee, Navigator
Encrypt, Ranger KMS
LDAP/AD
Apache Knox: AuthN, API
Gateway, Proxy, SSO
Apache Ranger : ABAC
AuthZ, Audits,
Anonymization
Apache Sentry: RBAC
AuthZ

Data Protection in Hadoop
must be applied at three different layers
in Apache Hadoop
Storage: encrypt data while it is at rest
Transparent Data Encryption in HDFS, Navigator Key Trustee, Navigator
Encrypt, Ranger KMS + HSM, Partner Products (HPE Voltage, Protegrity,
Dataguise)
Transmission: encrypt data as it is in motion
Wire encryption (TLS, SASL,..)
Upon Access: apply restrictions when accessed
Apache Ranger (Dynamic Column Masking + Row Filtering), Partner
Masking + Encryption
Data Protection

Encryption of Data in Hadoop
Volume
Encryption
Protects data after physical theft
or accidental loss of a disk volume.
Entire volume is encrypted: very
coarse-grained security
Does not protect against viruses or
other attacks that occur while a
system is running.
Application-
level encryption
Encryption within an application
running on top of Hadoop
Supports a higher level of
granularity and prevents "rogue
admin" access
Adds a layer of complexity to the
application architecture.
HDFS data-at-
rest encryption
Encrypts selected files and
directories stored ("at rest") in
HDFS.
Uses specially designated HDFS
directories known as "encryption
zones.”
End-to-end encryption of data
read from and written to HDFS.
HDFS does not have access to
unencrypted data or keys.

Dynamic Row Filtering & Column Masking With Apache Ranger & Apache Hive
User 2: Ivanna
Location : EU
Group: HR
User 1: Joe
Location : US
Group: Analyst
Original Query:
SELECT country, nationalid,
ccnumber, mrn, name FROM
ww_customers
Country National ID CC No DOB MRN Name Policy ID
US 232323233 4539067047629850 9/12/1969 8233054331 John Doe nj23j424
US 333287465 5391304868205600 8/13/1979 3736885376 Jane Doe cadsd984
Germany T22000129 4532786256545550 3/5/1963 876452830A Ernie Schwarz KK-2345909
Country National ID CC No MRN Name
US xxxxx3233 4539 xxxx xxxx xxxx null John Doe
US xxxxx7465 5391 xxxx xxxx xxxx null Jane Doe
Ranger Policy Enforcement
Query Rewritten based on Dynamic Ranger Policies:
Filter rows by region & apply relevant column masking
Users from US Analyst group see data for US
persons with CC and National ID (SSN) as
masked values and MRN is nullified
Country National ID Name MRN
Germany T22000129 Ernie Schwarz 876452830A
EU HR Policy Admins can see
unmasked but are restricted by row
filtering policies to see data for EU
persons only
Original Query:
SELECT country, nationalid,
name, mrn FROM
ww_customers
Analysts
HR Marketing

Framing the Problem…..
• Related data, different security requirements
• Authorization – who can see it
• Audit – track who read it
• Encrypt on disk – regulatory
• File-level (or blob) granularity isn’t enough
• File systems don’t understand columns

Requirements
• Readers should transparently decrypt data
• If and only if the user has access to the key
• The data must be decrypted locally
• Columns are only decrypted as necessary
• Master keys must be managed securely
• Support for Key Management Server & hardware
• Support for key rolling

Partial Solutions

Partial Solution – HDFS Encryption
• Transparent HDFS Encryption
• Encryption zones
• HDFS directory trees
• Unique master key for each zone
• Client decrypts data
• Key Management via KeyProvider API

HDFS Encryption Limitations
• Very coarse protection
• Only entire directory subtrees
• No ability to protect columns
• A lot of users need access to keys
• Moves between zones is painful
• When writing with Hive, data is moved
multiple times per a query

Hive Server 2 Limitations
• Limits access to Hive SQL
• Only user ‘hive’ has access
• Breaks Hadoop’s multi-paradigm data access
• Many customers use both Hive & Spark
• JDBC is not distributed
• New Spark to LLAP connector addresses this

Partial Solution – Separate tables
• Split private information out of tables
• Separate directories in HDFS
• HDFS and/or HS2 authorization
• Enables HDFS encryption
• Limitations
• Need to join with other tables
• Higher operational overhead

Partial Solution – Encryption UDF
• Hive has user defined functions
• aes_encrypt and aes_decrypt
• Limitations
• Key management is problematic
• Encryption is not seeded
• Size of value leaks information

Solution

Columnar Encryption
• Columnar file formats (eg. ORC)
• Write data in columns
• Column projection
• Better compression
• Encryption works really well
• Only encrypt bytes for column
• Can store multiple variants of data

ORC File Format
File Footer
Postscript
Index Data
Row Data
Stripe Footer
~200MBStripe
Index Data
Row Data
Stripe Footer
~200MBStripe
Index Data
Row Data
Stripe Footer
~200MBStripe Column 1
Column 2
Column 7
Column 8
Column 3
Column 6
Column 4
Column 5
Column 1
Column 2
Column 7
Column 8
Column 3
Column 6
Column 4
Column 5
Stream 2.1
Stream 2.2
Stream 2.3
Stream 2.4

User Experience
• Set table properties for encryption
• orc.encrypt.pii = ”ssn,email”
• orc.encrypt.credit = “card_info”
• Define where to get the encryption keys
• Configuration defines the key provider via URI

Key Management
• Create a master key for each use
• “pii”, “pci”, or “hipaa”
• Each column in each file uses unique local key
• Allows audit of which users read which files
• Ranger policies limit access to keys
• Who, What, When, Where

KeyProvider API
• Provides limited access to encryption keys
• Encrypts or decrypts local keys
• Users are never given master keys
• Key versions and key rolling of master keys
• Allows 3rd party plugins
• Supports Cloud, Hadoop or Ranger KMS

Encryption Data Flow

Encryption Flow
• Local key
• Random for each encrypted column in file
• Encrypted w/ master key by KMS
• Encrypted local key is stored in file metadata
• IV is generated to be unique
• Column, kind, stripe, & counter

Static Data Masking
• What happens without key access?
• Define static masks
• Nullify – all values become null
• Redact – mask values ‘Xxxxx Xxxxx!’
• Can define ranges to unmask
• SHA256 – replace with SHA256
• Custom - user defined

Data Masking
• Anonymization is hard!
• AOL search logs
• Netflix prize datasets
• NYC taxi dataset
• Always evaluate security tradeoffs
• Tokenization is a useful technique
• Assign arbitrary replacements

Key Disposal
• Often need to keep data for 90 days
• Currently the data is written twice
• With column encryption:
• Roll keys daily
• Delete master key after 90 days

ORC Encryption Design
• Write both variants of streams
• Masked unencrypted
• Unmasked encrypted
• Encrypt both data and statistics
• Maintain compatibility for old readers
• Read unencrypted variant
• Preserve ability to seek in file

ORC Write Pipeline
• Streams go through pipeline
• Run length encoding
• Compression (zlib, snappy, or lzo)
• Encryption
• Encryption is AES/CTR
• Allows seek
• No padding

Conclusions

Conclusions
• ORC column encryptions provides
• Transparent encryption
• Multi-paradigm column security
• Audit logging (via KMS logging)
• Static masking
• Supports file merging
• Different stripes with different local key

Integration with Other Tools
• Hive & Spark
• No change other than defining table properties
• Apache Hive’s LLAP
• Cache and fast processing of SQL queries
• Column encryption changes internal interfaces
• Cache both encrypted and unencrypted variants
• Ensure audit log reflects end-user and what they accessed

Limitations
• Need encryption policy for write
• Current Atlas & Ranger tags lag data
• Auto-discovery requires pre-access
• Changes to masking policy
• Need to re-write files
• Need additional data masks
• Credit card, addresses, etc.
• Decrypted local keys could be saved

Thank you!
Twitter: @owen_omalley
Email: owen@cloudera.com

Protect your Private Data in your Hadoop Clusters with ORC Column Encryption

More Related Content

What's hot (20)

Similar to Protect your Private Data in your Hadoop Clusters with ORC Column Encryption (20)

More from DataWorks Summit (20)

Recently uploaded (20)

Protect your Private Data in your Hadoop Clusters with ORC Column Encryption