Data Gloveboxes: A Philosophy of Data Science Data Security

© 2018 Bloomberg Finance L.P. All rights reserved.
Data Gloveboxes: A
Philosophy of Data Science
Data Security
DataWorks Summit - Barcelona
March 21, 2019
Clay Baenziger
Hadoop Infrastructure

© 2018 Bloomberg Finance L.P. All rights reserved.© 2018 Bloomberg Finance L.P. All rights reserved.
Bloomberg by the Numbers

Bloomberg By the Numbers
• Founded in 1981
• 325,000 subscribers in 170 countries
• Over 19,000 employees in 192 locations
— Over 5,000 software engineers
— 100+ machine learning data scientists and engineers
• More News reporters than The New York Times + Washington Post +
Chicago Tribune
— News content from 125K+ sources
— >1.5M news stories ingested / published each day (that's 500 news
stories ingested/second)
• One of the largest private networks in the world
• 100B+ tick messages per day, with a peak of more than 10 million
messages/second
• More than a billion messages (E-Mails and IB chats) processed each day

Nuclear Materials Manufacturing
Image: Office of Legacy Management, U.S. D.O.E., Rocky Flats Plant History & Information Used to Process EEOICPA Claim
Requests. 16 April, 2014
Former U.S. Department of Energy Rocky Flats Plant - South of Boulder, CO

Plutonium Dropbox? Isolation Glovebox?
Dropbox: [n] a container where one can deposit something to be retrieved later
Glovebox: [n] a sealed protective container in which one may safely manipulate a
dangerous substance using gloves attached to holes
Images:
(Top) Office of Legacy Management, U.S. D.O.E., CO-83-M-2 -
Interior view of X-Y retriever. 29 Nov, 1988
(Right) Office of Legacy Management, U.S. D.O.E., CO-83-K-15 -
View of safe geometry station from the inside of an input-output
station. 3 Dec, 1988

Data Dropbox? Data Glovebox?
Dropbox: [n] a data-system where one can deposit a file for later reading and
processing dependent on client (network) location; ideally providing a positive
verification of file contents
Glovebox: [n] a sealed compute environment in which one may safely
manipulate data using restricted access - with strong exfiltration controls
MySQL

Plutonium Enclave
Image: Office of Legacy Management, U.S. D.O.E.,CO-83-M-14 - Downdraft Table, 20 Aug. 2014

Data Enclave
Centralized Models:
• Curator Model: Restrict access, operations and results
— “The curator must remain present throughout the lifetime of the database”
(Dwork, Cynthia. “Differential Privacy: A Survey of Results”, 1, Apr. 2008)
— Statistical Disclosure Control
• Data Enclave:
(Lane and Shipp. “Using a Remote Access Data Enclave for Data Dissemination”. Intl. Journal of Digital Curation. 1.2 (2007))
— Allow for Direct and Exact Access
— Allow Arbitrary Computation
— Prevent Business Automation

Image: Denver Public Library, Rocky Mountain News Photographic Archives, “Rocky Flats
employee handles a robotic arm assembly.” 11 Nov. 1987
Plutonium Glovebox

Data Glovebox
• Leaded Pane of Glass: Remote Desktop Without Download
• Glove Ports: Arbitrary Code Execution (Run code to manipulate)
• Robotics: Workflow Management
— Deployment
— Routine operations
• Pass-throughs:
— Firewalls are insufficient
— Protocol aware deep packet inspection
— Databases
• Firewalls: Ensure user and workload isolation
— Distributed file-systems
— Local file-systems
— Processing

Glovebox Architecture
Copyrights: Git Logo, Jason Long; Python “Two Snakes” Logo, Python Software Foundation

Client Nodes
Client Nodes
Master Node
HBase Master
YARN Resource Manager
HDFS Namenode
Region Server
HBase
HDFS
Map/ReduceNovel Application
YARN
Spark
Region Server
HBase
HDFS
YARN
Spark
Region Server
HBase
HDFS
YARN
Spark
Region Server
HBase
HDFS
YARN
Spark
Hadoop Architecture
Client Nodes
Cluster Nodes
HBase
Region
Server
HDFS Datanode
YARN Nodemanager
Spark
Master Node
HBase Master
YARN Resource Manager
HDFS Namenode
<HTTP REST API>
YARN Job:
• Submission
• Status
• Logs
• App. WebUIs
<HTTP/2 REST API>
WebHDFS:
• GET Methods:
— Open (read)
— GetFileChecksum
• PUT Methods:
— Create
• POST Methods:
— Append
<HBaseProtobuf RPC>
• Get
• Put
• Scan

Handling Material
Image: Office of Legacy Management, U.S. D.O.E., “Rocky Flats Overview”, 20 Aug. 2014. Pg. 13

Properties of Radioactive Materials
Radioactive Materials:
• Can be harmful to people in small quantities
• Can have a very long hazard life if released
• Should be isolated to prevent their spread
• Should be cataloged and characterized to assess harm
• Can still be machined and worked with proper technique
— Robotics
— Personal Protective Equipment

Properties of Material Data
Material Data:
• Can be harmful to people in small quantities
• Can have a very long hazard life if released
• Should be isolated to prevent their spread
• Should be cataloged and characterized to assess harm
• Can still be used and analyzed with proper technique
— Continuous/Automated Deployment
— Workflow Automation and Gloveboxes

Nuclear Material (non)Proliferation
Image: Office of Legacy Management, U.S. D.O.E.,”Rocky Flats Overview”, 20 Aug. 2014. Pg. 59

Data Proliferation
• Terrorists? (Well, certainly hackers...)
• Accidental loss (USB sticks, laptops, etc.)
• No Price-Anderson Act for Data Incidents
— Quite the opposite with GDPR!
— GDPR limits untraceable mixing of data
• Data Sovereignty
— Requires data to remain geographically stationary
— Must move computation to the data
• Data:
— Swamps
— Lineage (Visibility e.g. via Apache Atlas)
— Masking (Curator Model e.g. via Apache Ranger)

Lock Everything Down
Image: Office of Legacy Management, U.S. D.O.E.,”Rocky Flats Overview”, 20 Aug. 2014. Pg. 21

Lock Data Down
Quality Attributes of a Dropbox
• Perimeter Controls (Network Firewall)
• Encryption:
— At rest
— On the wire
• Authentication (Kerberos)
• Client Location Controls Usage (Directionality)
— Data goes in from insecure networks
— Data cannot come back out to an insecure network
— Allow validation of transmission from anywhere
— “Normal” usage from trusted networks

DropboxFilter for HDFS (WebHDFS API)
• Upload:
$curl -X PUT "http://<HOST>:<PORT>/webhdfs/v1/<PATH>?op=CREATE
HTTP/1.1 307 TEMPORARY_REDIRECT
$curl -X PUT -T <File> "http://<DN>:<PORT>/webhdfs/v1/<PATH>?op=CREATE..."
HTTP/1.1 201 Created
• Download:
$curl -L "http://<HOST>:<PORT>/webhdfs/v1/<PATH>?op=OPEN
• Checksum:
$curl -L "http://<HOST>:<PORT>/webhdfs/v1/<PATH>?op=GETFILECHECKSUM"
{"FileChecksum": {
"algorithm": "MD5-of-0MD5-of-512CRC32C",
"bytes": "[...]00eb745ad2f5bd1dccab359b12f7f9411b00000000",
"length": 28
}}

DropboxFilter for HDFS (Architecture)
• HDFS Protocols
— Protobuf RPC
— RESTful API over HTTPS/2
• Servlet Based Web Server Design
— Filter
— Request Handler
HTTP
Server
(Netty)
Client
(curl)
Servlet Container
(Jetty)
WebHDFS Handler
AuthenticationFilter(request…)
DropboxFilter(request…)
Provides User Info
Provides User Info

DropboxFilter for HDFS (Examples)
<head><title>Error 401 Authentication
required</title></head>
<body><h2>HTTP ERROR 401</h2>
<p>Problem accessing /webhdfs/v1/user/ubuntu/foo. Reason:
<pre>Authentication required</pre></p>
<hr /><i><small>Powered by Jetty://</small></i><br/>
HTTP
Server
(Netty)
Client
(curl)
Servlet Container
(Jetty)
WebHDFS Handler
AuthenticationFilter(request…)
DropboxFilter(request…)
Provides User Info
Provides User Info
<head><title>Error 403 WebHDFS is configured write-only for
clay</title></head>
<body><h2>HTTP ERROR 403</h2>
<p>Problem accessing /webhdfs/v1/user/clay/foo. Reason:
<pre> WebHDFS is configured write-only for clay</pre></p>
<hr/><i><small>Powered by Jetty://</small></i><br/>
Download from:
https://issues.apache.org/jira/browse/HDFS-14234

Isolation Glovebox
Images:
(Left) Library of Congress, U.S. D.O.E.,View Of A Worker Holding A Plutonium 'Button.' 19 Sep. 1973
(Right) Office of Legacy Management, U.S. D.O.E.,CO-83-M-8 - View of foundry induction furnaces. N.D.

Glovebox Production Line
Images:
(Left) Library of Congress, U.S. D.O.E., View Of A Glovebox Line Used In Plutonium Operations. 5 May. 1970
(Right) Office of Legacy Management, U.S. D.O.E., CO-83-M-3 - View of Chainveyor. 25 Jan. 1993

Data Glovebox (Leaded Pane of Glass)
Avoid Overexposure to Raw Data
• Remote Desktop:
— Limited RDP
• Key Attributes:
— No copy out
— No file shares
— Isolation per user
• Useful to Have Tools:
— Web browser for Jupyter/Zeppelin
— SSH client for command-line access

Data Glovebox (Glove Ports & Robotics)
Manipulate Your Data - With Code
• Run on a compute cloud using Apache YARN; submit:
— SQL to Apache Hive
— Python or Scala to Apache Spark
— An arbitrary application
• Automation to ensure consistency (e.g. Apache Oozie)
— A workflow manager for Hive and Spark jobs
— Data transformations for expected reports -- known
processes generating “decontaminated” results
— Can run as a non-human service accounts to drop data in
directory for data exfiltration
— Can provide repeatable deployment of code using Git

Data Glovebox (Pass-Through)
Negative pressure (one-way) network; exfiltrate only “decontaminated” data
• Provide a process for data hand-off through an environment
• Firewalls:
— Mostly a transport OSI Layer 4 device (TCP/IP)
— Can do “deep packet inspection” - but need to MITM traffic
— Policy rules for which users can manipulate which data become extensive
— Prohibitively expensive
• Technology Specific:
— DropboxFilter for WebHDFS
— Database RPCs are complex but:
— GRANT INSERT ON DATABASE.* TO write_only@'%';
— GRANT SELECT ON DATABASE.* TO read_only@'%';
— HBase today has no built in client location filtering

Firewalls (Workload and User Isolation)
Don’t let your data spontaneously combust; clean up “chips”
File Systems Leak
• Permission on data sets
• User collaboration locations
• Temporary/failed job data
• Temporary data locations
— Distributed file systems
— Hive Warehouse
— /tmp
— Local file systems
— /tmp, /var/tmp, /dev/shm

Take out the Trash
Image: State of Idaho Oversight Monitor. Nov. 2006. Pg. 10

Private Temporary Directories
• To provide isolation, one can use pam_namespaces
• To setup directories and clean-up, one can use pam_exec
See also: Our integration of the work in https://github.com/bloomberg/chef-bach/pull/1278
Initial Mount Namespace
tmp (inode 100)
polyinst (inode 101)
tmp_clay (inode 201)
tmp_foo (inode 201)
home (inode 300)
User clay’s Mount Namespace
tmp (inode 201)
polyinst (inode 101)
tmp_clay (inode 201)
tmp_foo (inode 201)
home (inode 300)
/ (inode 2)/ (inode 2)

Keeping the Pipes Flowing
Image: Office of Legacy Management, U.S. D.O.E., CO-83-AF-1 - View of Building 215A. N.D.

YARN Node
HDFS
Data Node
YARN Node
HDFS
Data Node
YARN Network Isolation (Example)
YARN-7468 - Provide means for container network policy control
Database A WebService ADatabase B
YARN Nodes
HDFS
Data Node
Network Class 1
User A:
Novel
Application
YARN
Nodemanager
Network Class 2
User B:
Sparkiptables

Firewalls Are Important
Images:
(Left) Office of Legacy Management, U.S. D.O.E., CO-83-N-3 - Damaged Filter Plenums. 16 Sept. 1957
(Right) Office of Legacy Management, U.S. D.O.E., CO-83-M-5 - View of a glove box firewall detail. 8 May. 1970

Data Glovebox
• Leaded Pane of Glass: Remote Desktop Without Copy
• Glove Ports: Manipulate your Data at An Arm’s Length
• Robotics: Workflow Management
• Pass-throughs: Negative Pressure to Keep the Bits Flowing
• Firewalls: Ensure User and Workload Isolation

Cleanup Is Messy
Image: CO Dept. of Pub. Health, “Citizen Summary Rocky Flats Historical Public Exposures
Studies 1969 Fire”,

Data Gloveboxes: A Philosophy of Data Science Data Security

More Related Content

What's hot (20)

Similar to Data Gloveboxes: A Philosophy of Data Science Data Security (20)

More from DataWorks Summit (20)

Recently uploaded (20)

Data Gloveboxes: A Philosophy of Data Science Data Security