Solving Cybersecurity at Scale

1 © Hortonworks Inc. 2011–2018. All rights reserved.
Solving Cybersecurity at Scale
Laurence Da Luz & Mo Kamel

What Are We Talking About?
Cybersecurity Challenges
Solving Cybersecurity At Scale
Anatomy of Apache Metron
Use Case Walkthrough

3 © Hortonworks Inc. 2011–2018. All rights reserved. 3
CYBERSECURITY IS
A BIG DATA PROBLEM

Big Traffic, Big Trouble
Complexity Problem
• Too many point solutions
• Too many dashboards
• Too hard to correlate data across
silos
• Cybersecurity staff overwhelmed
with too many alerts

Big Traffic, Big Trouble + Capability Problem
• Huge volumes & Limited Storage
• Inconsistent data from multiple
sources
• Real-time context is crucial
• Missing Adv. Analytics
• Alert Fatigue – “Many False
Positives”

+ People Problem
• Skill Shortage around the globe
• Staff inefficiency & high cost
• Low value work of data gathering
and cleansing
• Impractical solution scaling people

+ Security Problem
• Distracting and Adv. attacks
• Lake of security context
• Asset Classifications
• Prioritization and Scoring
• Full access to historical data

A Community Solution Open Source Solution
• Volume
• Variety
• Value
• Automation
• Realtime
• Threat Intel

Advanced Use Cases
Open Source Solution
• Users Behavior
• Entities Behavior
• Advanced Analytics

An architecture for real-time cybersecurity analytics
REAL-TIME PROCESSING CYBER SECURITY ENGINE
Cyber Security Stream Processing Pipeline
Telemetry Data
Sources
Telemetry Data
Collectors
Telemetry
Parsers
Enrichment Threat
Intel
Profiler Alert
Triage
Indexers
and
Writers
SecurityEndPoint
Devices
(Fireye,PaloAlto,
BlueCoat,etc.)
Machine
GeneratedLogs
(AD,App/Web
Server,firewall,
VPN,etc.)
IDS
(Suricata,Snort,
etc.)
NetworkData
PCAP,Netflow,Bro,
etc.)
ThreatIntelligence
Feeds
(Soltra,OpenTaxi
third-partyfeeds)
Performance
NetworkIngest
Probes
Real-Time
Enrich/Threat
IntelStreams
/Other…
DataVault
Real-TimeSearch
EvidentiaryStore
ThreatIntelligence
Platform
ModelasaService
CommunityModels
DataScience
Workbench
PCAPForensics
Modules
Data Services
& Integration
Layer
Telemetry
Ingest Buffer
HORTONWORKS DATA PLATFORMHORTONWORKS DATA FLOW

An architecture for real-time cybersecurity analytics
Cyber Security Stream Processing Pipeline
Telemetry Data
Sources
Telemetry Data
Collectors
Telemetry
Parsers
Enrichment Threat
Intel
Profiler Alert
Triage
Indexers
and
Writers
SecurityEndPoint
Devices
(Fireye,PaloAlto,
BlueCoat,etc.)
Machine
GeneratedLogs
(AD,App/Web
Server,firewall,
VPN,etc.)
IDS
(Suricata,Snort,
etc.)
NetworkData
PCAP,Netflow,Bro,
etc.)
ThreatIntelligence
Feeds
(Soltra,OpenTaxi
third-partyfeeds)
Performance
NetworkIngest
Probes
Real-Time
Enrich/Threat
IntelStreams
/Other…
DataVault
Real-TimeSearch
EvidentiaryStore
ThreatIntelligence
Platform
ModelasaService
CommunityModels
DataScience
Workbench
PCAPForensics
Modules
Data Services
& Integration
Layer
Telemetry
Ingest Buffer
HORTONWORKS DATA PLATFORMHORTONWORKS DATA FLOW
Collect security device and
machine generated logs
Extendable data model
Enrichment on Ingest for
extra context
Behavior profiling and
advanced windowing
Flexible deployment of Data
Science
Alerting and triage (exposed
to SOC)
Hortonworks Cybersecurity Platform runs as an application
on top of HDF and HDP
REAL-TIME PROCESSING CYBER SECURITY ENGINE

Context is everything
Enrichments
User, group data,
internal business sources
Geospatial data, worldwide
shared threat intelligence Model predictions, via
Model As A Service framework Time

Time is context | Time matters
The Profiler
• A generalized solution for extracting model features and aggregations over time from high throughput,
streaming data
• Generates a profile describing the behavior of an entity; a host, user, subnet or application..
• A foundational component for both security model building and alerting in HCP
t = 1 t = 2 t = 3 t = n
Profile behavior across
windows in time, and
across multiple devices

t = 1 t = 2 t = 3 t = n
… how do we perform
behavioral profiling at
real-time scale?
Time is context | Time matters
The Profiler
Variety of different types of data sketches, but
general characteristics include:
• Stream friendly - each item examined only once,
can quickly update a small sketch data structure
• Scalable – effective for queries that do not scale
well; count distinct, quantiles, most frequent
items
• Approximate – faster results within
mathematically proven error bounds
• provide fixed size compute and
predictable space usage
Combined Sketch
Period: 0<t<3
Combined Sketch
Period: 0<t<1 + 2<t<3 +….
Sketch
Period: 0<t<1
Sketch
Period: 1<t<2
Sketch
Period: 2<t<3
Sketch
Period: n-1<t<n
Data sketches provide fast,
approximate answers to queries about
the underlying data.
Data sketches are combinable.
Allows us to slice and dice the windows and re-
combine them during read.
Can pick and mix sketches (skip certain days,
hours, etc..)

Streaming Analytics at Scale
Algorithms out of the box
Profiles
• HyperLogLog (Cardinality) – How many servers
does this user talk to usually?
• Bloom Filters – Have we seen this domain
before?
• T-Digest (distribution) – Personalized Baselining
and statistics
• Counters and descriptive statistics – Quick
results and triggers for more intensive
calculations.
• Mixed period windows – accounting for
holidays, typical working periods and seasons
Approximation algorithms - specialized algorithms that can produce results
orders-of-magnitude faster within mathematically proven error bounds - ideal
for real-time analytics
Natural Language Processing
(finding likely non-human behavior with Machine
Learning)
• Typosquat (mis-spellings, homoglyphs)
• DGA (Domain Generation Algorithm)
Streaming similarity and anomaly detection
• Mean Absolute Deviation
• TLSH (Locality Sensitive Hashing) – Finding events
similar to known bad
• GeoHash similarity
• Robust PCA

UNDER THE HOOD: ANATOMY OF THE
DATA ENGINEERING PIPELINE

Architecture & Capabilities

Pipelines are created and deployed via the Metron
framework - no custom storm code required
Extendable Domain Specific Language (DSL) used across Metron
for querying, transformation, and configuring rules
Core pipeline components: NiFi, Kafka, Storm, Spark, Solr.
Access and Visualization: Metron UI & Zoomdata (partner)
Generated alerts can be integrated
with external systems

ARCHITECTURE & CAPABILITIES
[#] #### :
#######
#####
[#] #### :
#######
#####
[#] #### :
#######
#####
Acquire
NiFi (& MiNiFi) acquire raw
data and handle routing
Devices generate
raw log messages
data formats from a variety of
disparate systems and sources

[#] #### :
#######
#####
{
a: ##
b: ####
c: ###
}
[#] #### :
#######
#####
[#] #### :
#######
#####
[#] #### :
#######
#####
Acquire Normalize
Out of the box device parsers
ASA, Bro, Fireeye, PaloAlto, …
Convert all data from raw source
logs into a common JSON format
simplifies downstream
analytics across devices
General purpose format parsers
Grok, Regex, CSV, JSON
Custom java based parsers

[#] #### :
#######
#####
{
a: ##
b: ####
c: ###
}
{
a: ##
b: ####
c: ###
d: ##
e: ##
}
{
a: ##
b: ####
c: ###
d: ##
e: ##
alert: true
sev: 1
}
[#] #### :
#######
#####
[#] #### :
#######
#####
[#] #### :
#######
#####
Acquire EnrichNormalize
Geo enrichment, hbase lookups
for custom enrichments, MaaS
additional additional information
to raw source during streaming
Assess against threat feeds,
and alert based on severity

[#] #### :
#######
#####
{
a: ##
b: ####
c: ###
}
{
a: ##
b: ####
c: ###
d: ##
e: ##
}
{
a: ##
b: ####
c: ###
d: ##
e: ##
alert: true
sev: 1
}
[#] #### :
#######
#####
[#] #### :
#######
#####
[#] #### :
#######
#####
t1 t2 [t3] --- !
Acquire Enrich ProfileNormalize
Profiler generates feature sets
that are stored within HBase
Profiler is a separate pipeline that
listens on all streaming events
Pipeline specialized to understand a series
of actions in time across multiple devices
windowed features can be looped
back for triage and alerting
Batch profiling is also supported, that can
“seed” a feature set from historical data

[#] #### :
#######
#####
{
a: ##
b: ####
c: ###
}
{
a: ##
b: ####
c: ###
d: ##
e: ##
}
{
a: ##
b: ####
c: ###
d: ##
e: ##
alert: true
sev: 1
}
[#] #### :
#######
#####
[#] #### :
#######
#####
[#] #### :
#######
#####
t1 t2 [t3] --- !
--- [alert] ---
a: ## b: #### c: ### d: ## e: ##
Acquire Enrich ProfileNormalize Security Data Lake
Data is indexed in Solr near-term
for random access and hot tiering
Data is stored in HDFS long-term
for historical access and analytics
SOC Analysts perform security
monitoring and threat hunting
Security data scientists train
against historical trends to
improve alerting models

[#] #### :
#######
#####
{
a: ##
b: ####
c: ###
}
{
a: ##
b: ####
c: ###
d: ##
e: ##
}
{
a: ##
b: ####
c: ###
d: ##
e: ##
alert: true
sev: 1
}
[#] #### :
#######
#####
[#] #### :
#######
#####
[#] #### :
#######
#####
t1 t2 [t3] --- !
--- [alert] ---
a: ## b: #### c: ### d: ## e: ##
Acquire Enrich ProfileNormalize Security Data Lake
End-to-end streaming data pipeline enables real-time
action against cyber threats in a repeatable patternArchitecture & Capabilities

USE CASE
WALK THROUGH

Deploying a Use Case in Apache Metron
What is Squid?
• Squid is a caching proxy for the Web supporting HTTP, HTTPS, FTP, and more. It reduces bandwidth and improves response
times by caching and reusing frequently-requested web pages
What does a Squid access log look like?
• When you make an outbound http connection to http://www.cnn.com, the following entry is added to a file called
access.log:
Squid Logs - Use Case Walkthrough
Unix Epoch Time
IP of host where connection was
made
Domain name of the outbound
connection

What is Squid?
• Squid is a caching proxy for the Web supporting HTTP, HTTPS, FTP, and more. It reduces bandwidth and improves response
times by caching and reusing frequently-requested web pages
What does a Squid access log look like?
• When you make an outbound http connection to http://www.cnn.com, the following entry is added to a file called
access.log:
Squid Logs - Use Case Walkthrough
Unix Epoch Time
IP of host where connection was
made
Domain name of the outbound
connection
Convert from Unix Epoch to
Timestamp
Asset enrichment to enrich IP
(hostname, type of device)
WHOIS enrichment to look up
domain name information
Threat Intel to cross-reference address with
intel feed to see if there is a hit
Index the event into Solr and persist in
HDFS (Security Data Lake)
What Metron will do to the
Squid telemetry in real-time

Step 1 NiFi TailFile
Step 2 Define Parser
Step 3 Enrichment Config
Step 4 Configure Alerts
Step 5 Configure Profiler

Step 1 – Telemetry Ingest
Streaming from NiFi to Kafka
Data is tailed from the Squid access-log files:

Step 2 – Configuring the Squid Parser
Defining a Grok Filter for the Squid data
• Grok parser à config driven
• Regex-based abstraction
• Grok is suitable for structured or
semi-structured logs
• Contains pre-defined mappings
Pre-defined grok mappings for IP

Step 3 – Configuring Streaming Enrichment
Enriching events with GEO data
• Leverage the out of the box GEO enrichment.
• Custom enrichment sources also supported – stored
in HBase & configuration driven
Enriching against a DGA model
$METRON_HOME/bin/maas_deploy.sh -zq node1:2181 -lmp $HOME/mock_dga -hmp
/user/$USER/models -mo ADD -m 512 -n dga -v 1.0 -ni 1
Mock DGA python model to
detect malicious domains
Deploy to Metron Model as a
Service (MaaS)
Call model in stream within
parser or enrichment configs

Step 4 – Configuring Alerts
Defining severity ratings based on threat triage rules
Raise an alert if our dga model finds
a detection
Set our score rating to 100 on hit
Multiple alert rules supported.
Aggregator defined for when
multiple conditions are met

enrichment from GEOIP lookup
enrichment from DGA python model
(from Model as a Service)

Step 5 – Configuring Profiler
Finding geographic anomalies in user login behavior - an authentication log example
Profile 1: Track locations by user
• geohashes of the locations the user has logged in from
• multiset of geohashes per user (mapping occurrence counts)
{
"profile": "locations_by_user",
"foreach": "user",
"onlyif": "hash != null && LENGTH(hash) > 0"
"init": {
"s": "MULTISET_INIT()"
},
"update": {
"s": "MULTISET_ADD(s, hash)"
},
"result": "s”
}
{
"profile": "geo_distribution_from_centroid",
"foreach": "'global'",
"onlyif": "geo_distance != null"
"init": {
"s": "STATS_INIT()"
},
"update": {
"s": "STATS_ADD(s, geo_distance)"
},
"result": "s”
}
Profile 2: Track geo distribution from centroid
• Statistical distribution of the distance between login location and the
geographic centroid of the user’s previous logins from within the last
5 minutes
These profiles will help us track if a user is logging in via vastly differing
geographic locations in a short period of time

{
"threatIntel": {
"fieldMap": {
"stellar" : {
"config" : [
"geo_distance_distr:= STATS_MERGE( PROFILE_GET('geo_distribution_from_centroid', 'global',
PROFILE_FIXED( 4, ’HOURS')))",
"dist_median := STATS_PERCENTILE(geo_distance_distr, 50.0)",
"dist_sd := STATS_SD(geo_distance_distr)",
"geo_outlier := ABS(dist_median - geo_distance) >= 5*dist_sd",
"is_alert := exists(is_alert) && is_alert",
"is_alert := is_alert || (geo_outlier != null && geo_outlier == true)",
"geo_distance_distr := null"
]
}
}
Compute the threat given global context and per-user context
Get the statistical distribution of the
‘geo_distance’ field for all users
Decide if the geo_distance is an outlier by
testing how many standard deviations it is
from the median
Update the ‘is_alert’ accordingly. If this is
true, then we need to triage the alert level

"triageConfig" : {
"riskLevelRules" : [
{
"name" : "Geographic Outlier",
"comment" : "Determine if the user's geographic distance from the centroid of the historic logins
is an outlier as compared to all users.",
"rule" : "geo_outlier != null && geo_outlier",
"score" : 10,
"reason" : "FORMAT('user %s has a distance (%d) from the centroid of their last login is 5 std
deviations (%f) from the median (%f)', user, geo_distance, dist_sd, dist_median)"
}
],
"aggregator" : "MAX"
}
Triage the threat
Because this is only a circumstantial
indicator, we’ll only give this a threat
score of 10
In a normal system, there would be many
rules triaging the threat. In this case the
max score would be taken
We need to ensure the SOC Analyst has enough
context to make a decision here

Key Takeaways
• Cybersecurity is a big data problem
We need a community driven approach to solve it
• Modern cybersecurity challenges require
a modern data architecture to facilitate real-time response
• Apache Metron provides an extensible, repeatable, and
configuration driven framework for real-time cybersecurity at scale
© Hortonworks Inc. 2011–2018. All rights reserved.

Thank you.

Solving Cybersecurity at Scale

More Related Content

What's hot (20)

Similar to Solving Cybersecurity at Scale (20)

More from DataWorks Summit (20)

Recently uploaded (20)

Solving Cybersecurity at Scale