Tracing your security telemetry with Apache Metron
Download as PPTX, PDF9 likes4,660 views
This document discusses Apache Metron, an open source framework for performing security analytics on large volumes of telemetry data using Hadoop. It describes how Metron ingests and parses security-related data from different sources using stream processing, enriches the data with additional context, and checks it against threat intelligence feeds. Specifically, it walks through an example of how Metron would handle log data from the Squid caching proxy, parsing, enriching, and checking it for malicious domains.
![20 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Squid – Topology Definition
{ "parserClassName": "org.apache.metron.parsers.GrokParser", "sensorTopic": "squid", "pars
erConfig":
{ "grokPath": "/apps/metron/patterns/squid", "patternLabel": "SQUID_DELIMITED", "tim
estampField": "timestamp" },
"fieldTransformations" : [
{
"transformation" : "MTL" ,"output" : [ "full_hostname",
"domain_without_subdomains" ] ,"config" : { "full_hostname" :
"URL_TO_HOST(url)" ,"domain_without_subdomains" :
"DOMAIN_REMOVE_SUBDOMAINS(full_hostname)" } } ] }](https://image.slidesharecdn.com/june291220hortonworksleet-160710172019/85/Tracing-your-security-telemetry-with-Apache-Metron-20-320.jpg)
![23 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Loading some WHOIS derived data.
– Not directly making WHOIS query, just using a CSV containing a few rows of data.
Squid – Enrichment Definition
{
"zkQuorum" : ”localhost:2181"
,"sensorToFieldList" : {
"squid" : {
"type" : "ENRICHMENT"
,"fieldToEnrichmentTypes" : {
"domain_without_subdomains" : [ "whois" ]
}
}
}
}](https://image.slidesharecdn.com/june291220hortonworksleet-160710172019/85/Tracing-your-security-telemetry-with-Apache-Metron-23-320.jpg)
![26 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Loading a list of malicious domains
– ZeuS tracker
Squid – Enrichment Definition
{
"zkQuorum": "localhost:2181",
"sensorToFieldList": {
"squid": {
"type": "THREAT_INTEL",
"fieldToEnrichmentTypes": {
"url": ["zeusList”]
}
}
}
}](https://image.slidesharecdn.com/june291220hortonworksleet-160710172019/85/Tracing-your-security-telemetry-with-Apache-Metron-26-320.jpg)
Tracing your security telemetry with Apache Metron
- 1. Tracing Your Security Telemetry With Apache Metron Justin Leet Systems Architect June 29, 2016
- 2. 2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved What is Apache Metron?
- 3. 3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved What Apache Metron Does? “Apache Metron provides a scalable advanced security analytics framework built with the Hadoop Community evolving from the Cisco OpenSOC Project. A cyber security application framework that provides organizations the ability to detect cyber anomalies and enable organizations to rapidly respond to identified anomalies.”
- 4. 4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Apache Metron Timeline Sep 2014 •OpenSOC Beta June 2015 •OpenSOC Community Edition Dec 2015 •Metron enters Apache Incubator April 2016 •Apache Metron 0.1 Now •Working towards 0.2 release
- 5. 5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Who is Metron for?
- 6. 6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Core Capabilities
- 7. 7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Architecture
- 8. 8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Streaming Parsing and Enrichment
- 9. 9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Metron’s parsing bolt can be configured two ways – And outputs JSON Grok Parser – Less work to implement – Regex-like syntax – Good for lower volumes of data Java Parser – More work to implement – Good for higher volumes of data Parsing
- 10. 10 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Enrichment / Threat Intel
- 11. 11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Add additional information to raw source during streaming Adding it during streaming allows ML models to score in real time instead of batch Primarily stored in HBase Several enrichments – GeoIP – Host – Threat Intelligence Enrichment
- 12. 12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Occurs in the same Storm topology as enrichment Very similar process and flow Use a threat feed aggregator! – Soltra adapter is provided to read feed and stream into HBase – Flat File loader and Stix bulk loader available without threat feed aggregator Threat Intel
- 13. 13 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Field Description ip_src_addr Octet source IP ip_dest_addr Octet destination IP ip_src_port Integer source port ip_dest_port Integer destination port protocol String protocol (e.g. TCP) timestamp Sensor epoch timestamp source.type yaf, snort, etc. start_time Metron epoch timestamp end_time Metron epoch timestamp Metron JSON
- 14. 14 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Standalone Storm topology Reads from Kafka Writes packets to HDFS Kibana panel forwards request to REST PCAP service – MR Job launched – Delivers results back to Kibana PCAP
- 15. 15 © Hortonworks Inc. 2011 – 2016. All Rights Reserved PCAP
- 16. 16 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Tracing a Source Through Metron
- 17. 17 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Sensor to Parser
- 18. 18 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Caching proxy – Mostly useful as a source of easy to get and easily readable logs Squid 1467125585.752 5288 127.0.0.1 TCP_MISS/200 32250 GET https://news.ycombinator.com/ - DIRECT/104.20.43.44 text/html Time Elapsed Remote Host Code/Statu s Bytes Metho d URL rfc931 Peer Status/ Peer Host Type 1467125585.752 5288 127.0.0.1 TCP_MISS/2 00 32250 GET https://news.ycombinator.com/ - DIRECT/104.20.43.44 text/html
- 19. 19 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Squid - Grok Time Elapsed Remote Host Code/Statu s Bytes Metho d URL rfc931 Peer Status/ Peer Host Type 1467125585.752 5288 127.0.0.1 TCP_MISS/2 00 32250 GET https://news.ycombinator.com/ - DIRECT/104.20.43.44 text/html SQUID_DELIMITED %{NUMBER:timestamp}%{SPACE:UNWANTED} %{INT:elapsed}%{SPACE:UNWANTED}%{IPV4:ip_src_addr} %{WORD:action}/%{NUMBER:code} %{NUMBER:bytes} %{WORD:method} %{NOTSPACE:url} - %{WORD:UNWANTED}/%{IPV4:ip_dst_addr} %{WORD:UNWANTED}/%{WORD:UNWANTED}
- 20. 20 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Squid – Topology Definition { "parserClassName": "org.apache.metron.parsers.GrokParser", "sensorTopic": "squid", "pars erConfig": { "grokPath": "/apps/metron/patterns/squid", "patternLabel": "SQUID_DELIMITED", "tim estampField": "timestamp" }, "fieldTransformations" : [ { "transformation" : "MTL" ,"output" : [ "full_hostname", "domain_without_subdomains" ] ,"config" : { "full_hostname" : "URL_TO_HOST(url)" ,"domain_without_subdomains" : "DOMAIN_REMOVE_SUBDOMAINS(full_hostname)" } } ] }
- 21. 21 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Squid – Topology Result
- 22. 22 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Enrichment Topology
- 23. 23 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Loading some WHOIS derived data. – Not directly making WHOIS query, just using a CSV containing a few rows of data. Squid – Enrichment Definition { "zkQuorum" : ”localhost:2181" ,"sensorToFieldList" : { "squid" : { "type" : "ENRICHMENT" ,"fieldToEnrichmentTypes" : { "domain_without_subdomains" : [ "whois" ] } } } }
- 24. 24 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Squid – Enrichment Result
- 25. 25 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Enrichment Topology
- 26. 26 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Loading a list of malicious domains – ZeuS tracker Squid – Enrichment Definition { "zkQuorum": "localhost:2181", "sensorToFieldList": { "squid": { "type": "THREAT_INTEL", "fieldToEnrichmentTypes": { "url": ["zeusList”] } } } }
- 27. 27 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Squid – Threat Intel Result
- 28. 28 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Questions? Justin Leet Systems Architect [email protected] [email protected]