Cybersecurity with Apache Metron and Apache Solr - Ward Bekker, Hortonworks & Scott Cote, Lucidworks
1 like317 views
This document summarizes a presentation about Apache Metron and Apache Solr for cybersecurity. The presentation introduces Apache Metron as an open source architecture for real-time cybersecurity analytics. It processes streaming data from various sources through security analytics modules and stores the results in Apache Solr for real-time search and visualization. Examples of companies using Apache Metron include managed security service providers, telecom providers, financial institutions, and government organizations. The presentation concludes with a demo of how Apache Metron can detect malware by comparing blob signatures ingested from log files.
Cybersecurity with Apache Metron and Apache Solr - Ward Bekker, Hortonworks & Scott Cote, Lucidworks
- 1. Cybersecurity with Apache Metron and Apache Solr Ward Bekker - Solutions Engineer, Hortonworks Scott Cote - Senior Software Engineer, Lucidworks 1
- 2. Agenda for today’s talk • Introduction • Why Apache Metron? • What is Apache Metron? • What does Apache Metron look like? • Who’s using Apache Metron? • Apache Metron Ecosystem • Demo! 2
- 3. Ward Bekker • Hortonworks Solutions Engineer NEMEA • SME Cybersecurity • Apache Metron Contributor • Twitter: @wardbekker Ward Bekker
- 4. Ward Bekker •Lucidworks Senior Software Engineer - Fusion Server •Core Engineering •Founder of DFW Data Science User Group •Twitter: @scottccote & @DFWDataScience Scott Cote
- 5. SCARY
- 9. • I commit code/docs/patches • Running in production • Testing in lab • Just read/researched not touched
- 11. Months until breach noticed Avg. months log retention 9 6 VS 3 Months missing ==
- 12. 28 Months Police One/Berkut Yahoo/? FB/Cambridge Analytica 35 Months 48 Months Time until breach actually noticed
- 13. “Sometime in the next few years we're going to have our first category-one cyber-incident; one that will need a national response.”
- 14. 2018 so far... 340M Records 150M Records 92M Records And many, many, many more.. https://en.wikipedia.org/wiki/List_of_data_ breaches
- 15. Already drowning in Data
- 16. Explosion of (new) source of data
- 18. 18 Realtime Verizon cyber report: “Breaches happen in minutes, not hours, days, weeks or months” "Lightspeed" by Sergei Vavinov is licensed under CC BY 2.0
- 19. What is Apache Metron? 19
- 20. An architecture for real-time cybersecurity analytics
- 23. Cyber Security Stream Processing Pipeline
- 25. Apache SOLR usage in metron Apache Metron Stream Processing pipeline WARM/COLD INDEX LAYER: Data Vault, Data Science workbench, PCAP forensics,... HOT INDEX LAYER: Real-time search Visualisation & investigation Apache Metron Investigator Apache Zeppelin
- 26. Built on top on proven open source big data technology
- 27. Profiling by Time t = 1 t = 2 t = 3 t = n ⬢ ⬢ ⬢ ⬢ ⬢ ⬢ ⬢ ⬢
- 28. What does Apache Metron look like? 28
- 33. Who is using Apache Metron? 33
- 34. Who is using Apache Metron? 34 • Managed security service providers • Telstra • QSight/KPN • Financial institutions • Capital One • Telecom providers • Automotive industry • Defense ministries • Country-wide government initiatives
- 36. Apache Metron Ecosystem 36 • Visualisation and Exploration • Real-time interactive dashboarding • Infrastructure • Pre-built appliances optimized for Metron • Reporting and Compliance • NIST and other frameworks https://hortonworks.com/blog/building-a-cybersecurity-eco-system-on-a-shared-data-platform/
- 37. Demo 37
- 38. Demo Steps • Uploading blobs to an Fusion Application • One of those blobs is actually malware • Fusion logs are ingested in Metron • Compare the md5 signature of blobs to know malware • Metron Investigator UI to triage the scored events 38
- 39. Questions?
- 40. Thank you!