Building a Modern, Scalable Cyber Intelligence Platform with Apache Kafka | Jac Noel, Intel Corp
0 likes609 views
The document describes a presentation by Jac Noel at the Kafka Summit Americas in September 2021, focusing on building a modern cyber intelligence platform using Apache Kafka. It highlights the architecture and capabilities of Intel's cyber intelligence platform, which integrates data management, stream processing, and machine learning to enhance security measures. The presentation emphasizes the benefits of a message bus architecture for improving data availability, detection, response times, and overall security efficiency.
Building a Modern, Scalable Cyber Intelligence Platform with Apache Kafka | Jac Noel, Intel Corp
- 1. Presenter: Jac Noel Prepared for Kafka Summit Americas – Sep 2021 Building a Modern, Scalable Cyber Intelligence Platform with Apache Kafka®
- 2. 2 Notices and Disclaimers This presentation is for informational purposes only. INTEL MAKES NO WARRANTIES, EXPRESS OR IMPLIED, IN THIS SUMMARY. Intel, the Intel logo, Intel Core, Intel Optane and Xeon are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others. Copyright © 2021, Intel Corporation. All rights reserved.
- 3. 3 Jac Noel has over 25 years of Information Technology and Cyber Security experience across the military, government, and corporate environments. He started his technical career in the United States Air Force supporting military intelligence systems for the AF mission in EMEA. He has spent the past 20 years serving in various technical roles in Intel’s IT organization. He’s currently serving as a Security Solutions Architect focusing on security intelligence and response capabilities. He’s the lead architect for Intel’s Cyber Intelligence Platform (CIP), which is a next- gen solution combining a data lake, message bus, stream processing, machine-learning, orchestration, and workflow automation into a single platform. Jac holds a Bachelor of Science degree from Chico State University and has earned numerous professional certifications over the years, including CISSP, GCFW, CCNA, and MCSE. He’s also a proud inventor, patent holder, author of several white papers, and presenter at many public and private forums. Jac Noel Security Solutions Architect Intel Corporation
- 4. 4 Intel’s Information Security Mission Our mission is to Keep Intel Legal and Secure! Ways we measure our success: 1. Better Preventive controls 2. More effective detective controls – Mean Time to Detect – Mean Time to Respond – Mean Time to Recover
- 5. Cyber Intelligence Platform - Reference Architecture People + Technology + Data 5
- 6. High Performance Compute & Storage BU Partners IT Partners Message Bus Stream Processing Cyber Intelligence Platform - Solution Stack 6 Enterprise Security Message Bus CONFLUENT PLATFORM
- 7. No Message Bus ▪ Point to point, complex ▪ Slow to implement ▪ Increased technical debt due to tightly-coupled solutions and brittle integrations ▪ No orchestration (custom-code it, multiple times) ▪ No transformation (custom-code it, multiple times) ▪ Slow to move data between multiple capabilities ▪ Harder to monitor and govern With Message Bus ▪ Data Transformation (enrich, aggregate, normalize) ▪ Near real-time integration (streaming) ▪ Resilient, robust, scalable, available ▪ Orchestrate multiple activities in one place ▪ Cross-capability consumption ▪ Platform independent, plug and play ▪ Apps loosely coupled but tightly integrated ▪ Common architectural element for large enterprises The Power of the Kafka Bus App App App App App App App App App App App App App App App App App App App App App App App App Message Bus Abstraction, Resiliency, Scalability, Availability Transform Orchestrate 7
- 8. Built with industry leading technologies Splunk and Kafka 8
- 9. Improving Data Availability with Confluent MRC 9 Single Cluster Data Center 3 Producers Consumers Streaming Apps Consumers Producers Data Center 1 Leaders (ISR) Zookeeper 1 Zookeeper 2 Broker n Broker 2 Broker 1 Broker 3 … Mirroring Data Center 2 Observers Zookeeper 3 Zookeeper 4 Broker n Broker 2 Broker 1 Broker 3 … Zookeeper 5
- 10. Asynchronous Replication for Faster Recovery 10 Single Cluster Data Center 3 Producers Consumers Streaming Apps Consumers Producers Data Center 1 Zookeeper 1 Zookeeper 2 Broker n Broker 2 Broker 1 Broker 3 … Mirroring Data Center 2 Leaders (ISR) Zookeeper 3 Zookeeper 4 Broker n Broker 2 Broker 1 Broker 3 … Zookeeper 5 Confluent Platform with Multi Region Clusters
- 11. Managing Vulnerabilities with Stream Processing 11 Confluent Platform Producers Kafka Streams API Stream Processing Kafka Bus Vulnerability Topic Filter Vulnerabilities by Business Unit IP Address Range Topic Join Asset Asset Inventory Topic Ownership with Consumers Vulnerable Assets BU #1’s Vulnerabilities Topic Data Lake BU Partners BU #2’s Vulnerabilities Topic IT Partners BU #3’s Vulnerabilities Topic SIEM Vulnerabilities with Owners Topic Enforcement SOAR Scanning Engine IP Address Management Asset Management Inventory Vulnerabilities Asset configuration, CVEs, CVSS IP Address Ranges Ownership, Business Units Asset Ownership
- 12. Kafka By The Numbers 12 20+ TB/DAY 135+ 32+ CONSUMERS DATA SOURCES 320+ TOPICS 90+ PRODUCERS >20B EVENTS/DAY Kafka by the Numbers ~8 trillion events indexed by Splunk in 2020
- 13. Kafka Maturity Timeline 13 Acquire once-consume many Integration efficiency Remove the noise, and duplication Cost savings for downstream consumers Join multiple sources Contextually rich + clean data downstream ACQUIRE DATA FILTERING ENRICHMENT SUMMARIZATION ADVANCE D Autonomous Actions e.g. Cluster analysis, ML Produce summary statistics State information, performance benefit and downstream cost savings
- 14. Kafka - Benefits to Intel 14 KAFKA LEADERSHIP THROUGH CONFLUENT EXPERTISE GENERATES CONTEXTUALLY RICH DATA MODERN ARCHITECTURE WITH THRIVING COMMUNITY GLOBAL SCALE AND REACH OPERATE ON DATA IN STREAM ECONOMIES OF SCALE REDUCE TECHNICAL DEBT AND DOWNSTREAM COSTS ALWAYS ON
- 15. 15 Summary Transformed how Intel’s Information Security works • Enterprise Security Message Bus • Modern and Scalable • Loosely coupled our People + Technology + Data • Highly Integrated and Automated • Faster Detection and Response • Greater Insights into better Prevention • A Platform for the Future • More Secure!