Building a Modern, Scalable Cyber Intelligence Platform with Apache Kafka | Jac Noel, Intel Corp
Download as PPTX, PDF0 likes494 views
The document presents a framework for building a scalable cyber intelligence platform using Apache Kafka, elaborated by Jac Noel at Kafka Summit Europe 2021. It emphasizes the importance of real-time data integration, automation, and improved incident response in enhancing information security at Intel. The architecture incorporates a data lake, message bus, machine-learning, and various advanced security measures to ensure robust performance and resilience in managing cyber threats.
Building a Modern, Scalable Cyber Intelligence Platform with Apache Kafka | Jac Noel, Intel Corp
- 1. Building a Scalable, Modern Cyber Intelligence Platform with Apache Kafka® Presenter: Jac Noel Kafka Summit Europe – May 2021
- 2. IT@Intel 2 Notices and Disclaimers This presentation is for informationalpurposes only. INTEL MAKES NO WARRANTIES, EXPRESS OR IMPLIED, IN THIS SUMMARY. Intel, the Intel logo, Intel Core, Intel Optane and Xeon are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the propertyof others. Copyright © 2021, Intel Corporation.All rights reserved. 2
- 3. IT@Intel 3 Jac Noel has over 25 years of Information Technology and Cyber Security experience across the military, government, and corporate environments. He started his technical career in the United States Air Force supporting defense intelligence systems for the AF mission in EMEA. He has spent the past 20 years serving in various technical roles in Intel’s IT organization. He’s currently serving as a Security Solutions Architect focusing on security intelligence and response capabilities. He’s the lead architect for Intel’s Cyber Intelligence Platform (CIP), which is a next- gen architecture combining a data lake, message bus, stream processing, machine-learning, orchestration, and workflow automation into a single platform. Jac holds a Bachelor of Science degree from Chico State University and has earned numerous professional certifications over the years, including CISSP, GCFW, CCNA, and MCSE. He’s also a proud inventor, patent holder, and author of several white papers. Jac Noel Security Solutions Architect
- 4. IT@Intel 4 Intel Information Security’s Mission 4 Our mission is to keep Intel legal and secure. This mission is never “done.” Best ways to measure our success: Reduce Mean Time to Detect (MTTD) and Mean Time to Respond (MTTR) Identify and implement more effective preventative controls Improve our agility to respond to new and changing threats and regulations
- 5. IT@Intel 5 API Data Virtualization Layer Information Security Business Role Incident Response Vulnerability Management Compliance Enforcement Data Protection Threat Intelligence Common Work Surface Layer Query Search Reporting Dashboards Visualizations Analytics Workbench Workflow Automation Infrastructure Clients Servers Network Infrastructure Other Data Sources Data Blueprint Security Data Lake Control Layer Security Event Management User Event Behavior Analytics Vulnerability Scanning Threat Intelligence Advanced Analytics Deceptions Intrusion Detection Firewalls Intrusion Prevention Endpoint Detection and Response Data Loss Prevention Intrusion Scanning Connectors Enterprise Security Message Bus Topics, Publish/Subscribe, Transform, Enrich, Filter, Join CyberIntelligencePlatform-ReferenceArchitecture A platform that supports our entire InfoSec organization 5
- 6. IT@Intel 6 High Performance Compute & Storage BU Partners IT Ops Partners Confluent Platform Message Bus Stream Processing Cyber Intelligence Platform - Solution Stack Our partners produce and consume data, too! 6
- 7. 7 Cyber Intelligence Platform – Solution Stack (cont) Built with industry leading technologies Splunk and Kafka
- 8. IT@Intel 8 The Power of the Kafka Bus No Message Bus Point to point, complex Slow to implement Increased technical debt due to tightly-coupled solutions and brittle integrations No orchestration (custom-code it, multiple times) No transformation (custom-code it, multiple times) Slow to move data between multiple capabilities Harder to monitor and govern With Message Bus Data Transformation (enrich, aggregate, normalize) Near real-time integration (streaming) Resilient, robust, scalable, available Orchestrate multiple activities in one place Cross-capability consumption Platform independent, plug and play Apps loosely coupled but tightly integrated Common architectural element for large enterprises App App App App App App App App App App App App App App App App App App App App App App App App Message Bus Abstraction, Resiliency, Scalability, Availability Transform Orchestrate
- 9. IT@Intel 9 Improving Data Availability with Confluent MRC 9 Single Cluster Data Center 3 Producers Consumers Streaming Apps Consumers Producers Data Center 1 Leaders (ISR) Zookeeper 1 Zookeeper 2 Broker n Broker 2 Broker 1 Broker 3 … Mirroring Data Center 2 Observers Zookeeper 3 Zookeeper 4 Broker n Broker 2 Broker 1 Broker 3 … Zookeeper 5
- 10. IT@Intel 10 Asynchronous Replication for Faster Recovery 10 Single Cluster Data Center 3 Producers Consumers Streaming Apps Consumers Producers Data Center 1 Zookeeper 1 Zookeeper 2 Broker n Broker 2 Broker 1 Broker 3 … Mirroring Data Center 2 Leaders (ISR) Zookeeper 3 Zookeeper 4 Broker n Broker 2 Broker 1 Broker 3 … Zookeeper 5 Confluent Platform with Multi Region Clusters
- 11. IT@Intel 11 TLS Confluent Control Center LDAP/TLS Schema Registry SASL Digest MD5 Admin User SASL TLS Digest MD5 Zookeeper 1 Broker Cluster TLS Zookeeper 2 Connectors SASL Digest MD5 Zookeeper 3 Authorization ACL Zookeeper Broker 1 Producers (Client App) Broker 2 TLS Stream Processor 1 Broker 3 Stream Processor 2 … TLS Consumers Stream Processor 3 Broker n (Client App) TLS Stream Processor Securing Our Confluent Platform 11
- 12. IT@Intel 12 Monitoring Our Kafka Clusters 12 Our C3 server requires Intel 2nd gen Xeon processors for high-performance compute and Intel Optane DC SSDs for low latency and high-endurance storage. Kafka Admins All-in-One Kafka Cluster Confluent Control Center Server (C3) (Broker, ZooKeeper, Connect, Kafka Streams) Kafka Streams App “Stream Processor” C3 Web App Consumers UI Trouble- shooting Producers Producers Kafka Production Monitoring Data Metrics Data Metrics Reporter Monitoring Interceptor Topics Topics Topics Topics Topics Topics Topics Topics Topics Topics Topics Topics Topics Topics Topics Topics Topics Consumers Metrics Topic Monitoring Topic Transformed Topics Health Monitoring
- 13. IT@Intel 13 Managing Vulnerabilities with Stream Processing 13 Confluent Platform Producers Kafka Streams API Stream Processing Kafka Bus Vulnerability Topic Filter Vulnerabilities by Business Unit IP Address Range Topic Join Asset Asset Inventory Topic Ownership with Consumers Vulnerable Assets BU #1’s Vulnerabilities Topic Data Lake BU Partners BU #2’s Vulnerabilities Topic IT Partners BU #3’s Vulnerabilities Topic SIEM Vulnerabilities with Owners Topic Enforcement SOAR Scanning Engine IP Address Management Asset Management Inventory Vulnerabilities Asset configuration, CVEs, CVSS IP Address Ranges Ownership, Business Units Asset Ownership
- 14. IT@Intel 14 Kafka Maturity Timeline 14 Acquire once-consume many Integration efficiency Remove the noise, and duplication Cost savings for downstream consumers Join multiple sources Contextually rich + clean data downstream ACQUIRE DATA FILTERING ENRICHMENT SUMMARIZATION ADVANCED Autonomous Actions e.g. Cluster analysis, ML Produce summary statistics State information, performance benefit and downstream cost savings
- 15. IT@Intel 15 Kafka By The Numbers 15 20+ TB/DAY 135+ 32+ CONSUMERS DATA SOURCES 320+ TOPICS 90+ PRODUCERS >18B EVENTS/DAY Kafka by the Numbers ~8 trillion events indexed by Splunk in 2020
- 16. IT@Intel 16 Kafka - Benefits to Intel 16 KAFKA LEADERSHIP THROUGH CONFLUENT EXPERTISE GENERATES CONTEXTUALLY RICH DATA MODERN ARCHITECTURE WITH THRIVING COMMUNITY GLOBAL SCALE AND REACH OPERATE ON DATA IN STREAM ECONOMIES OF SCALE REDUCE TECHNICAL DEBT AND DOWNSTREAM COSTS ALWAYS ON
- 17. IT@Intel 17 People + Technology + Data Transforming How Information Security Works 17 Reduced Risk to Intel Greater Insight and Tighter Collaboration Highly Integrated and Automated A Force Multiplier Faster Detection and Response Speaking a Common Language A Platform for the Future
- 18. IT@Intel 18 Additional Resources 18 Solution Brief and Reference Architecture
Editor's Notes
- #6: People + Technology + Data Transforming How Information Security Works
- #9: Abstraction Layer
- #17: Economies of Scale via acquire data once consume many Operate on Data In Stream – near real time identification and response to threats Reduce downstream costs, e.g. filtering data and transforming data (contextually rich) in kafka before applications and data lakes like Splunk, consumes Reduce technical Debt by eliminating custom connectors Generates Contextually rich data Global Scale and Reach – distributed bus technology that connects to cloud, IOT , other buses, kafka in backpack because records even when elements of assets are offline/separate Always On – no downtime, producers and consumers do not impact each other, kafka in backpack because it brings the data back online Modern Architecture with Thriving Community – great minds working across many distributed systems, data types, message bus systems, new APIs, always innovating Kafka leadership Through Confluent expertise – Confluent is technology leader and partnering with Intel to innovate
- #19: 18