SlideShare a Scribd company logo

Data Streaming with Apache Kafka in the Defence and Cybersecurity Industry

Kai Wähner
Kai Wähner

The document discusses the integration of data streaming, particularly through Apache Kafka, in the defense industry for enhancing cybersecurity, real-time situational awareness, threat detection, and modernization of security operations like SIEM and SOAR. It emphasizes the importance of secure and scalable defense mechanisms, efficient data correlation, and the implementation of zero trust environments to protect critical systems from cyber threats. Additionally, it highlights the need for real-time data analysis and flexibility in cybersecurity strategies to address complex cyber challenges successfully.

Software
1 of 56
1
2
3
4
5
6
7
8
9
10
11
12
13
14
Most read
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
Most read
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
Data Streaming in the Defence Industry Data in Motion for Situational Awareness, Threat Detection, Forensics, Zero Trust Zones Kai Waehner Field CTO kai.waehner@confluent.io @KaiWaehner confluent.io kai-waehner.de linkedin.com/in/kaiwaehner
@KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Agenda 1) Defence and Cybersecurity in 202X 2) Data in Motion as Defence Backbone 3) Situational Awareness 4) Threat Intelligence 5) Forensics 6) Air-Gapped and Zero Trust Environments 7) SIEM / SOAR Modernization
@KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Agenda 1) Defence and Cybersecurity in 202X 2) Data in Motion as Defence Backbone 3) Situational Awareness 4) Threat Intelligence 5) Forensics 6) Air-Gapped and Zero Trust Environments 7) SIEM / SOAR Modernization
@KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Modern Warfare Information technology and real-time information as game changer for defence 4
@KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Cybersecurity Protection of computer systems and networks from information disclosure and theft Web Scraping, hackers, criminals, terrorists, state-sponsored and state-initiated actors 5
@KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka The need for secure and scalable defence Defence requires real-time information everywhere! Challenges Stealing IP DDoS Ransomware / wiperware WannaCry, NotPetya, SolarWinds … Damage: Billions of dollars ”Supply chain attack” Digital Transformation Networking Communication Connectivity Open standards ”Always-on” Billions of devices
@KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Supply Chain Attack Targeting less-secure elements in the supply chain 7 https://www.nortonrosefulbright.com/en/knowledge/publications/dfa3603c/six-degrees-of-separation-cyber-risk-across-global-supply-chains https://www.reuters.com/article/us-tmobile-dataprotection-idUSKCN0RV5PL20151002
@KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka SECURITY Security SIEM Encryption OT Security Hardware- based Security Cybersecurity How would you have a holistic view and understanding of all the events and potential abuses that are taking place within your organization? Collect and correlate the different activities happening on critical networks CYBERSECURITY is a key piece of the security strategy SIEM and SOAR a (key) piece of the cybersecurity strategy Sometimes not needed (in DMZ / air gapped env) Complex and error prone No help against insiders Continuous real-time data correlation required SOAR Avoid risk (change operations) + Transfer some risk (buy insurance) Real-time Monitoring (Logging, SiteOps, …) Access Control (RBAC, Audit Logs, …)
@KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Key Challenge: Find the Needle(s) in the Haystack Detect true positives in real-time • Threat detection • Intrusion prevention • Anomaly detection • Compliance auditing • Proactive response Reduce false positives • Automation • Process big volumes of data in real-time • Integration of all sources • No ‘ignore’ on certain events • Creation of filters and correlated event rules • Improve signal-to-noise ratio (SNR) • Correlate “collection of needles” in “signature needle”
@KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Agenda 1) Defence and Cybersecurity in 202X 2) Data in Motion as Defence Backbone 3) Situational Awareness 4) Threat Intelligence 5) Forensics 6) Air-Gapped and Zero Trust Environments 7) SIEM / SOAR Modernization
@KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka This is a fundamental paradigm shift... 20 Infrastructure as code Data in motion as continuous streams of events Future of the datacenter Future of data Cloud Event Streaming
@KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Real-time Data beats Slow Data. Cybersecurity Risk classification Threat detection Intrusion detection Incident response Command Post Intelligent Navigation Vehicle Inspection Location-based Services Logistics Supply Chain Inventory management Fleet Management Military Security monitoring Surveillance Command and Control Military Intelligence
@KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Apache Kafka is the Platform for Data in Motion MES ERP Sensors Mobile Customer 360 Real-time Alerting System Data warehouse Producers Consumers Streams and storage of real time events Stream processing apps Connectors Connectors Stream processing apps Supplier Alert Forecast Inventory Customer Order 22
@KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Data in Motion The Backbone for Defence Command Post Enterprise IT Strategic Planning Logs Personal Sensors Security Streams of real time events 23 Connected Vehicles Cyber Security Continuous Data Correlation Monitoring Alerting Proactive Actions
@KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka End-to-End Cyber Defence with the Kafka Ecosystem Personel Crew, Cargo Vessel Fuel Consumption, Speed, Planned Maintenance Tracking Position, Course, Weather, Draft Drone or Satellite Relay COMMs Resilient Kafka Edge Analytics Bidirectional Ship Edge to Cloud, Shore Edge to Cloud Relay Ingestion Data Integration Streaming Analytics Machine Doing On-Prem Systems Bi-Directional Hybrid Cloud Replication ON SHORE ON PREM Staging, Filtering Shore Edge Analytics
@KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Integrate with all legacy and modern interfaces Record, filter, curate a broad set of traffic streams Let analytic sinks consume just the right amount of data Drastically reduce the complexity of the enterprise architectures Drastically reduce the cost of SIEM / SOAR deployments Add new analytics engines Add stream-speed detection and response at scale in real-time Add mission-critical (non-) security-related applications … is the backbone for data streams in defence!
@KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Every enterprise is different… Flexibility is key for your cybersecurity initiative! Confluent is an independent foundation. 30
@KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Kafka Connect Confluent Various Data Producers Flexible Scalable Real-Time Backplane for the Defence Platform Splunk TensorFlow Kafka Forwarder TensorFlow + Kafka plugin Event Streaming Platform OT Domain SIEM Domain Analytics Domain 31 Huge volumes of real-time data from various Kafka topics Backpressure handling and a low velocity Kafka topic High velocity, raw Kafka topic for forensics and ML
@KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Agenda 1) Defence and Cybersecurity in 202X 2) Data in Motion as Defence Backbone 3) Situational Awareness 4) Threat Intelligence 5) Forensics 6) Air-Gapped and Zero Trust Environments 7) SIEM / SOAR Modernization
@KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Cyber Situational Awareness is the subset of all situation awareness necessary to support taking actions in cyber 36 Endsley, M. R. Toward a Theory of Situation Awareness in Dynamic Systems. Human Factors, 1995, 37(1), 32-64
@KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Human – Computer Interface for Decision Making 38 https://www.youtube.com/watch?v=mPJdzzm67sg
@KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Human – Computer Interface for Decision Making 39 https://www.youtube.com/watch?v=mPJdzzm67sg
@KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Firewalls & Network Devices Antivirus Access Logs Intrusion Detection Audit Logs Text Files Binary Files Databases APIs Network Flows Syslog The Data
@KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Transactions Low Velocity, Low Volume Netflow / PCAP High Velocity, High / Ridiculous Volume Ingested via Network Analyzer Gateway Logs Low Velocity, Moderate Volume Store PCAP headers in Tiered Storage or 3rd Party like Corelight as intermediary Data Producers
@KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Streams Streams Streams Streams Event type-specific parsing and normalization logs-conn-shared logs-resolve-names logs-geoip-asn-iprep Streams Streams logs-index Established connection and client/server detection DNS name resolution GeoIP, IP Reputation and Autonomous System lookup Data Normalization and Enrichment à Improve the signal + filter to lower the noise
@KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka SIEM Forwarder Threat Detection Near real-time Data Consumers No constraints on integration flows Data curation on the fly Flexible choice of (multiple) consumers Sink to Data Lake Analytical Workloads Batch Native Kafka App Transactional Workloads Real-time
@KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Sigma 44 • Open-source framework • Domain specific language (DSL) • Specify patterns in cyber data https://github.com/SigmaHQ/sigma
@KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Sigma Rule Detections • List of detections for each condition • Single or list of values • Individual values or regex • Detection names can also include operators (ex. name|endswith, name|contains, name|greater_than) • Aggregations and windowing Conditions • Nested conditions based on defined detections Detection Names • Generic Sigma names defined • Translated during parsing to meet end SIEM tool using field mapping file
@KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Confluent Sigma Sigma Stream Processors Zeek Data and Detections Viewer Sigma Rule Editor sigma rules topic DNS dns detections topic dns topic rule parsing, filtering, aggregation, windowing sigma rules cache CONN DHCP HTTP SSL x509 Zeek Data https://github.com/confluentinc/cyber/tree/master/confluent-sigma
@KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Agenda 1) Defence and Cybersecurity in 202X 2) Data in Motion as Defence Backbone 3) Situational Awareness 4) Threat Intelligence 5) Forensics 6) Air-Gapped and Zero Trust Environments 7) SIEM / SOAR Modernization
@KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Threat Intelligence 54 Mitigate harmful events in cyberspace Proactive cybersecurity posture that is predictive, not just reactive Bolster overall risk management policies Improved detection of threats Better decision-making during and following the detection of a cyber intrusion See the whole board, more quickly. See around corners. See the enemy before they see you.
@KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Transactions vs. Analytics 55 Threat intelligence = awareness-in-motion The PATTERN is valuable, not the data.
@KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Streams logs-index Authorized access using RBAC Machine Learning Predictions via UDFs PII Anonymization logs-alerts logs-index-gdpr Analytics and Actionable Insights in Motion Make sense of the signal and the noise of the data Continuous signature processing Prevent, contain, and neutralize threats proactively Access for data science teams
@KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Cyber Intelligence Platform leveraging Kafka Connect, Kafka Streams, Multi-Region Clusters (MRC), and more… https://www.intel.com/content/www/us/en/it-management/intel-it-best-practices/modern-scalable-cyber-intelligence-platform-kafka.html
@KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Agenda 1) Defence and Cybersecurity in 202X 2) Data in Motion as Defence Backbone 3) Situational Awareness 4) Threat Intelligence 5) Forensics 6) Air-Gapped and Zero Trust Environments 7) SIEM / SOAR Modernization
@KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Digital Forensics 61 • Application of science to criminal and civil laws, mainly during criminal investigation • Forensic scientists collect, preserve, and analyze scientific evidence during the course of an investigating digital media in a forensically sound manner • Identify, preserve, recover, analyze and present facts and opinions about the digital information
@KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Distributed Digital Forensics at Scale with Kafka and Spark 62 • Digital Forensics Compute Cluster (DFORC2) • High Speed Distributed Computing Capability for Digital Forensics • Extended the digital forensics platform Autopsy with Kafka and Spark to add distributed compute power for data processing https://publications.waset.org/10007817/digital-forensics-compute-cluster-a-high-speed-distributed-computing-capability-for-digital-forensics
@KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Forensics on Historical Events Give me all events from time A to time B Real-time Producer Time • Capture the complete attack vector • Playback of an attack for the training of humans or machines • Create threat surface simulations • Compliance / regulatory processing Real-time Consumer for an automated actuation Consumer of Historical Data
@KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Confluent Tiered Storage for Kafka for Forensics of Historical Data 65 (Only available in Confluent Platform) Store data forever Hot and cold storage Cheap object store Easy scale up/down No changes in clients
@KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Direct streaming ingestion for model training with TensorFlow I/O + Kafka Plugin (no additional data storage like S3 or HDFS required!) Time Model B Model A Producer Distributed Commit Log The Role of AI and Machine Learning for Forensics Model Training with Kafka and TensorFlow I/O https://github.com/tensorflow/io 66 Model X (at a later time)
@KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka “CREATE STREAM AnomalyDetection AS SELECT facility_code, detectAnomaly(syslog_values) WHERE severity_level = ’Warning’ FROM syslog_source_topic;“ User Defined Function (UDF) 67 The Role of AI and Machine Learning for Forensics Model Deployment with ksqlDB and TensorFlow
@KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Agenda 1) Defence and Cybersecurity in 202X 2) Data in Motion as Defence Backbone 3) Situational Awareness 4) Threat Intelligence 5) Forensics 6) Air-Gapped and Zero Trust Environments 7) SIEM / SOAR Modernization
@KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Zero Trust 70 • EVERYTHING needs protecting, not just firewalls and computing assets • It is not cyber network security, but threat intelligence that includes human intelligence • Safe IT/OT integration at industrial sites • There is no such thing as a “unidirectional firewall” • Hardware and / or software-based • Replica servers instead of direct access • Surveillance for Safety and Theft Protection
@KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Unidirectional Networks for Air-gapped Environments When a Firewall is NOT Enough! 77 • Secure OT – IT bridge • Hardware based data diode or unidirectional gateway • Real time monitoring of safety-critical networks • Secure cloud connectivity of critical OT networks • Database replication and file transfer • Transferring application and operating system updates • Vendors use different terms: Unidirectional network = Unidirectional Gateway = Data Diode
@KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Confluent Data Diode https://docs.confluent.io/kafka-connect-data-diode Software-based Unidirectional Gateway for Zero Trust Security Architectures Streaming from Industrial Networks to Enterprise Networks UDP-based Source and Sink Kafka Connectors for High Volume and Open Architecture Run over a one-way/UDF hardware interface (Ethernet cable, OWL Cyber, Waterfall, etc.) Optionally/eventually do filtering, anomaly detection, analytics, receive upstream traffic, etc. Site Site Site Work Center Apache PLC4x Work Center Cloud Streams processing Data Lake Data Diode UDP Sink Data Diode UDP Source Kafka Cluster NUC Pair Kafka Instance
@KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Real-time Anomaly Detection Machine Learning Real-time Situational Awareness Event Driven Mission Distributed Command and Control Security Operations Adversarial Threats Assets, Weapons, Sensors, etc. Zero Trust Edge Architecture OBSERVE ORIENT DECIDE ACT Collect, analyze, and share data in real-time. Provide a fuller picture of the operating environment.
@KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka 85 Sensor A Sensor B Sensor X MQTT Confluent Platform (Single Broker) Confluent Platform deployed on a small computer and leveraging Cluster Linking to publish sensor data to Command Post in a DDIL environment. Command Post running Confluent Platform aggregating information from Squires and other sensor data Weather Personnel Logistics Targets Sensor data published to Command Post when connected to network Enhanced Situational Awareness Smart Soldiers at the Edge
@KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Agenda 1) Defence and Cybersecurity in 202X 2) Data in Motion as Defence Backbone 3) Situational Awareness 4) Threat Intelligence 5) Forensics 6) Air-Gapped and Zero Trust Environments 7) SIEM / SOAR Modernization
@KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka The Challenge with SIEM / SOAR Platforms Forwarder Network traffic Firewall logs RDBMS Application logs Adaptors Beats Machine Data HTTP proxy logs Splunk ArcSight Elastic Proprietary forwarders can only send data to single tool Data is locked from being shared Difficult to scale with growing data volumes High indexing costs of proprietary tools hinder wide adoption Filtering out noisy data is complex and slows response No one tool can support all security and SIEM requirements
@KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Forensic Archive AI/ML Build A Real-time SIEM / SOAR Pipeline Filter, transform aggregate APP SIEM Index Search Curated streams HDFS S3 Big Query CDC Syslog Network traffic Firewall logs RDBMS Application logs HTTP proxy logs QRadar Arcsight Splunk Elastic Machine Data spooldir (files), SNMP Traps, Databases, Sftp, MQs
@KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Old New Scan Scan Scan Each SIEM has its own position (offset) Raw-Big-Data-Topic Small-Data-Topic Preprocess and consolidate
@KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Confluent + Splunk SIEM Reference Architecture (assuming that Splunk UFs already exist – otherwise integrate directly via Kafka to Splunk) Splunk Universal Forwarders (UFs) Windows Event Logs SNMP Syslog Watchlist Zeek IDS Splunk Heavy Forwarders Machine Learning Splunk S2S Connector Splunk HEC Splunk Indexers Splunk Search Head Real-time stream processing with ksqlDB ... 3rd party apps / ecosystems Moving log data from Splunk UFs to your destination of choice
@KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Palo Alto Networks SOAR 97 Cortex Data Lake collects, transforms and integrates enterprise’s security data to enable Palo Alto Networks solution Billions of messages pass through the Kafka clusters Multiple Kafka clusters in production, size from 10 to just under a 100 brokers each Leverages various Confluent components Design principles: • Cloud agnostic infrastructure • Massively scalable • Aggressive ETA on integrations • Schema versioning support • Microservices architecture • Operational efficiency https://medium.com/engineering-at-palo-alto-networks
@KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Why Confluent?
@KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Car Engine Car Self-driving Car Confluent Completes Apache Kafka. Cloud-native. Everywhere.
Kai Waehner Field CTO kai.waehner@confluent.io @KaiWaehner confluent.io kai-waehner.de linkedin.com/in/kaiwaehner Questions? Feedback? Let’s connect!

More Related Content

What's hot (20)

PPTX
The Top 5 Apache Kafka Use Cases and Architectures in 2022
Kai Wähner
 
PDF
Apache Kafka in the Public Sector (Government, National Security, Citizen Ser...
Kai Wähner
 
PPTX
DW Migration Webinar-March 2022.pptx
Databricks
 
PDF
Can Apache Kafka Replace a Database?
Kai Wähner
 
PPTX
Streaming Data and Stream Processing with Apache Kafka
confluent
 
PDF
Modernizing to a Cloud Data Architecture
Databricks
 
PDF
Serverless Kafka and Spark in a Multi-Cloud Lakehouse Architecture
Kai Wähner
 
PDF
Processing IoT Data from End to End with MQTT and Apache Kafka
confluent
 
PDF
Serverless Kafka on AWS as Part of a Cloud-native Data Lake Architecture
Kai Wähner
 
PDF
Apache Kafka in the Healthcare Industry
Kai Wähner
 
PDF
When NOT to use Apache Kafka?
Kai Wähner
 
PDF
Architect’s Open-Source Guide for a Data Mesh Architecture
Databricks
 
PPTX
Microsoft Azure Databricks
Sascha Dittmann
 
PPTX
Microservices, Apache Kafka, Node, Dapr and more - Part Two (Fontys Hogeschoo...
Lucas Jellema
 
PPTX
Databricks Fundamentals
Dalibor Wijas
 
PDF
KSQL-ops! Running ksqlDB in the Wild (Simon Aubury, ThoughtWorks) Kafka Summi...
confluent
 
PDF
Kafka for Live Commerce to Transform the Retail and Shopping Metaverse
Kai Wähner
 
PDF
5 Critical Steps to Clean Your Data Swamp When Migrating Off of Hadoop
Databricks
 
PDF
Apache Kafka in Financial Services - Use Cases and Architectures
Kai Wähner
 
PDF
Making Apache Spark Better with Delta Lake
Databricks
 
The Top 5 Apache Kafka Use Cases and Architectures in 2022
Kai Wähner
 
Apache Kafka in the Public Sector (Government, National Security, Citizen Ser...
Kai Wähner
 
DW Migration Webinar-March 2022.pptx
Databricks
 
Can Apache Kafka Replace a Database?
Kai Wähner
 
Streaming Data and Stream Processing with Apache Kafka
confluent
 
Modernizing to a Cloud Data Architecture
Databricks
 
Serverless Kafka and Spark in a Multi-Cloud Lakehouse Architecture
Kai Wähner
 
Processing IoT Data from End to End with MQTT and Apache Kafka
confluent
 
Serverless Kafka on AWS as Part of a Cloud-native Data Lake Architecture
Kai Wähner
 
Apache Kafka in the Healthcare Industry
Kai Wähner
 
When NOT to use Apache Kafka?
Kai Wähner
 
Architect’s Open-Source Guide for a Data Mesh Architecture
Databricks
 
Microsoft Azure Databricks
Sascha Dittmann
 
Microservices, Apache Kafka, Node, Dapr and more - Part Two (Fontys Hogeschoo...
Lucas Jellema
 
Databricks Fundamentals
Dalibor Wijas
 
KSQL-ops! Running ksqlDB in the Wild (Simon Aubury, ThoughtWorks) Kafka Summi...
confluent
 
Kafka for Live Commerce to Transform the Retail and Shopping Metaverse
Kai Wähner
 
5 Critical Steps to Clean Your Data Swamp When Migrating Off of Hadoop
Databricks
 
Apache Kafka in Financial Services - Use Cases and Architectures
Kai Wähner
 
Making Apache Spark Better with Delta Lake
Databricks
 

Similar to Data Streaming with Apache Kafka in the Defence and Cybersecurity Industry (20)

PDF
Apache Kafka as the Backbone for Cybersecurity with Kai Waehner | Kafka Summi...
HostedbyConfluent
 
PDF
Apache Kafka for Cybersecurity and SIEM / SOAR Modernization
Kai Wähner
 
PPTX
SIEM Modernization: Build a Situationally Aware Organization with Apache Kafka®
confluent
 
PDF
Lessons Learned Fighting Modern Cyberthreats in Critical ICS Networks
Angeloluca Barba
 
PDF
El contexto de la integración masiva de datos
Software Guru
 
PDF
Next generation security analytics
Christian Have
 
PDF
Hunting for cyber threats targeting weapon systems
Fidelis Cybersecurity
 
PDF
Resiliency-Part One -11-3-2015
Dr Robert D. Childs
 
PDF
Secure Decisions - Cyber Security Sensemaking
Anita D'Amico
 
PDF
APAC Exec Roundtable
confluent
 
PDF
Gus Hunt's Work-Bench Enterprise Security Summit Keynote
Work-Bench
 
PDF
DETECTE E INVESTIGUE LAS AMENAZAS AVANZADAS
Cristian Garcia G.
 
PDF
ANZ C-Level Roundtable
confluent
 
PPTX
Operational Security Intelligence
Splunk
 
PPT
Jim grubb internet of everything
MassTLC
 
PDF
Architecting cybersecurity to future proof smart cities against emerging cybe...
NUS-ISS
 
PPTX
2016 10 31_mef_brief_nonotes_v2
David Stern
 
PPTX
Crisis-Ready Crisis-Proof IT Infrastructure for the New Normal
Kalin Hitrov
 
PDF
Cyber Defense Automation
♟Sergej Epp
 
PPT
Perimeter Defense in a World Without Walls
Dan Houser
 
Apache Kafka as the Backbone for Cybersecurity with Kai Waehner | Kafka Summi...
HostedbyConfluent
 
Apache Kafka for Cybersecurity and SIEM / SOAR Modernization
Kai Wähner
 
SIEM Modernization: Build a Situationally Aware Organization with Apache Kafka®
confluent
 
Lessons Learned Fighting Modern Cyberthreats in Critical ICS Networks
Angeloluca Barba
 
El contexto de la integración masiva de datos
Software Guru
 
Next generation security analytics
Christian Have
 
Hunting for cyber threats targeting weapon systems
Fidelis Cybersecurity
 
Resiliency-Part One -11-3-2015
Dr Robert D. Childs
 
Secure Decisions - Cyber Security Sensemaking
Anita D'Amico
 
APAC Exec Roundtable
confluent
 
Gus Hunt's Work-Bench Enterprise Security Summit Keynote
Work-Bench
 
DETECTE E INVESTIGUE LAS AMENAZAS AVANZADAS
Cristian Garcia G.
 
ANZ C-Level Roundtable
confluent
 
Operational Security Intelligence
Splunk
 
Jim grubb internet of everything
MassTLC
 
Architecting cybersecurity to future proof smart cities against emerging cybe...
NUS-ISS
 
2016 10 31_mef_brief_nonotes_v2
David Stern
 
Crisis-Ready Crisis-Proof IT Infrastructure for the New Normal
Kalin Hitrov
 
Cyber Defense Automation
♟Sergej Epp
 
Perimeter Defense in a World Without Walls
Dan Houser
 
Ad

More from Kai Wähner (18)

PDF
Apache Kafka as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!)
Kai Wähner
 
PDF
Apache Kafka vs. Cloud-native iPaaS Integration Platform Middleware
Kai Wähner
 
PDF
Resilient Real-time Data Streaming across the Edge and Hybrid Cloud with Apac...
Kai Wähner
 
PDF
Apache Kafka in the Healthcare Industry
Kai Wähner
 
PDF
Apache Kafka for Real-time Supply Chain in the Food and Retail Industry
Kai Wähner
 
PDF
Kafka for Real-Time Replication between Edge and Hybrid Cloud
Kai Wähner
 
PDF
Apache Kafka for Predictive Maintenance in Industrial IoT / Industry 4.0
Kai Wähner
 
PDF
Apache Kafka Landscape for Automotive and Manufacturing
Kai Wähner
 
PDF
Kappa vs Lambda Architectures and Technology Comparison
Kai Wähner
 
PDF
Event Streaming CTO Roundtable for Cloud-native Kafka Architectures
Kai Wähner
 
PDF
Telco 4.0 - Payment and FinServ Integration for Data in Motion with 5G and Ap...
Kai Wähner
 
PDF
Apache Kafka in the Automotive Industry (Connected Vehicles, Manufacturing 4....
Kai Wähner
 
PDF
IBM Cloud Pak for Integration with Confluent Platform powered by Apache Kafka
Kai Wähner
 
PDF
Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies?
Kai Wähner
 
PDF
Apache Kafka in the Insurance Industry
Kai Wähner
 
PDF
Apache Kafka and MQTT - Overview, Comparison, Use Cases, Architectures
Kai Wähner
 
PDF
Connected Vehicles and V2X with Apache Kafka
Kai Wähner
 
PDF
Apache Kafka in the Airline, Aviation and Travel Industry
Kai Wähner
 
Apache Kafka as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!)
Kai Wähner
 
Apache Kafka vs. Cloud-native iPaaS Integration Platform Middleware
Kai Wähner
 
Resilient Real-time Data Streaming across the Edge and Hybrid Cloud with Apac...
Kai Wähner
 
Apache Kafka in the Healthcare Industry
Kai Wähner
 
Apache Kafka for Real-time Supply Chain in the Food and Retail Industry
Kai Wähner
 
Kafka for Real-Time Replication between Edge and Hybrid Cloud
Kai Wähner
 
Apache Kafka for Predictive Maintenance in Industrial IoT / Industry 4.0
Kai Wähner
 
Apache Kafka Landscape for Automotive and Manufacturing
Kai Wähner
 
Kappa vs Lambda Architectures and Technology Comparison
Kai Wähner
 
Event Streaming CTO Roundtable for Cloud-native Kafka Architectures
Kai Wähner
 
Telco 4.0 - Payment and FinServ Integration for Data in Motion with 5G and Ap...
Kai Wähner
 
Apache Kafka in the Automotive Industry (Connected Vehicles, Manufacturing 4....
Kai Wähner
 
IBM Cloud Pak for Integration with Confluent Platform powered by Apache Kafka
Kai Wähner
 
Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies?
Kai Wähner
 
Apache Kafka in the Insurance Industry
Kai Wähner
 
Apache Kafka and MQTT - Overview, Comparison, Use Cases, Architectures
Kai Wähner
 
Connected Vehicles and V2X with Apache Kafka
Kai Wähner
 
Apache Kafka in the Airline, Aviation and Travel Industry
Kai Wähner
 
Ad

Recently uploaded (20)

PDF
Build It, Buy It, or Already Got It? Make Smarter Martech Decisions
bbedford2
 
PDF
Unlock Efficiency with Insurance Policy Administration Systems
Insurance Tech Services
 
PPTX
Engineering the Java Web Application (MVC)
abhishekoza1981
 
PPTX
Tally_Basic_Operations_Presentation.pptx
AditiBansal54083
 
PDF
iTop VPN With Crack Lifetime Activation Key-CODE
utfefguu
 
PPTX
Human Resources Information System (HRIS)
Amity University, Patna
 
PDF
Alarm in Android-Scheduling Timed Tasks Using AlarmManager in Android.pdf
Nabin Dhakal
 
PDF
Streamline Contractor Lifecycle- TECH EHS Solution
TECH EHS Solution
 
PDF
MiniTool Partition Wizard 12.8 Crack License Key LATEST
hashhshs786
 
PDF
Alexander Marshalov - How to use AI Assistants with your Monitoring system Q2...
VictoriaMetrics
 
PPT
MergeSortfbsjbjsfk sdfik k
RafishaikIT02044
 
PDF
Odoo CRM vs Zoho CRM: Honest Comparison 2025
Odiware Technologies Private Limited
 
PDF
Linux Certificate of Completion - LabEx Certificate
VICTOR MAESTRE RAMIREZ
 
PPTX
Fundamentals_of_Microservices_Architecture.pptx
MuhammadUzair504018
 
PDF
Understanding the Need for Systemic Change in Open Source Through Intersectio...
Imma Valls Bernaus
 
PDF
vMix Pro 28.0.0.42 Download vMix Registration key Bundle
kulindacore
 
PDF
Revenue streams of the Wazirx clone script.pdf
aaronjeffray
 
PPTX
Platform for Enterprise Solution - Java EE5
abhishekoza1981
 
PDF
Capcut Pro Crack For PC Latest Version {Fully Unlocked} 2025
hashhshs786
 
PDF
Executive Business Intelligence Dashboards
vandeslie24
 
Build It, Buy It, or Already Got It? Make Smarter Martech Decisions
bbedford2
 
Unlock Efficiency with Insurance Policy Administration Systems
Insurance Tech Services
 
Engineering the Java Web Application (MVC)
abhishekoza1981
 
Tally_Basic_Operations_Presentation.pptx
AditiBansal54083
 
iTop VPN With Crack Lifetime Activation Key-CODE
utfefguu
 
Human Resources Information System (HRIS)
Amity University, Patna
 
Alarm in Android-Scheduling Timed Tasks Using AlarmManager in Android.pdf
Nabin Dhakal
 
Streamline Contractor Lifecycle- TECH EHS Solution
TECH EHS Solution
 
MiniTool Partition Wizard 12.8 Crack License Key LATEST
hashhshs786
 
Alexander Marshalov - How to use AI Assistants with your Monitoring system Q2...
VictoriaMetrics
 
MergeSortfbsjbjsfk sdfik k
RafishaikIT02044
 
Odoo CRM vs Zoho CRM: Honest Comparison 2025
Odiware Technologies Private Limited
 
Linux Certificate of Completion - LabEx Certificate
VICTOR MAESTRE RAMIREZ
 
Fundamentals_of_Microservices_Architecture.pptx
MuhammadUzair504018
 
Understanding the Need for Systemic Change in Open Source Through Intersectio...
Imma Valls Bernaus
 
vMix Pro 28.0.0.42 Download vMix Registration key Bundle
kulindacore
 
Revenue streams of the Wazirx clone script.pdf
aaronjeffray
 
Platform for Enterprise Solution - Java EE5
abhishekoza1981
 
Capcut Pro Crack For PC Latest Version {Fully Unlocked} 2025
hashhshs786
 
Executive Business Intelligence Dashboards
vandeslie24
 

Data Streaming with Apache Kafka in the Defence and Cybersecurity Industry

  • 1. Data Streaming in the Defence Industry Data in Motion for Situational Awareness, Threat Detection, Forensics, Zero Trust Zones Kai Waehner Field CTO [email protected] @KaiWaehner confluent.io kai-waehner.de linkedin.com/in/kaiwaehner
  • 2. @KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Agenda 1) Defence and Cybersecurity in 202X 2) Data in Motion as Defence Backbone 3) Situational Awareness 4) Threat Intelligence 5) Forensics 6) Air-Gapped and Zero Trust Environments 7) SIEM / SOAR Modernization
  • 3. @KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Agenda 1) Defence and Cybersecurity in 202X 2) Data in Motion as Defence Backbone 3) Situational Awareness 4) Threat Intelligence 5) Forensics 6) Air-Gapped and Zero Trust Environments 7) SIEM / SOAR Modernization
  • 4. @KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Modern Warfare Information technology and real-time information as game changer for defence 4
  • 5. @KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Cybersecurity Protection of computer systems and networks from information disclosure and theft Web Scraping, hackers, criminals, terrorists, state-sponsored and state-initiated actors 5
  • 6. @KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka The need for secure and scalable defence Defence requires real-time information everywhere! Challenges Stealing IP DDoS Ransomware / wiperware WannaCry, NotPetya, SolarWinds … Damage: Billions of dollars ”Supply chain attack” Digital Transformation Networking Communication Connectivity Open standards ”Always-on” Billions of devices
  • 7. @KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Supply Chain Attack Targeting less-secure elements in the supply chain 7 https://www.nortonrosefulbright.com/en/knowledge/publications/dfa3603c/six-degrees-of-separation-cyber-risk-across-global-supply-chains https://www.reuters.com/article/us-tmobile-dataprotection-idUSKCN0RV5PL20151002
  • 8. @KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka SECURITY Security SIEM Encryption OT Security Hardware- based Security Cybersecurity How would you have a holistic view and understanding of all the events and potential abuses that are taking place within your organization? Collect and correlate the different activities happening on critical networks CYBERSECURITY is a key piece of the security strategy SIEM and SOAR a (key) piece of the cybersecurity strategy Sometimes not needed (in DMZ / air gapped env) Complex and error prone No help against insiders Continuous real-time data correlation required SOAR Avoid risk (change operations) + Transfer some risk (buy insurance) Real-time Monitoring (Logging, SiteOps, …) Access Control (RBAC, Audit Logs, …)
  • 9. @KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Key Challenge: Find the Needle(s) in the Haystack Detect true positives in real-time • Threat detection • Intrusion prevention • Anomaly detection • Compliance auditing • Proactive response Reduce false positives • Automation • Process big volumes of data in real-time • Integration of all sources • No ‘ignore’ on certain events • Creation of filters and correlated event rules • Improve signal-to-noise ratio (SNR) • Correlate “collection of needles” in “signature needle”
  • 10. @KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Agenda 1) Defence and Cybersecurity in 202X 2) Data in Motion as Defence Backbone 3) Situational Awareness 4) Threat Intelligence 5) Forensics 6) Air-Gapped and Zero Trust Environments 7) SIEM / SOAR Modernization
  • 11. @KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka This is a fundamental paradigm shift... 20 Infrastructure as code Data in motion as continuous streams of events Future of the datacenter Future of data Cloud Event Streaming
  • 12. @KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Real-time Data beats Slow Data. Cybersecurity Risk classification Threat detection Intrusion detection Incident response Command Post Intelligent Navigation Vehicle Inspection Location-based Services Logistics Supply Chain Inventory management Fleet Management Military Security monitoring Surveillance Command and Control Military Intelligence
  • 13. @KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Apache Kafka is the Platform for Data in Motion MES ERP Sensors Mobile Customer 360 Real-time Alerting System Data warehouse Producers Consumers Streams and storage of real time events Stream processing apps Connectors Connectors Stream processing apps Supplier Alert Forecast Inventory Customer Order 22
  • 14. @KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Data in Motion The Backbone for Defence Command Post Enterprise IT Strategic Planning Logs Personal Sensors Security Streams of real time events 23 Connected Vehicles Cyber Security Continuous Data Correlation Monitoring Alerting Proactive Actions
  • 15. @KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka End-to-End Cyber Defence with the Kafka Ecosystem Personel Crew, Cargo Vessel Fuel Consumption, Speed, Planned Maintenance Tracking Position, Course, Weather, Draft Drone or Satellite Relay COMMs Resilient Kafka Edge Analytics Bidirectional Ship Edge to Cloud, Shore Edge to Cloud Relay Ingestion Data Integration Streaming Analytics Machine Doing On-Prem Systems Bi-Directional Hybrid Cloud Replication ON SHORE ON PREM Staging, Filtering Shore Edge Analytics
  • 16. @KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Integrate with all legacy and modern interfaces Record, filter, curate a broad set of traffic streams Let analytic sinks consume just the right amount of data Drastically reduce the complexity of the enterprise architectures Drastically reduce the cost of SIEM / SOAR deployments Add new analytics engines Add stream-speed detection and response at scale in real-time Add mission-critical (non-) security-related applications … is the backbone for data streams in defence!
  • 17. @KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Every enterprise is different… Flexibility is key for your cybersecurity initiative! Confluent is an independent foundation. 30
  • 18. @KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Kafka Connect Confluent Various Data Producers Flexible Scalable Real-Time Backplane for the Defence Platform Splunk TensorFlow Kafka Forwarder TensorFlow + Kafka plugin Event Streaming Platform OT Domain SIEM Domain Analytics Domain 31 Huge volumes of real-time data from various Kafka topics Backpressure handling and a low velocity Kafka topic High velocity, raw Kafka topic for forensics and ML
  • 19. @KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Agenda 1) Defence and Cybersecurity in 202X 2) Data in Motion as Defence Backbone 3) Situational Awareness 4) Threat Intelligence 5) Forensics 6) Air-Gapped and Zero Trust Environments 7) SIEM / SOAR Modernization
  • 20. @KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Cyber Situational Awareness is the subset of all situation awareness necessary to support taking actions in cyber 36 Endsley, M. R. Toward a Theory of Situation Awareness in Dynamic Systems. Human Factors, 1995, 37(1), 32-64
  • 21. @KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Human – Computer Interface for Decision Making 38 https://www.youtube.com/watch?v=mPJdzzm67sg
  • 22. @KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Human – Computer Interface for Decision Making 39 https://www.youtube.com/watch?v=mPJdzzm67sg
  • 23. @KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Firewalls & Network Devices Antivirus Access Logs Intrusion Detection Audit Logs Text Files Binary Files Databases APIs Network Flows Syslog The Data
  • 24. @KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Transactions Low Velocity, Low Volume Netflow / PCAP High Velocity, High / Ridiculous Volume Ingested via Network Analyzer Gateway Logs Low Velocity, Moderate Volume Store PCAP headers in Tiered Storage or 3rd Party like Corelight as intermediary Data Producers
  • 25. @KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Streams Streams Streams Streams Event type-specific parsing and normalization logs-conn-shared logs-resolve-names logs-geoip-asn-iprep Streams Streams logs-index Established connection and client/server detection DNS name resolution GeoIP, IP Reputation and Autonomous System lookup Data Normalization and Enrichment à Improve the signal + filter to lower the noise
  • 26. @KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka SIEM Forwarder Threat Detection Near real-time Data Consumers No constraints on integration flows Data curation on the fly Flexible choice of (multiple) consumers Sink to Data Lake Analytical Workloads Batch Native Kafka App Transactional Workloads Real-time
  • 27. @KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Sigma 44 • Open-source framework • Domain specific language (DSL) • Specify patterns in cyber data https://github.com/SigmaHQ/sigma
  • 28. @KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Sigma Rule Detections • List of detections for each condition • Single or list of values • Individual values or regex • Detection names can also include operators (ex. name|endswith, name|contains, name|greater_than) • Aggregations and windowing Conditions • Nested conditions based on defined detections Detection Names • Generic Sigma names defined • Translated during parsing to meet end SIEM tool using field mapping file
  • 29. @KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Confluent Sigma Sigma Stream Processors Zeek Data and Detections Viewer Sigma Rule Editor sigma rules topic DNS dns detections topic dns topic rule parsing, filtering, aggregation, windowing sigma rules cache CONN DHCP HTTP SSL x509 Zeek Data https://github.com/confluentinc/cyber/tree/master/confluent-sigma
  • 30. @KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Agenda 1) Defence and Cybersecurity in 202X 2) Data in Motion as Defence Backbone 3) Situational Awareness 4) Threat Intelligence 5) Forensics 6) Air-Gapped and Zero Trust Environments 7) SIEM / SOAR Modernization
  • 31. @KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Threat Intelligence 54 Mitigate harmful events in cyberspace Proactive cybersecurity posture that is predictive, not just reactive Bolster overall risk management policies Improved detection of threats Better decision-making during and following the detection of a cyber intrusion See the whole board, more quickly. See around corners. See the enemy before they see you.
  • 32. @KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Transactions vs. Analytics 55 Threat intelligence = awareness-in-motion The PATTERN is valuable, not the data.
  • 33. @KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Streams logs-index Authorized access using RBAC Machine Learning Predictions via UDFs PII Anonymization logs-alerts logs-index-gdpr Analytics and Actionable Insights in Motion Make sense of the signal and the noise of the data Continuous signature processing Prevent, contain, and neutralize threats proactively Access for data science teams
  • 34. @KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Cyber Intelligence Platform leveraging Kafka Connect, Kafka Streams, Multi-Region Clusters (MRC), and more… https://www.intel.com/content/www/us/en/it-management/intel-it-best-practices/modern-scalable-cyber-intelligence-platform-kafka.html
  • 35. @KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Agenda 1) Defence and Cybersecurity in 202X 2) Data in Motion as Defence Backbone 3) Situational Awareness 4) Threat Intelligence 5) Forensics 6) Air-Gapped and Zero Trust Environments 7) SIEM / SOAR Modernization
  • 36. @KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Digital Forensics 61 • Application of science to criminal and civil laws, mainly during criminal investigation • Forensic scientists collect, preserve, and analyze scientific evidence during the course of an investigating digital media in a forensically sound manner • Identify, preserve, recover, analyze and present facts and opinions about the digital information
  • 37. @KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Distributed Digital Forensics at Scale with Kafka and Spark 62 • Digital Forensics Compute Cluster (DFORC2) • High Speed Distributed Computing Capability for Digital Forensics • Extended the digital forensics platform Autopsy with Kafka and Spark to add distributed compute power for data processing https://publications.waset.org/10007817/digital-forensics-compute-cluster-a-high-speed-distributed-computing-capability-for-digital-forensics
  • 38. @KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Forensics on Historical Events Give me all events from time A to time B Real-time Producer Time • Capture the complete attack vector • Playback of an attack for the training of humans or machines • Create threat surface simulations • Compliance / regulatory processing Real-time Consumer for an automated actuation Consumer of Historical Data
  • 39. @KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Confluent Tiered Storage for Kafka for Forensics of Historical Data 65 (Only available in Confluent Platform) Store data forever Hot and cold storage Cheap object store Easy scale up/down No changes in clients
  • 40. @KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Direct streaming ingestion for model training with TensorFlow I/O + Kafka Plugin (no additional data storage like S3 or HDFS required!) Time Model B Model A Producer Distributed Commit Log The Role of AI and Machine Learning for Forensics Model Training with Kafka and TensorFlow I/O https://github.com/tensorflow/io 66 Model X (at a later time)
  • 41. @KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka “CREATE STREAM AnomalyDetection AS SELECT facility_code, detectAnomaly(syslog_values) WHERE severity_level = ’Warning’ FROM syslog_source_topic;“ User Defined Function (UDF) 67 The Role of AI and Machine Learning for Forensics Model Deployment with ksqlDB and TensorFlow
  • 42. @KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Agenda 1) Defence and Cybersecurity in 202X 2) Data in Motion as Defence Backbone 3) Situational Awareness 4) Threat Intelligence 5) Forensics 6) Air-Gapped and Zero Trust Environments 7) SIEM / SOAR Modernization
  • 43. @KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Zero Trust 70 • EVERYTHING needs protecting, not just firewalls and computing assets • It is not cyber network security, but threat intelligence that includes human intelligence • Safe IT/OT integration at industrial sites • There is no such thing as a “unidirectional firewall” • Hardware and / or software-based • Replica servers instead of direct access • Surveillance for Safety and Theft Protection
  • 44. @KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Unidirectional Networks for Air-gapped Environments When a Firewall is NOT Enough! 77 • Secure OT – IT bridge • Hardware based data diode or unidirectional gateway • Real time monitoring of safety-critical networks • Secure cloud connectivity of critical OT networks • Database replication and file transfer • Transferring application and operating system updates • Vendors use different terms: Unidirectional network = Unidirectional Gateway = Data Diode
  • 45. @KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Confluent Data Diode https://docs.confluent.io/kafka-connect-data-diode Software-based Unidirectional Gateway for Zero Trust Security Architectures Streaming from Industrial Networks to Enterprise Networks UDP-based Source and Sink Kafka Connectors for High Volume and Open Architecture Run over a one-way/UDF hardware interface (Ethernet cable, OWL Cyber, Waterfall, etc.) Optionally/eventually do filtering, anomaly detection, analytics, receive upstream traffic, etc. Site Site Site Work Center Apache PLC4x Work Center Cloud Streams processing Data Lake Data Diode UDP Sink Data Diode UDP Source Kafka Cluster NUC Pair Kafka Instance
  • 46. @KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Real-time Anomaly Detection Machine Learning Real-time Situational Awareness Event Driven Mission Distributed Command and Control Security Operations Adversarial Threats Assets, Weapons, Sensors, etc. Zero Trust Edge Architecture OBSERVE ORIENT DECIDE ACT Collect, analyze, and share data in real-time. Provide a fuller picture of the operating environment.
  • 47. @KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka 85 Sensor A Sensor B Sensor X MQTT Confluent Platform (Single Broker) Confluent Platform deployed on a small computer and leveraging Cluster Linking to publish sensor data to Command Post in a DDIL environment. Command Post running Confluent Platform aggregating information from Squires and other sensor data Weather Personnel Logistics Targets Sensor data published to Command Post when connected to network Enhanced Situational Awareness Smart Soldiers at the Edge
  • 48. @KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Agenda 1) Defence and Cybersecurity in 202X 2) Data in Motion as Defence Backbone 3) Situational Awareness 4) Threat Intelligence 5) Forensics 6) Air-Gapped and Zero Trust Environments 7) SIEM / SOAR Modernization
  • 49. @KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka The Challenge with SIEM / SOAR Platforms Forwarder Network traffic Firewall logs RDBMS Application logs Adaptors Beats Machine Data HTTP proxy logs Splunk ArcSight Elastic Proprietary forwarders can only send data to single tool Data is locked from being shared Difficult to scale with growing data volumes High indexing costs of proprietary tools hinder wide adoption Filtering out noisy data is complex and slows response No one tool can support all security and SIEM requirements
  • 50. @KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Forensic Archive AI/ML Build A Real-time SIEM / SOAR Pipeline Filter, transform aggregate APP SIEM Index Search Curated streams HDFS S3 Big Query CDC Syslog Network traffic Firewall logs RDBMS Application logs HTTP proxy logs QRadar Arcsight Splunk Elastic Machine Data spooldir (files), SNMP Traps, Databases, Sftp, MQs
  • 51. @KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Old New Scan Scan Scan Each SIEM has its own position (offset) Raw-Big-Data-Topic Small-Data-Topic Preprocess and consolidate
  • 52. @KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Confluent + Splunk SIEM Reference Architecture (assuming that Splunk UFs already exist – otherwise integrate directly via Kafka to Splunk) Splunk Universal Forwarders (UFs) Windows Event Logs SNMP Syslog Watchlist Zeek IDS Splunk Heavy Forwarders Machine Learning Splunk S2S Connector Splunk HEC Splunk Indexers Splunk Search Head Real-time stream processing with ksqlDB ... 3rd party apps / ecosystems Moving log data from Splunk UFs to your destination of choice
  • 53. @KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Palo Alto Networks SOAR 97 Cortex Data Lake collects, transforms and integrates enterprise’s security data to enable Palo Alto Networks solution Billions of messages pass through the Kafka clusters Multiple Kafka clusters in production, size from 10 to just under a 100 brokers each Leverages various Confluent components Design principles: • Cloud agnostic infrastructure • Massively scalable • Aggressive ETA on integrations • Schema versioning support • Microservices architecture • Operational efficiency https://medium.com/engineering-at-palo-alto-networks
  • 54. @KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Why Confluent?
  • 55. @KaiWaehner - www.kai-waehner.de – Cybersecurity and SIEM / SOAR Modernization with Apache Kafka Car Engine Car Self-driving Car Confluent Completes Apache Kafka. Cloud-native. Everywhere.