Real-time Geospatial Aircraft Monitoring Using Apache Kafka
0 likes296 views
The document outlines a presentation given at the Kafka Summit London in March 2024, focusing on real-time geospatial aircraft monitoring using Apache Kafka. It addresses various challenges, including data formats (XML and JSON), throughput and latency issues, authentication and authorization, and real-time data enrichment. The presentation includes technical configurations and performance optimizations for handling massive data streams effectively.
![March
2024
·
Kafka
Summit
London
Message Format XML
● Custom Serializers - when XML parsing is "read-only"
○ XMLWrapper(byte[] bytes, Document document)
○ wrapper.serialize(xmlWrapper)](https://image.slidesharecdn.com/bs32-20240319-ksl24-raft-sharma-bhaarat-kineticedge-buesing-neil-240402154727-15846888/85/Real-time-Geospatial-Aircraft-Monitoring-Using-Apache-Kafka-14-320.jpg)
![March
2024
·
Kafka
Summit
London
Authorization – Open Policy Agent
Challenge 3 – Authentication & Authorization
● OPA Rego ☞ reminds me of Prolog
● .rego examples
consumer_operations = {
"TOPIC" : [ "READ", "DESCRIBE" ],
"GROUP" : [ "READ", "DESCRIBE" ]
}
is_consumer_group {
user_groups(principal)[_] == topic_consumer_groups(topic_name)[_]
}
allow_consumer_group {
is_group_resource
is_consumer_operation
startswith(group_name, concat("-", [principal, ""]))
}](https://image.slidesharecdn.com/bs32-20240319-ksl24-raft-sharma-bhaarat-kineticedge-buesing-neil-240402154727-15846888/85/Real-time-Geospatial-Aircraft-Monitoring-Using-Apache-Kafka-25-320.jpg)
Real-time Geospatial Aircraft Monitoring Using Apache Kafka
- 1. March 2024 · Kafka Summit London Real-time Geospatial Aircraft Monitoring Using Apache Kafka Bhaarat Sharma – CTO & Co-Founder Raft Neil Buesing – CTO & Co-Founder Kinetic Edge
- 2. March 2024 · Kafka Summit London We bridge the gap between humans and data through radical transparency and our obsession with the mission..
- 4. March 2024 · Kafka Summit London Inspired by: Perishable Insights, Mike Gualtier, Forrester
- 7. March 2024 · Kafka Summit London SQL over Kafka (backed by Pinot)
- 8. March 2024 · Kafka Summit London Fine grained access to dashboard
- 10. March 2024 · Kafka Summit London Architecture - Focus for today
- 13. March 2024 · Kafka Summit London Message Format XML ● XML ⇒ JSON ✔ ● JSON ⇒ XML ✖ consumers expected XML (& valid against XSD schema) ● Additional Challenges ○ marshaling bytes into XML ○ schema validation speed ○ thread safety
- 14. March 2024 · Kafka Summit London Message Format XML ● Custom Serializers - when XML parsing is "read-only" ○ XMLWrapper(byte[] bytes, Document document) ○ wrapper.serialize(xmlWrapper)
- 15. March 2024 · Kafka Summit London Message Format XML ● XOM ○ Slightly better thread safety ○ Faster Implementation ○ Still built on existing Parser technologies ■ so not fully better thread safety
- 16. March 2024 · Kafka Summit London Message Format XML ● XSD Validation ○ Only on incoming or outgoing topics.
- 17. March 2024 · Kafka Summit London Message Format XML ● Staying with XML easier than converting back to XML. ● Optimize Read-Only for speed by compromising on Storage. ● Lessons learned here would apply to other formats; just more obvious with XML.
- 18. March 2024 · Kafka Summit London Throughput and Latency Challenge 2 100k msgs/sec 150+ Sensors Multiple Sensor Fusion Engines Enhanced Search
- 19. March 2024 · Kafka Summit London Configurations - Topics Challenge 2 – Throughput and Latency ● 12 partitions ○ even distribution across availability zones (÷3) ○ even consumer workloads ■ 1/2/3/4/6/12 (6) ● Currently evaluating 24 for some topics ○ also evenly distributed across availability zones (÷3) ○ 1/2/3/4/6/8/12/24 (8) ■ 30->1/2/3/5/6/10/15/30 (7) ■ 36->1/2/3/4/6/9/12/18/36 (9)
- 20. March 2024 · Kafka Summit London Configurations - Producer Challenge 2 – Throughput and Latency ● buffer.size=200_000 ○ or more ● linger.ms=10-50 ○ balance of latency & throughput ● compression.type=lz4 ○ Always test your data against your compression ○ Never compress compressed data
- 21. March 2024 · Kafka Summit London Configurations - Consumer Challenge 2 – Throughput and Latency ● 12 partitions ○ even distribution across availability zones (÷3) ○ even consumer workloads ■ 1/2/3/4/6/12 ● max.partition.fetch.bytes & fetch.max.bytes ○ adding partitions can increase latency (especially if number of consumers isn't increased)
- 22. March 2024 · Kafka Summit London Configurations - Performance Challenge 2 – Throughput and Latency ● Start with the producer ● Then the partitions ● Kafka Streams ○ State Store - Caching (Disable for Latency) ○ Commit Interval (Reduce for Visibility and Latency) ○ Threading - depends on topology & number of containers.
- 24. March 2024 · Kafka Summit London Authentication - Keycloak Challenge 3 – Authentication & Authorization ● OAuth Callback Handler ○ Kafka 3.2.1+ ○ self-signed certificates ... ● Custom Callback Handler ○ error handling considerations ● Librdkafka (Non-Java) ○ Leverage callback handler doing RESTful operation (custom callback)
- 25. March 2024 · Kafka Summit London Authorization – Open Policy Agent Challenge 3 – Authentication & Authorization ● OPA Rego ☞ reminds me of Prolog ● .rego examples consumer_operations = { "TOPIC" : [ "READ", "DESCRIBE" ], "GROUP" : [ "READ", "DESCRIBE" ] } is_consumer_group { user_groups(principal)[_] == topic_consumer_groups(topic_name)[_] } allow_consumer_group { is_group_resource is_consumer_operation startswith(group_name, concat("-", [principal, ""])) }
- 26. March 2024 · Kafka Summit London Pattern of life analysis – Warm Data Challenge 4 Source: ADSB Exchange
- 27. March 2024 · Kafka Summit London Custom Kafka Rest API Challenge 4 – Pattern of Life Analysis ● One of the easiest parts to build ○ Producer ■ linger.ms - major impact - chose wisely ○ Consumer -- not RESTful ☞ Websockets ● One of the easiest mistakes to make - waiting... ○ producer flushing ○ linger.ms ○ Callback vs. Waiting on Future Leverage Framework, e.g. Spring's Deferred Result ● Http Response Codes ○ 200 - OK ○ 201 - Created (try to avoid using this one, but some clients....) ○ 202 - Accepted
- 28. March 2024 · Kafka Summit London Real Time Data – Hot Data Challenge 5 Source: ADSB Exchange
- 29. March 2024 · Kafka Summit London Websocket - Consumer Challenge 5 – Real Time Data – Hot Data ● consumer.subscribe() vs. consumer.assign() ● handling backpressure ● tried Java 21 and Virtual Threads - did not help... ● 2 implementation ○ web-socket consumption drains queue ○ 30 second eviction (independent of consumption) ● garbage collection topic poll() websocket LinkedBlockingQueue push()
- 30. March 2024 · Kafka Summit London Websocket - Consumer Challenge 5 – Real Time Data – Hot Data ● Data Mashing over Websocket ○ XML - not great ○ JSON - better ○ Apache Arrow - best topic poll() websocket LinkedBlockingQueue push() consumer thread thread / websocket
- 32. March 2024 · Kafka Summit London Kafka Streams Challenge 6 – Real Time Data Enrichment ● Avoid initial rekeying (trust but verify), every rekey adds latency. filter((k, v) -> { if (k ≠ v.id()) { log.error("invalid..."); return false; } ● Global Tables / KTables
- 33. March 2024 · Kafka Summit London Kafka Streams Challenge 6 – Real Time Data Enrichment ● Commit Interval ○ 100ms - 5000ms ● Threads ○ Containers vs Stream Threads ● If Scaling down and up (and not using static membership) internal.leave.group.on.close = true ● If Scaling down and up (and static membership + Kafka Streams 3.3+) KafkaStreams.CloseOptions closeOptions = new KafkaStreams.CloseOptions().timeout(SHUTDOWN).leaveGroup(true); streams.close(closeOptions);