![Apache Beam
public class PageViewCount {
public static void main(String[] args) {
...
pipeline
.apply(LiKafkaIO.<PageViewEvent>read()
.withTopic("PageView")
.withTimestampFn(kv -> new Instant(kv.getValue().header.time))
.withWatermarkFn(kv -> new Instant(kv.getValue().header.time - 60000))
.apply(Values.create())
.apply(MapElements
.into(TypeDescriptors.kvs(TypeDescriptors.strings(), TypeDescriptors.integers()))
.via((PageViewEvent pv) -> KV.of(String.valueOf(pv.header.memberId), 1)))
.apply(Window.into(TumblingWindows.of(Duration.standardMinutes(5))))
.apply(Count.perKey())
.apply(MapElements
.into(TypeDescriptors.kvs(TypeDescriptors.strings(), TypeDescriptor.of(Counter.class)))
.via(newCounter()))
.apply(LiKafkaIO.<Counter>write().withTopic("PageViewCount")
pipeline.run();
}
}](https://image.slidesharecdn.com/scalablestreamprocessingwithapachesamza-190820181131/85/Scalable-Stream-Processing-with-Apache-Samza-22-320.jpg)
![Apache Beam: Python
p = Pipeline(options=pipeline_options)
(p
| 'read' >> ReadFromKafka(cluster="tracking",
topic="PageViewEvent", config=config)
| 'extract' >> beam.Map(lambda record: (record.value['memberId'], 1))
| "windowing" >> beam.WindowInto(window.FixedWindows(60*5))
| "compute" >> beam.CombinePerKey(beam.combiners.CountCombineFn())
| 'write' >> WriteToKafka(cluster = "queuing",
topic = "PageViewCount", config = config)
p.run().waitUntilFinish()](https://image.slidesharecdn.com/scalablestreamprocessingwithapachesamza-190820181131/85/Scalable-Stream-Processing-with-Apache-Samza-23-320.jpg)
Scalable Stream Processing with Apache Samza
- 1. Scalable Stream Processing With Apache Samza Prateek Maheshwari Apache Samza PMC
- 2. Agenda ● Stream Processing at LinkedIn ○ Scale at LinkedIn ○ Scenarios at LinkedIn ● Apache Samza ○ Processing Model ○ Stateful Processing ○ Processing APIs ○ Deployment Model
- 3. Apache Kafka 5 Trillion+ messages ingested per day 1.5+ PB data per day 100k+ topics, 5M+ partitions Brooklin 2 Trillion+ messages moved per day 10k+ topics mirrored 2k+ change capture streams Apache Samza 1.5 Trillion+ messages processed per day 3k+ jobs in production 500 TB+ local state Scale at LinkedIn
- 4. Scenarios at LinkedIn DDoS prevention, bot detection, access monitoring Security Email and Push notifications Notifications Topic tagging, NER in news articles, image classification Classification Site speed and health monitoring Site Speed Monitoring inter-service dependencies and SLAs Call Graphs
- 5. Scenarios at LinkedIn Tracking ad views and clicks Ad CTR Tracking Pre-aggregated real-time counts by dimensions Business Metrics Standardizing titles, companies, education Profile Standardization Updating search indices with new data Index Updates Tracking member page views, dwell-time, sessions Activity Tracking
- 6. Hardened at Scale In production at LinkedIn, Slack, Intuit, TripAdvisor, VMWare, Redfin, etc. Processing events from Kafka, Brooklin, Kinesis, EventHubs, HDFS, DynamoDB Streams, Databus, etc. Apache Samza Incremental checkpoints for large local state and instant recovery. Local state that works seamlessly across upgrades and failures. APIs for simple and efficient remote I/O Best In Class Stateful Processing Stream and batch processing without changing code. Convenient High-level DSLs and a powerful Low-level API. Universal Processing APIs Write once, run anywhere. Run on a multi-tenant cluster or as an embedded library. Flexible Deployment Model
- 7. Brooklin Hadoop Task-1 Task-2 Task-3 Container-1 Container-2 Kafka Heartbeat Job Coordinator Samza Application Processing Model Kafka Hadoop Serving Stores (e.g. Espresso, Venice, Pinot) Elasticsearch
- 8. ● Parallelism across tasks by increasing the number of containers. ○ Up to 1 container per task. ● Parallelism across partitions by increasing the number of tasks. ○ Up to 1 task per partition. ● Parallelism within a partition for out of order processing. ○ Any number of threads. Scaling a Samza Application
- 9. Hardened at Scale In production at LinkedIn, Uber, Slack, Intuit, TripAdvisor, VMWare, Redfin, etc. Processing events from Kafka, Brooklin, Kinesis, EventHubs, HDFS, DynamoDB Streams, Databus, etc. Apache Samza Incremental checkpoints for large local state and instant recovery. Local state that works seamlessly across upgrades and failures. APIs for simple and efficient remote I/O Best In Class Stateful Processing Stream and batch processing without changing code. Convenient High-level DSLs and a powerful Low-level API. Universal Processing APIs Write once, run anywhere. Run on a multi-tenant cluster or as an embedded library. Flexible Deployment Model
- 10. • State is used for performing lookups and joins, caching data, buffering/batching data, and writing computed results. • State can be local (in-memory or on disk) or remote. Samza Local Store I/O Samza Why State Matters and Remote DB I/O
- 11. Why Local State Matters: Throughput on disk w/ caching comparable with in memory changelog adds minimal overhead remote state 30-150x worse than local state Terminology Disk Type: SSD Max-Net: Max network bandwidth CLog: Kafka changelog ReadOnly: read only workloads (lookups) ReadWrite: read - write workloads (counts) Shadi A. Noghabi et al. Samza: stateful scalable stream processing at LinkedIn. Proc. VLDB Endow. 10, 12 (August 2017), 1634-1645.
- 12. Why Local State Matters: Latency on disk w/ caching comparable with in memory changelog adds minimal overhead > 2 orders of magnitude slower compared to local state Shadi A. Noghabi et al. Samza: stateful scalable stream processing at LinkedIn. Proc. VLDB Endow. 10, 12 (August 2017), 1634-1645.
- 13. Optimizations for Local State Task-1 Container-1 Samza Application Master Durable Container ID – host mapping 1. Log state changes to a Kafka compacted topic for durability. 2. Catch up on only the delta from the change log topic on restart. Task-2 Container-2
- 14. Optimizations for Local State 1. Host Affinity 2. Parallel Recovery 3. Bulk Load Mode 4. Standby Containers 5. Log Compaction Task-1 Container-1 Samza Application Master Durable Container ID – host mapping Task-2 Container-2
- 15. Why Remote I/O Matters • Data is only available in the remote store (no change capture). • Need strong consistency or transactions. • Data cannot be partitioned but is too large to copy to every container. • Writing processed results for online serving. • Calling other services to handle complex business logic.
- 16. Optimizations for Remote I/O: Table API • Async Requests • Rate Limiting • Batching • Caching • Retries • Stream Table Joins
- 17. Hardened at Scale In production at LinkedIn, Uber, Slack, Intuit, TripAdvisor, VMWare, Redfin, etc. Processing events from Kafka, Brooklin, Kinesis, EventHubs, HDFS, DynamoDB Streams, Databus, etc. Apache Samza Incremental checkpoints for large local state and instant recovery. Local state that works seamlessly across upgrades and failures. APIs for simple and efficient remote I/O Best In Class Stateful Processing Stream and batch processing without changing code. Convenient High-level DSLs and a powerful Low-level API. Universal Processing APIs Write once, run anywhere. Run on a multi-tenant cluster or as an embedded library. Flexible Deployment Model
- 18. Example Application Count number of "Page Views" for each member in a 5 minute window 18 Page View Page View Per Member Repartition by member id Window Map SendTo Intermediate Stream
- 19. High Level API ● Complex Processing Pipelines ● Easy Repartitioning ● Stream-Stream and Stream-Table Joins ● Processing Time Windows and Joins
- 20. High Level API public class PageViewCountApplication implements StreamApplication { @Override public void describe(StreamApplicationDescriptor appDescriptor) { KafkaSystemDescriptor ksd = new KafkaSystemDescriptor("tracking"); KafkaInputDescriptor<PageViewEvent> pageViews = ksd.getInputDescriptor("PageView", serde); KafkaOutputDescriptor<PageViewCount> pageViewCounts = ksd.getOutputDescriptor("PageViewCount", serde); appDescriptor.getInputStream(pageViews) .partitionBy(m -> m.memberId, serde) .window(Windows.keyedTumblingWindow(m -> m.memberId, Duration.ofMinutes(5), initialValue, (m, c) -> c + 1)) .map(PageViewCount::new) .sendTo(appDescriptor.getOutputStream(pageViewCounts)); } }
- 21. Apache Beam ● Event Time Processing ● Multi-lingual APIs (Java, Python, Go*) ● Advanced Windows and Joins
- 22. Apache Beam public class PageViewCount { public static void main(String[] args) { ... pipeline .apply(LiKafkaIO.<PageViewEvent>read() .withTopic("PageView") .withTimestampFn(kv -> new Instant(kv.getValue().header.time)) .withWatermarkFn(kv -> new Instant(kv.getValue().header.time - 60000)) .apply(Values.create()) .apply(MapElements .into(TypeDescriptors.kvs(TypeDescriptors.strings(), TypeDescriptors.integers())) .via((PageViewEvent pv) -> KV.of(String.valueOf(pv.header.memberId), 1))) .apply(Window.into(TumblingWindows.of(Duration.standardMinutes(5)))) .apply(Count.perKey()) .apply(MapElements .into(TypeDescriptors.kvs(TypeDescriptors.strings(), TypeDescriptor.of(Counter.class))) .via(newCounter())) .apply(LiKafkaIO.<Counter>write().withTopic("PageViewCount") pipeline.run(); } }
- 23. Apache Beam: Python p = Pipeline(options=pipeline_options) (p | 'read' >> ReadFromKafka(cluster="tracking", topic="PageViewEvent", config=config) | 'extract' >> beam.Map(lambda record: (record.value['memberId'], 1)) | "windowing" >> beam.WindowInto(window.FixedWindows(60*5)) | "compute" >> beam.CombinePerKey(beam.combiners.CountCombineFn()) | 'write' >> WriteToKafka(cluster = "queuing", topic = "PageViewCount", config = config) p.run().waitUntilFinish()
- 24. Samza SQL ● Declarative streaming SQL API ● Managed service at LinkedIn ● Create and deploy applications in minutes using SQL Shell
- 25. Samza SQL INSERT INTO kafka.tracking.PageViewCount SELECT memberId, count(*) FROM kafka.tracking.PageView GROUP BY memberId, TUMBLE(current_timestamp, INTERVAL '5' MINUTES)
- 26. Low Level High Level Samza SQL Apache Beam Java Python Samza APIs
- 27. Hardened at Scale In production at LinkedIn, Uber, Slack, Intuit, TripAdvisor, VMWare, Redfin, etc. Processing events from Kafka, Brooklin, Kinesis, EventHubs, HDFS, DynamoDB Streams, Databus, etc. Apache Samza Incremental checkpoints for large local state and instant recovery. Local state that works seamlessly across upgrades and failures. APIs for simple and efficient remote I/O Best In Class Stateful Processing Stream and batch processing without changing code. Convenient High-level DSLs and a powerful Low-level API. Universal Processing APIs Write once, run anywhere. Run on a multi-tenant cluster or as an embedded library. Flexible Deployment Model
- 28. Samza on a Multi-Tenant Cluster • Uses a cluster manager (e.g. YARN) for resource management, coordination, liveness monitoring, etc. • Better resource utilization in a multi-tenant environment. • Works well for large number of applications.
- 29. Samza as an Embedded Library • Embed Samza as a library in an application. No cluster manager dependency. • Dynamically scale out applications by increasing or decreasing the number of processors at run-time. • Supports rolling upgrades and canaries.
- 30. ● Uses ZooKeeper for leader election and liveness monitoring for processors. ● Leader JobCoordinator performs work assignments among processors. ● Leader redistributes partitions when processors join or leave the group. Samza as a Library ZooKeeper Based Coordination Zookeeper StreamProcessor Samza Container Job Coordinator StreamProcessor Samza Container Job Coordinator StreamProcessor Samza Container Job Coordinator… Leader
- 31. Apache Samza • Mature, versatile, and scalable processing framework • Best-in-class support for local and remote state • Powerful and flexible APIs • Can be operated as a platform or used as an embedded library