Realizing the promise of portable data processing with Apache Beam
0 likes509 views
Apache Beam is a unified programming model that facilitates the creation of efficient and portable data processing pipelines across various languages and platforms. The document discusses the capabilities of Apache Beam, including its architecture, the importance of portability, and the integration within the big data ecosystem. It also covers the evolution of the project, the model's operation, and future challenges in data processing.
![● Beam’s Java SDK runs on multiple
runtime environments, including:
○ Apache Apex
○ Apache Spark
○ Apache Flink
○ Google Cloud Dataflow
○ [in development] Apache Gearpump
● Cross-language infrastructure is in
progress.
○ Beam’s Python SDK currently runs
on Google Cloud Dataflow
Beam Vision: as of June 2017
Beam Model: Fn Runners
Apache
Spark
Cloud
Dataflow
Beam Model: Pipeline Construction
Apache
Flink
Java
Java
Python
Python
Apache
Apex
Apache
Gearpump](https://image.slidesharecdn.com/davorbonacirealizingthepromiseofportabledataprocessingwithapachebeam-170621223726/85/Realizing-the-promise-of-portable-data-processing-with-Apache-Beam-24-320.jpg)
Realizing the promise of portable data processing with Apache Beam
- 1. Abstract The world of big data involves an ever changing field of players. Much as SQL stands as a lingua franca for declarative data analysis, Apache Beam aims to provide a portable standard for expressing robust, out-of-order data processing pipelines in a variety of languages across a variety of platforms. In a way, Apache Beam is a glue that can connect the Big Data ecosystem together; it enables users to "run-anything-anywhere". This talk will briefly cover the capabilities of the Beam model for data processing, as well as the current state of the Beam ecosystem. We'll discuss Beam architecture and dive into the portability layer. We'll offer a technical analysis of the Beam's powerful primitive operations that enable true and reliable portability across diverse environments. Finally, we'll demonstrate a complex pipeline running on multiple runners in multiple deployment scenarios (e.g. Apache Spark on Amazon Web Services, Apache Flink on Google Cloud, Apache Apex on-premise), and give a glimpse at some of the challenges Beam aims to address in the future. This session is a (Intermediate) talk in our IoT and Streaming track. It focuses on Apache Flink, Apache Kafka, Apache Spark, Cloud, Other and is geared towards Architect, Data Scientist, Data Analyst, Developer / Engineer, Operations / IT audiences.
- 2. Realizing the promise of portable data processing with Apache Beam Davor Bonaci PMC Chair, Apache Beam Senior Software Engineer, Google Inc.
- 3. Apache Beam: Open Source data processing APIs ● Expresses data-parallel batch and streaming algorithms using one unified API ● Cleanly separates data processing logic from runtime requirements ● Supports execution on multiple distributed processing runtime environments
- 4. Apache Beam is a unified programming model designed to provide efficient and portable data processing pipelines
- 5. Agenda 1. Road to the first stable release 2. Expressing data-parallel pipelines with the Beam model 3. The Beam vision for portability a. Parallel and portable pipelines in practice 4. Extensibility to integrate the entire Big Data ecosystem
- 6. Apache Beam at DataWorks Summit ● Realizing the promise of portable data processing with Apache Beam ○ Speaker: Davor Bonaci, Google ○ Wednesday @ 11:30 am ● Stateful processing of massive out-of-order streams with Apache Beam ○ Speaker: Kenneth Knowles, Google ○ Wednesday @ 3:00 pm ● Birds-of-a-feather: IoT, Streaming and Data Flow ○ Panel: Yolanda Davis, Davor Bonaci, P. Taylor Goetz, Sriharsha Chintalapani, and Joseph Nimiec ○ Thursday @ 5:00 pm
- 7. Road to the first stable release State of the project
- 8. What we accomplished so far? 02/01/2016 Enter Apache Incubator 5/16/2017 First stable release Early 2016 Design for use cases, begin refactoring Late 2016 Community growth Early 2017 API stabilization 06/14/2016 1st incubating release 01/10/2017 Graduation as a top-level project
- 9. Announcing the first stable release (5/16/17)
- 10. Expressing data-parallel pipelines with the Beam model A unified model for batch and streaming
- 11. Processing time vs. event time
- 12. The Beam Model: asking the right questions What results are calculated? Where in event time are results calculated? When in processing time are results materialized? How do refinements of results relate?
- 13. PCollection<KV<String, Integer>> scores = input .apply(Sum.integersPerKey()); The Beam Model: What is being computed?
- 14. The Beam Model: What is being computed?
- 15. PCollection<KV<String, Integer>> scores = input .apply(Window.into(FixedWindows.of(Duration.standardMinutes(2))) .apply(Sum.integersPerKey()); The Beam Model: Where in event time?
- 16. The Beam Model: Where in event time?
- 17. PCollection<KV<String, Integer>> scores = input .apply(Window.into(FixedWindows.of(Duration.standardMinutes(2)) .triggering(AtWatermark())) .apply(Sum.integersPerKey()); The Beam Model: When in processing time?
- 18. The Beam Model: When in processing time?
- 19. PCollection<KV<String, Integer>> scores = input .apply(Window.into(FixedWindows.of(Duration.standardMinutes(2)) .triggering(AtWatermark() .withEarlyFirings( AtPeriod(Duration.standardMinutes(1))) .withLateFirings(AtCount(1))) .accumulatingFiredPanes()) .apply(Sum.integersPerKey()); The Beam Model: How do refinements relate?
- 20. The Beam Model: How do refinements relate?
- 21. Customizing What Where When How 3 Streaming 4 Streaming + Accumulation 1 Classic Batch 2 Windowed Batch
- 22. The Beam vision for portability Write once, run anywhere“ ”
- 23. Beam Vision: mix and match SDKs and runtimes ● The Beam Model: the abstractions at the core of Apache Beam Runner 1 Runner 3Runner 2 ● Choice of SDK: Users write their pipelines in a language that’s familiar and integrated with their other tooling ● Choice of Runners: Users choose the right runtime for their current needs -- on-prem / cloud, open source / not, fully managed / not ● Scalability for Developers: Clean APIs allow developers to contribute modules independently The Beam Model Language A Language CLanguage B The Beam Model Language A SDK Language C SDK Language B SDK
- 24. ● Beam’s Java SDK runs on multiple runtime environments, including: ○ Apache Apex ○ Apache Spark ○ Apache Flink ○ Google Cloud Dataflow ○ [in development] Apache Gearpump ● Cross-language infrastructure is in progress. ○ Beam’s Python SDK currently runs on Google Cloud Dataflow Beam Vision: as of June 2017 Beam Model: Fn Runners Apache Spark Cloud Dataflow Beam Model: Pipeline Construction Apache Flink Java Java Python Python Apache Apex Apache Gearpump
- 25. Example Beam Runners Apache Spark ● Open-source cluster-computing framework ● Large ecosystem of APIs and tools ● Runs on premise or in the cloud Apache Flink ● Open-source distributed data processing engine ● High-throughput and low-latency stream processing ● Runs on premise or in the cloud Google Cloud Dataflow ● Fully-managed service for batch and stream data processing ● Provides dynamic auto-scaling, monitoring tools, and tight integration with Google Cloud Platform
- 26. How to think about Apache Beam?
- 27. How do you build an abstraction layer? Apache Spark Cloud Dataflow Apache Flink ???????? ????????
- 28. Beam: the intersection of runner functionality?
- 29. Beam: the union of runner functionality?
- 32. Parallel and portable pipelines in practice A Use Case
- 48. Getting Started with Apache Beam Quickstarts ● Java SDK ● Python SDK Example walkthroughs ● Word Count ● Mobile Gaming Extensive documentation
- 49. Extensibility to integrate the entire Big Data ecosystem Integrating Up, Down, and Sideways “ ”
- 50. Extensibility points ● Software Development Kits (SDKs) ● Runners ● Domain-specific extensions (DSLs) ● Libraries of transformations ● IOs ● File systems
- 51. Software Development Kits (SDKs) Runner 1 Runner 3Runner 2 The Beam Model Language A SDK Language C SDK Language B SDK
- 52. Runners Runner 1 Runner 3Runner 2 The Beam Model Language A SDK Language C SDK Language B SDK
- 53. Domain-specific extensions (DSLs) The Beam Model Language A SDK Language C SDK Language B SDK DSL 2 DSL 3DSL 1
- 54. Libraries of transformations The Beam Model Language A SDK Language C SDK Language B SDK Library 2 Library 3Library 1
- 55. IO connectors The Beam Model Language A SDK Language C SDK Language B SDK IO connector 2 IO connector 3 IO connector 1
- 56. File systems The Beam Model Language A SDK Language C SDK Language B SDK File system 2 File system 3 File system 1
- 57. Ecosystem integration ● I have an engine → write a Beam runner ● I want to extend Beam to new languages → write an SDK ● I want to adopt an SDK to a target audience → write a DSL ● I want a component can be a part of a bigger data-processing pipeline → write a library of transformations ● I have a data storage or messaging system → write an IO connector or a file system connector
- 58. Apache Beam is a glue that integrates the big data ecosystem
- 59. Learn more and get involved! Apache Beam https://beam.apache.org Join the Beam mailing lists! [email protected] [email protected] Follow @ApacheBeam on Twitter
- 60. Apache Beam is a unified programming model designed to provide efficient and portable data processing pipelines
- 61. Still coming up... ● Stateful processing of massive out-of-order streams with Apache Beam ○ Speaker: Kenneth Knowles, Google ○ Wednesday @ 3:00 pm ● Birds-of-a-feather: IoT, Streaming and Data Flow ○ Panel: Yolanda Davis, Davor Bonaci, P. Taylor Goetz, Sriharsha Chintalapani, and Joseph Nimiec ○ Thursday @ 5:00 pm