![#MDBlocal
Streaming Data on the
Shoulders of Giants
[NETCONOMY]
Hans-Peter Grahsl
https://www.surveymonkey.com/r/KXC6DB6](https://image.slidesharecdn.com/streamingdatamdblondon20190917-190927054858/85/MongoDB-local-London-2019-Streaming-Data-on-the-Shoulders-of-Giants-36-320.jpg)
MongoDB .local London 2019: Streaming Data on the Shoulders of Giants
- 1. @ #MDBlocal Hans-Peter Grahsl, NETCONOMY Streaming Data on the Shoulders of Giants hpgrahsl LONDON
- 2. Speed & Agility Among Top Tech Risks For businesses to stay relevant they must deliver value at a breakneck pace and be constantly seeking new sources of value…
- 3. #MDBLocal Managing, Processing & Analyzing Data We all use Data to unlock insights and drive value mongodb.com/local/london
- 4. #MDBLocal Diminishing Value of Data mongodb.com/local/london near real time seconds minutes hours days months VALUEofdatatodecision-making Traditional “batch” business intelligenceTime critical decisions actionable reactive historical preventive/predictive Source: Perishable Insights, Mike Gualtieri, Forrester
- 5. #MDBLocal Historic ETL can be very painful • batch-driven • brittle & error prone • (too) late answers Speed & Agility Antipattern mongodb.com/local/london
- 6. #MDBLocal Alleviate ETL Pain with Streaming Speed & Agility Enabler mongodb.com/local/london • event-centric & stream-oriented • loosely coupled fabric • quick insights & fast answers
- 7. #MDBLocal Streams Processors Connected Apps Architecture of a Modern Data Platform? mongodb.com/local/london Streaming Data Fabric Data Sources Connected Apps Data Sources
- 8. #MDBLocal On the Shoulders of Giants mongodb.com/local/london Kafka®MongoDB®
- 10. #MDBLocal Modern Database Document Model Run Anywhere Distributed & Scalable Resilient & Performant mongodb.com/local/london
- 11. Apache Kafka® Minimum Viable Introduction
- 13. #MDBLocal Streaming Platform ü distributed ü horizontally scalable ü highly fault-tolerant mongodb.com/local/london
- 14. #MDBLocal What is Streaming? “… a type of data processing that is designed with infinite data sets in mind …” – Tyler Akidau mongodb.com/local/london
- 15. "…everything that happens in a company – every customer interaction, every API request, every database change – can be represented as real-time stream that anything else can tap into, process or react to."
- 16. "…Kafka and the whole category of stream processing represents a fundamental paradigm shift in how the digital part of a company is built, how data is used, and how applications are built. This is actually a pretty rare thing…" – Jay Kreps
- 17. #MDBLocal KStreams App Data Sources Data Sinks KSQL App Streams API KSQL Consumer API Connect API App Apps App Apps Connect API Producer API mongodb.com/local/london
- 18. #MDBLocal Kafka APIs in a Nutshell… • Producer & Consumer API à publish-subscribe scenarios • Connect API à streaming data integration scenarios • Streams API & KSQL à code or SQL-based streaming scenarios mongodb.com/local/london
- 19. Kafka® Connect What’s it about?
- 20. #MDBLocal Kafka Connect Basics ANY sink Connect Connect ANY sink ANY à e.g. file systems, data stores, REST endpoints, … mongodb.com/local/london
- 21. #MDBLocal Kafka Connect Basics often about data stores Connect ConnectSOURCE SINK mongodb.com/local/london
- 22. #MDBLocal Kafka Connect Basics or more concretely Connect Connect https://hub.confluent.io à many many more mongodb.com/local/london
- 23. #MDBLocal Kafka Connect Basics or more concretely Connect Connect mongodb.com/local/london https://hub.confluent.io à many many more
- 24. Kafka® Connect How do connectors operate?
- 25. #MDBLocal Kafka Source Connectors Source Connector Converter Serialize S M T 1 … N Single Message Transforms for basic in-flight manipulations … S M T mongodb.com/local/london
- 26. #MDBLocal Kafka Sink Connectors Converter Deserialize Sink Connector S M T 1 … N Single Message Transforms for basic in-flight manipulations … S M T mongodb.com/local/london
- 27. Official Connector Announced at #MDBW19
- 28. #MDBLocal MongoDB Connector for Apache Kafka mongodb.com/local/london Map and persist events from Kafka topics directly to MongoDB Publish data changes from MongoDB into Kafka topics
- 29. #MDBLocal MongoDB Connector for Apache Kafka ü officially supported by MongoDB ü Verified Gold certified by Confluent mongodb.com/local/london Available for Download on the Confluent Hub https://www.conflent.io/hub/mongodb/kafka-connect-mongodb
- 30. Use Cases MongoDB Connector for Apache Kafka
- 31. #MDBLocal Single Customer View for eCommerce MongoDB Sinks Single Source of Truth Source Connectors
- 32. #MDBLocal Data Synchronization between Microservices Service 1 Service N MongoDB Sinks . . .
- 33. #MDBLocal Recommendation Engine for Opinion Mining Surveys & Polls Data MongoDB Source Change Streams Change Streams User Recommendation Engine
- 34. Demo Scenario Let’s see it in action!
- 35. #MDBLocal Producer APIdata generation Stream Processor MongoDB Sink Connector MongoDB Source Connector data serving REST Change Streams device management SSE Demo Scenario ! mongodb.com/local/london
- 36. #MDBlocal Streaming Data on the Shoulders of Giants [NETCONOMY] Hans-Peter Grahsl https://www.surveymonkey.com/r/KXC6DB6
- 37. THANK YOU!