Using Kafka: Anatomy of the Flowable event registry
1 like930 views
The document discusses the implementation of event-driven architectures using Flowable, highlighting its flexibility and ease of use in various environments. It describes a case study involving a customer review process that triggers sentiment analysis through AWS Comprehend, emphasizing the integration of Kafka and AWS services. Key features include a central event registry, pipeline management, and the potential for serverless deployment, showcasing Flowable's adaptability for modern applications.
Using Kafka: Anatomy of the Flowable event registry
- 1. Using Kafka: the Flowable Event Registry Joram Barrez (@jbarrez) Filip Hrisafov (@filiphr)
- 2. ©2019FlowableAG. All rights reserved. Event-Driven Architectures • In the Flowable community, we have seen • People experimenting with event-driven/event-sourcing architectures • (or more often architectures enriched with events) • Demand for out-of-the-box support 2
- 3. ©2019FlowableAG. All rights reserved. Event-Driven Architectures • A process/case fits naturally • Can be the ‘Command’ of an event (e.g. the C in CQRS) • Have state to react on the change expressed in the event (state is often the hard part of event-sourcing) 3
- 4. ©2019FlowableAG. All rights reserved. Quick Demo 4
- 5. ©2019FlowableAG. All rights reserved. Implementation – The event registry 5 Flowable Engine EventRegistry EventBus EventDefinitionEventDefinitionEventDefinition Consumers Central repository of event metadata Dispatches events to consumers (default process/case behaviors are also a consumer) Description of event structure: - Payload - Correlation
- 6. ©2019FlowableAG. All rights reserved. Implementation – Channels and Pipelines 6 Flowable Engine EventRegistry EventBus Channel Adapter Event Consumer Inbound Channel Inbound Pipeline Deserialization Event key detection * All shown here on the slide are pluggable Payload extraction Event transformation
- 7. ©2019FlowableAG. All rights reserved. Implementation – Channels and Pipelines 7 Flowable Engine EventRegistry EventBus Channel Adapter Event Producer Outbound Channel Outbound Pipeline Serialization Event transformation
- 9. ©2019FlowableAG. All rights reserved. • Fictional electrical kick- scooter startup • Customer satisfaction is priority #1 Use Case 9
- 10. ©2019FlowableAG. All rights reserved. • Customer can provide a review • Review à Kafka • A customer case • Groups all reviews • Kicks of a sentiment analysis process • Analysis is done using a process that calls out to AWS Comprehend • Bad review à A task for an employee is created to try to fix the relationship Use Case 10
- 11. ©2019FlowableAG. All rights reserved. Architecture 11 * (5th service (Spring Cloud API gateway) not shown to simplify things) UI for customers Review Service Customer Relationship Service Flowable Task Sentiment Analysis service UI for employees AWS Comprehend Review Event Review Event Analysis Result Analysis Result Analysis Request Analysis Request
- 12. ©2019FlowableAG. All rights reserved. Demo: The models explained 12
- 13. ©2019FlowableAG. All rights reserved. Demo • Simulating reviews • Stepping through a bad user review 13
- 14. ©2019FlowableAG. All rights reserved. Demo: Review Service • Simple Spring Webflux REST API • Reactive pushing of events from UI into Kafka 14
- 15. ©2019FlowableAG. All rights reserved. Demo: Customer Case Service • Spring Boot application • Flowable Spring Boot starters • Autodeploy cases and process models • Register event channels and definitions • Postgres DB 15
- 16. ©2019FlowableAG. All rights reserved. Demo: Customer Case Service • The sentiment process doesn’t use the ‘old way’ • Send-and-wait-for-response pattern • Race condition problem 16
- 17. ©2019FlowableAG. All rights reserved. Demo: Customer Case Service • The sentiment process uses a triggerable service task 17 • During execution (tx1) • Create event subscription • Create async job data (to send event) • Post-commit (tx2) • Trigger job executor (or other node) • Send event asynchronously • Go into wait state • Event receival (tx3) • Trigger for continuation
- 18. ©2019FlowableAG. All rights reserved. Demo: Customer Case Service • Can this also be made reactive? • Yes! • Nicely scaling of message stream handling depending on load • (insertion of async Flowable job at event receival using R2DBC) 18
- 19. ©2019FlowableAG. All rights reserved. Demo: The Sentiment Analysis Service • Spring Boot application • Flowable Spring Boot starters • Autodeploy process model • Uses AWS SDK to call out to AWS Comprehend 19
- 20. ©2019FlowableAG. All rights reserved. Demo: Flowable Task • Bad review à employee form 20
- 21. ©2019FlowableAG. All rights reserved. Demo: The Sentiment Analysis Service • This service is effectively a function • Can be run serverless • With all the benefits of using Flowable • Visual models, audit/history, etc. 21
- 22. ©2019FlowableAG. All rights reserved. Serverless Core Concepts • Ephemeral (short living) • Only booted up when needed • Cold starts • Auto scaling • Heavy load à boot up more instances • No Server management • Logic is deployed to a ‘serverless provider’ • Don’t care, up to the cloud vendor • Pay as you use 22
- 23. ©2019FlowableAG. All rights reserved. What is serverless? • A better name would be ‘on-demand/event server’ • (it’s still running on a server) 23
- 24. ©2019FlowableAG. All rights reserved. Flowable Serverless (Experimental) • Challenges • Cold start • State • Solutions • In memory (no Database) • No wait states (straight-through processes) 24
- 25. ©2019FlowableAG. All rights reserved. What is serverless? • Example: Amazon AWS Lambda • AWS Lamba in/out can be chained to the whole AWS ecosystem: S3/SQS/Alexa/DynamoDB/… 25
- 26. ©2019FlowableAG. All rights reserved. Architecture 26 * (5th service (Spring Cloud API gateway) not shown to simplify things) UI for customers Review Service Customer Relationship Service Flowable Task Sentiment Analysis service UI for employees AWS Comprehend Review Event Review Event Analysis Result Analysis Result Analysis Request Analysis Request
- 27. ©2019FlowableAG. All rights reserved. AWS Architecture 27 AWS Cloud API Gateway AWS SQS Review Event Frontend (S3) Review Event Analysis Request Analysis Result Analysis Request Analysis Result Customer Review Application (EC2) Sentiment Analysis Browser Comprehe nd
- 28. ©2019FlowableAG. All rights reserved. Review Service Differences • Doesn’t exist J • API Gateway pushes events directly to Amazon SQS 28
- 29. ©2019FlowableAG. All rights reserved. Customer Case Service Differences • Listens from Amazon SQS instead of Kafka • The channel/adapter concept makes this very flexible • Without impact on using the events in the model 29
- 30. ©2019FlowableAG. All rights reserved. Sentiment Analysis Service Differences • Flowable with in memory DB (no SQL) • Triggers via Amazon SQS as a Lambda • Timing (with 256MB Lambda) • Cold start billed 30s / Engine start 6s / Deploy 1.2s • Preheat start: billed 1s / Engine start 60ms / Deploy 80ms • Sends analysis results synchronously to Amazon SQS 30
- 31. ©2019FlowableAG. All rights reserved. Serverless: No Free Lunch • Vendor lock-in • Security? • Debugging? 31
- 32. ©2019FlowableAG. All rights reserved. Future Ideas • Aurora serverless • Passing state around with event payload 32
- 33. ©2019FlowableAG. All rights reserved. Conclusions • We’ve shown that • using Flowable in event-driven architectures is easy, yet very powerful • Flowable is extremely flexible when it comes to • switching queue/event stream choice • deploying your Flowable applications in different environments • running a process as a serverless function 33
- 34. Thank you.