Spring Boot+Kafka: the New Enterprise Platform
Download as PPTX, PDF2 likes1,587 views
This document discusses how Spring Boot and Kafka can form the basis of a new enterprise application platform focused on continuous delivery, event-driven architectures, and streaming data. It provides examples of companies that have successfully adopted this approach, such as Netflix transitioning to Spring Boot and a banking brand building a new core banking system using Spring Streams and Kafka. The document advocates an "event-first" and microservices-oriented mindset enabled by a streaming data platform and suggests that Spring Boot, Kafka, and related technologies provide a turnkey solution for implementing this new application development approach at large enterprises.
Spring Boot+Kafka: the New Enterprise Platform
- 1. © Copyright 2019 Pivotal Software, Inc. All rights Reserved. James Watters April 2, 2019 Spring Boot+Kafka: the New Enterprise Platform
- 2. Our Journey: Bring continuous delivery with platform efficiency to enterprises.
- 3. New Projects Waterfall ITIL VMs JavaEE Batch Products Lean CI/CD Cloud Native Platforms Spring Boot Kafka Streaming Platform Old
- 4. “For tech to be a real driver of innovation and growth, IT needs to reorganize itself around flexible platforms.” —McKinsey
- 6. “We are happy to announce that starting in 2018, Netflix is also making the transition to Spring Boot as our core Java framework, leveraging the community’s contributions via Spring Cloud Netflix.”
- 7. Platform at T-Mobile SPEED ➔Time to Prod: 6 Months to weeks ➔# of deployments: 10x increase SCALABILITY ➔Avg: 300M Transactions a Day ➔Peak: 1Billion Transactions a Day STABILITY ➔43% reduction in app response time ➔83% fewer incidents, fixed 67% faster
- 9. Full transformation required a fundamental rethink….
- 10. “A business is a series of events and the reactions to those events.” —Jay Kreps, CEO, Confluent
- 11. “The only reason we don’t think of events….is that so far the technology has trained us to think of data as a static store.” —Neha Narkhede
- 12. We restarted with events first
- 13. What we’ve learned ➔ Events are the language bridge to the business ➔ This method of identifying bounded contexts is a secret to decoupled architecture ➔ “Tell don’t ask”
- 14. Completing the Journey IBM App Cache App Win? Microservices Event-driven Microservices AppAppApp AppAppApp AppAppApp
- 15. A platform built for a new way of thinking ➔Event + Microservice first ➔Team autonomy with platform efficiency ➔Arbitrary scaling/scope across enterprise ➔Turnkey multi-cloud
- 18. ➔ Global banking brand building greenfield core banking and payments with Spring Streams + Kafka ➔ Focus on microservices velocity required platform first thinking ➔ Kleppman like view of Kafka: “We count on Kafka for consistency, strict ordering, replay, durability and auditability.” ➔ Cross cloud replication A new continuously delivered banking platform
- 19. ➔ Turning moving packages into streaming data with RFID, Kafka and Spring Streams event based microservices ➔ Kafka, Kubernetes and Spring Boot in every shipping center ➔ Multiple business microservices teams can layer onto streaming platform to bin pack last mile services. ➔ Prepared for unanticipated uses cases Revolutionize our shipping efficiency with streaming microservices
- 20. PKS Managed Clusters Messaging Middleware Kafka Binder Spring Data Repository Event Driven Microservices LTL Quote Service Scan RFIC Services RFID Triggered Automation Services
- 21. ➔ 100,00+ container build out of Spring Streams, Kafka, key-value store ➔ Durability and consistency are critical for potential legal actions ➔ Multi-phase stream processing with Spring Streams leading to real-time microservices alerting analysts ➔ Cross-cloud replication based on Kafka ➔ Continuously delivery required for real time apps to improve accuracy and functionality as project expands Help secure a European country?
- 22. Receiver App process queue Fault tolerant receiver pairs staging and replication Apps Stream Workers Data Enrichment Stream Workers Data Enrichment Stream Workers Data Enrichment process queue Stream Workers Data Classification Stream Workers Data Classification Stream Workers Data Classification buffer queue S3 RAW Store Receiver App X.000 Channel Streams RDBMS Store 3 DC KAFKA Replication NoSQL Store Index Store
- 23. ➔ Mainframe and monolithic RDBMS data teams often the last to move to continuous delivery ➔ CDC, Event Shunting, patterns emerging allow streaming data platform teams to offer mainframe and legacy RDBMS events to microservices teams ➔ “AirBnB Pattern” growing across enterprises ➔ Each team can build appropriate persistence and achieve multi-DC replication with streaming platform Let’s empower pharmacy microservices developers while evolving our legacy?
- 24. PKS Foundation 1 PAS Foundation 1 Event Driven Microservice (Producers) Event Driven Microservice (Consumers) Domain Driven Microservice Legacy ODS ODS Kafka Cluster 1 Broker MirrorMaker cluster 1: topics cluster 1: topics PKS Foundation 2 PAS Foundation 2 Event Driven Microservice (Producers) Event Driven Microservice (Consumers) Domain Driven Microservice ODS Kafka Cluster 2 Broker MirrorMaker cluster 1: topics cluster 1: topics
- 25. Spring Kafka ++ We are all in… doubling down on a platform
- 26. Spring Cloud Stream deals with the Kafka scaffolding, so you don’t have to The power of Kafka streams for developers ➔ Rapid on-ramp for Kafka Streams consumption ➔ Simplifies construction of Event- Driven Stateful Microservices ➔ Focus on your processing logic not on configuration ➔ Full support of all Kafka streams functionality
- 27. We are all in… doubling down on a platform Spring Kafka ++ PCC Kafka Integration Pivotal Function Service (Knative ++ ) PAS Service Brokers (Custom today) PKS Confluent Operator Testing and Integration AppTX + Kafka CDC ++
- 28. Call to action ➔ Tim Berglund and Josh Long talk on Spring and Kafka ➔ Contribute to GitHub features and issues for Spring Stream ➔ Come visit us at the booth to talk streaming microservices
Editor's Notes
- #3: Excited to be here, Kafka community is one of my favorite places to study the evolution of enterprise organization and architecture. Ended up here in part because through customer experience came to the personal belief that Spring Boot and Kafka are the two must have components of a flexible yet efficient enterprise platform Spoiler alert: my experience leads me to believe every enterprise needs a focus on broad potentials of Sring Boot + Kafka
- #4: New ways of thinking, organizing, and architecture all at once; generational perfect storm is hitting enterprises today. -
- #5: The business is aware that a broad change like this will require a platform enablement to achieve at scale; we must partner with the business to communicate at all times how these org/architecture changes can unleash them.
- #7: Yes, Spring Boot has 60M downloads but it’s one of a few technologies used at scale in enterprise and cutting edge cloud native companies like Netflix. Netflix was one of the early champions of radically changing org/architecture and using continuous delivery to outpace a whole industry. Netflix counts on Spring to abstract distributed infra scaffolding for their teams….even though they wrote most of it
- #8: Modern software infrastructure platforms must do three things: Radical gains in continuous delivery, team autonomy Economies of scope across most teams Dramatic efficiency gains which enable a small platform team to deliver for for thousands
- #9: “Mosty enterprises are taking step one” The first one should say “IBM” for App to show monolith start if possible. Got it. Want to do a fresh view of each of these with maybe a spring boot icon for each app. Want to show that step one was to break apart the app tier; also likely adding a cache above the DB for the microservice. Happy to discuss this is one of the critical graphics to show the progression of the story. Winning/helping an enterprise is always evolutionary and very rarely binary to “winning the logo”.
- #10: Outcomes, outcomes outcomes---driven by customer demand to double down on the success they have had. “I want this operating model for EVERYTHING”
- #11: The Kafka community was working on this problem from a different but vital perspective. Convergent evolution ….
- #12: Paul Graham on programing languages…..
- #13: Stubhub Event Storming; even a relatively recent born on the web company but pre microservices and CI/CD “Tell don’t ask”
- #15: -Without “tell don’t ask” incidental coupling across microservices can easily creep back in... -we see success with a blend of pane 2-3….
- #16: Cool Thanks!
- #17: --The most important lines are the human to human connections here. --Platform + Slack --The platform minimizes toil --Make it FUN --Common tools keep communication and experience meaningful across teams
- #19: Martin Kleppman like API interfaces where the API/web application immediately writes to Kafka with everything else as a kafka consumer. Conversation about microservices enablement with platforms; two big bets, Kafka and Spring Boot. “Be prepared for unanticipated use cases in the future (for free)”
- #20: “Be prepared for unanticipated use cases in the future (for free) Fed Ex edge streaming ingest project (unnamed) In the Freight business, Less-Than-Load (LTL) and Last Mile are the two most critical and hard problems to solve and FedEx Freight in one of several companies throwing their hat in the ring to provide this service (eg: Customers buying appliances and furniture online and companies that will deliver this and assemble if needed) Initial use case is to provide a “centrally managed” platform across 300+ FedEx Freight Service Centers. The software stack running in each service center includes vSphere on VxRail, BOSH, PKS, Kafka, K/V Store TBD (Cassandra preferred and tested), Spring Boot [more detail here on scale] Data generated by RFID scanners which are placed in the ceiling of service centers and scan RFID codes in their cone of vision. These scanners talk to Spring Boot apps which send the data to Kafka queues Data produced locally is stored for approx. 36 hours and 10 days for some other data Data capacity is approx. 1TB per day Data is used to make critical decisions like which pallets to route where using which trucks. Kafka is the “glue” getting data from and pushing data to Spring Boot apps, while also pushing data into Cassandra for search and other processing (analytics)
- #21: Spring Boot + Kafka for, price offer generation, data ingest, and real time automation applications!
- #23: European crime agency. Developed for communication surveillance to support real time surveillance as well mandated analytics work. This solution should never lose data or parts of streams, this could invalidate the consistency and detectability Thousands of streams needs to be buffered, classified, enriched and stored to be replicated to create a data loss free solution Using microservices gives the ability to plug in new modules for encoding and enrichment in real time Kafka is connecting the processes together giving the ability to asynchronous handle thousands over thousands messages delivered as streams Most of the microservices are based on spring boot using the spring kafka consumer and producer
- #24: EKafka CDC Use Cases can apply toenterprises Cache Invalidation Search Indexing Offline Processing Signaling (triggers)
- #27: Autoconfiguration, monitoring, metrics These slides have attempted to clarify how Spring's projects wrap and compliment Kafka, and to demonstrate the value that this brings to developers in terms of raising the value line.
- #28: This would be a build that starts with PCC.
- #30: Spring Cloud Stream Loves Developers & Developers Love Spring Cloud DataFlow I've tried to demonstrate how Spring's projects wrap and compliment Kafka, and to demonstrate the value that this brings to developers in terms of raising the value line.
- #33: European crime agency. Developed for communication surveillance to support real time surveillance as well mandated analytics work. This solution should never lose data or parts of streams, this could invalidate the consistency and detectability Thousands of streams needs to be buffered, classified, enriched and stored to be replicated to create a data loss free solution Using microservices gives the ability to plug in new modules for encoding and enrichment in real time Kafka is connecting the processes together giving the ability to asynchronous handle thousands over thousands messages delivered as streams Most of the microservices are based on spring boot using the spring kafka consumer and producer