PubSub+ Event Portal for Apache Kafka
Download as PPTX, PDF0 likes455 views
The document discusses the importance of managing Kafka event streams effectively, highlighting challenges organizations face without a centralized solution for discovery, sharing, and governance. It proposes the concept of an 'event portal' that would allow users to design, catalog, visualize, and manage event streams, offering benefits for collaboration and compliance. The event portal aims to streamline interactions and provide high-level productivity tools, similar to those used in API management, for event-driven applications.
PubSub+ Event Portal for Apache Kafka
- 1. How to Discover, Visualize, Catalog, Share and Reuse your Kafka Event Streams Jonathan Schabowsky Senior Architect, CTO, Solace
- 2. Agenda 2 ⹠Your growing Kafka event streams ⹠Flying blind ⹠Event portal: an API portal for event streams ⹠Demo: How to discover, visualize, catalog, share and reuse your Kafka event streams ⹠Summary © Solace 2
- 3. Does Your Organization Look Like this? 3 © Solace Data Stream Flow Legacy Application Infrastructure âŠ. ? ? ? JSON, XML, Plain Text⊠IoT/Device Management Producers Consumers Connectors Stream Processors Avro
- 4. Youâre Flying Blind⊠Where do you discover event streams for reuse? Why does a given event stream exist? What topic do you use to subscribe to it? What changed? How do you determine the data structure of its payload? Who should have access to it? Who made the latest revision? Who can tell you more about an event? Is your change backward compatible? Who is impacted by your change? Does it comply with security policy? 4 © Solace
- 5. Some Try Solutions Not Intended for Event Stream Management âą It's a pain⊠â No single place to discover and share/reuse event streams â No best practices guidance âą Youâre breaking systems and donât know who changed what??? â Suffer from changes with unintended consequences â Hard to audit changes when data is stored in different tools Brittle, complex, manual solutions Custom Developed Catalog PowerPoint API Management Spread sheets GitConfluence 5 © Solace
- 6. API Management Solved this for REST 6 © Solace
- 7. They answer Who, What, When, Where, Why and How For RESTful APIs⊠but what about event streams? Inspired by API Management Platforms 7 © Solace API Gateway Documentation ⹠Discover Registration ⹠Govern Analysis ⹠Improve ⹠Monetize Community ⹠Collaborate Architects and Developers (Internal, Partner and Third-Party) Productivity ⹠Generate Code API Portal
- 8. Sinks âWhile most organizations have basic event processing infrastructure - such as notification services, message buses, and event brokers - many donât have the high-level productivity tooling that helps developers design, develop, test and manage event-centric applications. Nor do they have tools for governing, publishing and managing event-based interfaces. The market for these tools is sparse.â Gartner âTop 3 Trends in Application Architecture That Enable Digital Businessâ Anne Thomas, Yefim Natis, Mark OâNeill, 28 Oct 2019 8 © Solace Event Broker Security Persistence Protocol Translation One-to-Many Distribution Source Runtime We Need an Event Portal for Asynchronous Interactions Design, Develop and Manage Event Portal Documentation âą Discover Registration âą Govern Analysis âą Improve âą Monetize Community âą Collaborate Productivity âą Generate Code (code gen) (spec)
- 9. 9 © Solace Single place to design, create, catalog, visualize, discover, share, secure and manage all events within your ecosystem Introducing Event Portal Event Portal Mainframe/ SAP Distributed Apps IoTMicro services Server Less Event Brokers
- 10. Event Portal How Users Interact with Event Portal 10 Event Management (Design Time) © Solace Your Event-Driven Enterprise B2B/B2C Ecosystem IoTMicro- services Server- less Generate Code REST API Existing Tools ⹠Schema registries ⹠Data governance ⹠CI/CD pipelines Customers & Partners ⹠Discover and consume events in real-time Architects ⹠Discover, create, update, review apps, events and payload schemas ⹠Map payload schemas to events; and events to apps ⹠Govern event consumption across app domain boundaries Developers ⹠Collaborate with architects on design ⹠Export AsyncAPI app contracts for each app ⹠Generate and implement event-driven apps that provide business value Data Scientists ⹠Understand event-driven data ⹠Discover new insights by combining events
- 12. Benefits of an Event Portal 12 Solace Your business can⊠⹠Understand who is consuming which events, how much, for what ⹠Create value-added services and gain insights by combining events ⹠Make it easy for users across app teams, LoBs and partner ecosystem to find and reuse events ⹠Monetize popular or particular high-value event streams Reuse/Sharing Architects & developers can: ⹠Collaborate on events and event-driven apps ⹠Visually see event-driven interactions to understand impact of changes ⹠Low Code: Generate app code by exporting AsyncAPI definitions ⹠Benefit from best practices & consistently apply conventions Productivity CDO/data governance can: ⹠Understand where data is coming from (lineage) and going ⹠Track changes and audit for deviations while promoting apps and dependencies though Dev, Test, QA, and Prod ⹠Use application/event relationships to create security policies and validate compliance with schema Control
- 13. So, if you⊠Have a lot of event- driven applications 13 © Solace You need an event portal! Have (or need to define) a lot of event streams Want to manage events as easily as you manage your APIs
- 14. Event Portal = API Portal for Events/Streams One tool to design, create, catalog, visualize, discover, share, secure and manage all of your events and streams (including Kafka and any other event broker) 14 © Solace Sign up for free! ⹠About data and apps, NOT managing Kafka infrastructure ⹠High-level productivity tooling to help you fully embrace EDA ⹠Governance of apps, event and data