Time to Talk about Data Mesh
- 1. It is time to talk about DATA MESH Distributed Data Management for Microservices
- 3. 3 THE CHALLENGES OF MICROSERVICE DESIGN TRANSFORM BUSINESS LOGIC AND DATA INTO MICROSERVICE ARCHITECTURE Business flow combing, Function and service separating Data decoupling and fragmentation management ◼ DDD (Domain-Driven Design) ◼ Split services by business flow ◼ EDA (Event-driven Architecture) ◼ Microservice component design ◼ Microservice horizontal scaling strategies ◼ Separating data by business flow ◼ Sharing data across domains ◼ Vast data deployment and caching strategies ◼ Data consistency(Data fragmentation management) ◼ The final data consistent ◼ Strong data consistent ◼ Absolute data consistent CHALLENGE DONE WELL Business Logic Data CELL GATEWAY Database Service Service Analyze Decouple Aggregate To implement the microservice architecture with decision of design patterns, deployment and maintenance strategies, integration of technical components, planning and actual software architectures. Implement API SERVICES/ EDGE SERVICES COMPOSITE SERVICES/ INTEGRATION SERVICES CORE SERVICES/ ATOMIC SERVICES CLIENT APPLICATIONS To implement the design and choose an immediately available solution to implement various microservice design patterns and basic architectures. Based on stable software components and platforms, determine service roles and implement business logic. Special design with stateful atomic services
- 4. 4 TRANSFORM TO DATABASE PER SERVICE For Microservice Architecture App service App service App service DATABASE App service App service App service Database per service Shared Database How? Risk? Performance? Plan?
- 5. 5 PROBLEMS WITH DATABASE PER SERVICE WHEN WE IMPLEMENT IT App service App service App service Round-trips Problem Busy App service App service App service Anti-Pattern App service App service App service Confusion OR OR How do we solve these problems ?
- 6. 6 PROVISION ONCE, SNAPSHOT MANY CACHEING FOR MULTIPLE SNAPSHOTS CLIENT APPLICATIONS Service API SNAPSHOT Service API DATA FLOW Source VIRTIAL TABLE VIRTUAL TABLE BUSINESS BUSINESS 100M Records 100M Records 100M Records Service Owner According to requirements of business task, selecting database, table and fields to make a copy into a virtual table. Update Database per service 1 Provision 2 3 Program A Program B
- 7. 7 The scale of the supply pipelines and the degree of deployment efficiency THE KEY TO THE SMOOTH IMPLEMENTATION OF MICROSERVICE ARCHITECTURE
- 8. 8 MANAGE DATA SUPPLY CHAIN WITH DATA PLATFORM FAST TO BUILD DATA SUPPLY CHAIN FOR APPLICATIONS CLIENT APPLICATIONS Service API SNAPSHOT Service API DATA PLATFORM Source VIRTIAL TABLE VIRTUAL TABLE BUSINESS BUSINESS DATA PROVISION AND CACHING 100M Records 100M Records 100M Records Service Owner According to requirements of business task, selecting database, table and fields to make a copy into a virtual table. Update Database per service 1 Provision 2 4 Caching 3
- 9. 9 App App APPLICATION-DRIVEN DATA SUPPLY DISTRIBUTED DATA INTEGRATION VIRTUAL TABLE App App App App DATA MESH VIRTUAL TABLE VIRTUAL TABLE VIRTUAL TABLE VIRTUAL TABLE VIRTUAL TABLE DATABASE DATA SOURCE CUSTOMIZATION DATA SOURCE FILES DATA SOURCE
- 10. 10 On-Premise APPLICATIONS WHY NOT THE TRADITIONAL SOLUTIONS? CENTRALIZED DATA MANAGEMENT CANNOT FIT TO MICROSERVICE ARCHITECTURE Public FILES DATABASE CUSTOMIZATION DATA SOURCE DATA SOURCE DATA SOURCE Cloud Private Cloud Cloud VIRTUAL TABLE Centralized Data Process VIRTUAL TABLE VIRTUAL TABLE Traditional ETL, Data Virtualization and Data Warehouse encounter problems with centralized system design. ETL Batch Polling
- 11. 11 On-Premise APPLICATIONS DATA MESH CONCEPT DISTRIBUTED DATA MANAGEMENT WITH DATA MESH Public FILES DATABASE DATA SOURCE CUSTOMIZATION DATA SOURCE DATA SOURCE Cloud Private Cloud Cloud VIRTUAL TABLE VIRTUAL TABLE VIRTUAL TABLE DATA MESH Data Mesh focusing on supplying data to applications as fast as possible with distributed system design. Provision Provision Provision
- 12. HOW TO BUILD DATA MESH INFRASTRUCTURE 02
- 13. 13 Data Fabric HOW TO BUILD DATA MESH INFRASTRUCTURE Data Virtualization and Relay Data Caching Presentation Deployment and Management Data Source Data Mesh Data Provision Infrastructure Services Business ◼ Provision and manage data with pipelines ◼ Establish data relay mechanism（Caching） ◼ Rapidly deployment and management data pipeline clusters（Container） ◼ Manage vast multiplexed data channels and pipelines (Multiplexer) Objective Applications Legacy and others Service BUSINESS Service BUSINESS Service BUSINESS Service BUSINESS Service BUSINESS Service BUSINESS Service BUSINESS Service BUSINESS Last Mile Infra needs to be able to quickly deploy all data management mechanisms based on application requirements.
- 14. 14 BUILD DATA NODE BASED ON KUBERNETES YAML WAY TO DEPLOY DATA NODE AND PIPELINE INFRSTRUCTURE Data Mesh Controller YAML Relay Database Relay Database Relay Database YAML YAML Service BUSINESS Service BUSINESS Service BUSINESS Service BUSINESS Service BUSINESS Service BUSINESS Container Platform Orchestrator Kubernetes ◼ Declarative pipeline deployment with integrated container platform ◼ Meet the demand for a large number of microservice data supply ◼ Manage and maintain a large number of provision pipelines Objective DATA PLATFORM Hybrid Clouds
- 15. 15 Using YAML SCENARIO: Starting from event data provision, caching, building virtual table, and quickly generate APIs. No need to write a single line of program, build the entire framework in a few minutes, from Oracle to MongoDB and generate usable data API SERVICE CDC Protocol Data Platform Kubernetes cluster VM / Container Presenter component Event changed Intercept database change events The data source can be an application or a database Select the required data fields Other applications Customized Query from cached data Deploy a pipeline mechanisms Caching API Auto generated Imported from templates 1 Apply API settings 2 3 CLIENT APPLICATIONS DATA SOURCE INFRA AND PLATFORM DATA PROVISION AND CACHING 11g R2
- 16. 16 DATABASE RESULTS Legacy Program USE CASE 1: Improve the efficiency of data interface to the legacy system SOLVE PROBLEM THAT LEGACY SYSTEM BRINGS ON LOW LATENCY AND INTEGRATION ISSUES DATABASE DATA SOURCE Query Storage Program DATABASE RESULTS Polling Export Insert File DATABASE DATA SOURCE DATA MESH Event Update Real-time and Low Latency Provision and Caching
- 17. 17 B Batch DATABASE RESULTS USE CASE 2: Solve the inefficiency and performance impact of various batch processing SOLVE PROBLEM THAT INTEGRATION ISSUES DATABASE DATA SOURCE Query A Program DATABASE RESULTS Insert DATABASE DATA SOURCE DATA MESH NODE Provision Supply to Multiple Applications without Performance Impact A Service DATA MESH PIPELINE Push DATA MESH PIPELINE B Service Push Parallel Caching A Batch Query and Extract B Program A B Subscribe Subscribe Query and Extract Insert
- 18. 18 Program SNAPSHOT USE CASE 3: High-speed parallel processing of the aggregation and correlation of multiple data sources USE CASE: INTEGRATION WITH MULTIPLE SOURCES IT DATA SOURCES OT DATA SOURCES Program Output Aggregate and Transform Round-Trip Problem Round-Trip Problem IT DATA SOURCES OT DATA SOURCES DATA MESH NODE Event DATA MESH NODE Event DATA MESH PIPELINE Present Aggregate Snapshot Snapshot Output Transform Aggregation in Parallel for High Performance Parallel Caching Caching
- 19. 19 Program SNAPSHOT USE CASE 4: Integration of hybrid cloud, cross-site and cross- environment USE CASE: INTEGRATION FOR HYBRID CLOUD IT DATA SOURCES OT DATA SOURCES Program Output Aggregate and Transform Round-Trip Problem Round-Trip Problem IT DATA SOURCES OT DATA SOURCES DATA MESH NODE Event DATA MESH NODE Event DATA MESH NODE Present Aggregate Snapshot Snapshot Output Transform Aggregation Across Clouds With Parallel Caching Caching On-Premise Public Cloud External Site On-Premise Public Cloud External Site
- 20. ＤＥＭＯ 0３
- 21. About Brobridge Brobridge refers to Broad Bridge Co., Ltd. (a limited company formed under the laws of Taiwan). Please refer to www.brobridge.com for a detailed description. Brobridge Co., Ltd. provides information technology consulting, application development, IT infrastructure design, consulting services and system maintenance services to listed and unlisted clients in various industries. Brobridge Co., Ltd. recruits the most professionals in the industry, is committed to the pursuit of excellence and setting an example. Brobridge Co., Ltd. Brobridge refers to Brobridge Co., Ltd.. Brobridge Co., Ltd. enjoys a good reputation in the industry for its excellent customer service, excellent talents, perfect training and rigorous development technology. Through the resources of Brobridge Co., Ltd., we provide a full range of services for customer information systems, including virtual desktop systems, multi-cloud integration, IT infrastructure planning and design, and advanced cutting-edge program development. This publication is compiled based on general information and is for readers' reference only. Brobridge Co., Ltd. is not deemed to provide professional advice or services to anyone because of this publication. Any individual of Brobridge Co., Ltd. shall not be held responsible for any loss caused by reliance on this publication. © 2016 Broad Bridge Co., Ltd. All rights reserved. 寬橋 THANK YOU!