Data Services and the Modern Data Ecosystem
1 like116 views
The document discusses a webinar series on data virtualization, addressing key challenges in data integration, modern data ecosystems, APIs, and the significance of digital transformation for organizations. It outlines the importance of data services in providing agile access to information, supporting various consumption patterns, and enhancing governance. Key takeaways emphasize how data virtualization improves time-to-market and data asset utilization through decoupling access and storage, particularly in real-time scenarios.
Data Services and the Modern Data Ecosystem
- 1. DATA VIRTUALIZATION Packed Lunch Webinar Series Sessions Covering Key Data Integration Challenges Solved with Data Virtualization
- 2. Data Services and the Modern Data Ecosystem Paul Fearon Sr. Solutions Consultant
- 3. Agenda 1. Data Ecosystems 2. APIs 3. Patterns & Practices 4. Use Cases 5. Capabilities 6. Q&A 7. Next Steps
- 6. The Web (Data Ecosystem) 6 Instant access to everything without needing to store anything
- 7. The Traditional Data Ecosystem 7 Delayed access to somethings while needing to store everything
- 8. The Modern Data Ecosystem Deployment On-premise Cloud Processing Batch Real-time Access Point-to-point Decoupled Data Models Rigid Flexible Orchestration Manual Automated 8
- 9. 9 Digital Transformation ▪ Digital transformation is a strategic initiative for most organizations ▪ The concept reflects technology’s role in strategic decision-making, with its ability to automate and simplify business processes, improve customer relationships, enhance productivity, and cost savings ▪ Driven from CEO’s office: Highest level of visibility & fully funded ▪ Gartner – 28% of CIO budget in 2018 ▪ IDC – 2/3 of CEOs in global 2000 have digital transformation in the center of their corporate strategy ▪ Seen as do-or-die initiative ▪ “If you don’t, someone else will”
- 10. 10 The Rise of Digital IT And the challenges for traditional IT ▪ Often called shadow IT, LOB’s have more sophisticated tools and needs. ▪ Digitization of society provides more channels or systems of engagements managed/owned or interacted with by LOB’s. ▪ Operational systems are automated and access to information is vital to help manage operations (supply chain, order fulfillment, etc.) ▪ LOB’s need agile access to information to analyze data, experiment and fail fast where necessary. ▪ Increased demand for different types of data including image, video, audio, etc.
- 11. The Relevance of APIs • The internet has created an interconnected world • Similarly, different processes and applications within a company also need to communicate with each other • Web services are the building blocks of this interconnected world • The concept of “an application exposing functionality” has evolved into “the web service is the application” (microservices) 11 Definitions API - an interface Web Service - a remote API via the web Data Service - a web service for data Microservice - an architectural style
- 12. Modern Data Ecosystem from an App Perspective 12 https://www.omg.org/cloud/deliverables/CSCC-Cloud-Customer-Architecture-for-Hybrid-Integration.pdf
- 13. 13 Why are data services important 13 • Abstraction • Consumer need not concern themselves with the complexities of data acquisition and composition. • IT flexibility with limited impact on business • Aggregation of data providers • Utilizable • Multiple consumers can share the same service for a myriad of use cases (generic, interoperable, flexible consumption patterns), • Governance • Data services also perform a critical governance function - they help centralize metrics, monitoring, version management, reuse of data types, and enforce data visibility and access rules. • Semantics – • Alignment with logical data models • Controlled Access • single point of interaction.
- 14. 14 Common Scenarios for Data Services
- 15. Patterns and Practices Data Virtualization in the API
- 16. 16 Data Services Layer (Data API) A data access layer that abstracts underlying data sources and exposes them as discrete services to form a ‘data API’ ▪ Different users and developers across the enterprise can access data in a secure and managed fashion and share a common data ‘model’ ▪ Provides secure and managed access to data across the enterprise ▪ Provides consistency of data ▪ Hides complexity, format, and location of actual data sources ▪ Supports many consumption protocols and patterns Example: Single data access layer for all development teams to avoid ‘hunting down and interpreting data differently by project’
- 17. 17 Data Virtualization for Data-as-a-Service Denodo provides one-click, zero development REST web services on top of any data model with full-fledge capabilities: • Support XML, JSON, GeoJSON, RSS and HTML • Support for hierarchical structures • Authentication with basic HTTP, Kerberos, OAuth 2.0 and SAML • Self documented with OpenAPI • Available in REST, OData, and GraphQL formats
- 18. 18 Uses – Service Container
- 19. 19 Uses – Data Source
- 20. 20 Uses – API Gateway Denodo – API Gateway
- 21. 21 Patterns – Data Access Pattern Description Use Aggregation Aggregate result sets for consumption Offload processing from front-end to back-end. Exploits pushdown capability. Also, can be used in conjunction with caching and/or query acceleration. Augmentation (Fusion) Provide additional calculated or derived elements New fields that contain data expanded from existing. Often done to avoid storing or modifying the underlying system. Examples include adding geospatial info or formulas sourced from multiple columns. Blending Link multiple data elements (from different sources) into one service Service that incorporates multiple elements together. May specify rules for what/when to blend. Data exists in separate repositories linked by unique identifier(s). Filtering Limit data returned in result Often driven by security to limit (rows) or restrict (columns) based on the type of requestor. When used for performance, becomes Microservice Architecture Pattern. 21
- 22. 22 Patterns – Microservice Architecture Pattern Description Use (Entity) Domains Surface logical data domains over existing systems Enable microservices to be grouped together to express particular (functional) domain areas. Often driven by ownership (centralized or distributed) and control (e.g. official/curated). Leverage traversable relationships. Composite Combine (multiple) other service calls through a single “composite” service Abstract complexity of underlying services, including drivers for security, transactionality, performance, and modeling. Sharing Expose service to a different tier Typically used to expose “data” to “application” tier or from “internal” an org to “external.” Often used in association with an API Management tool and use of Web-based IdP (SAML, OpenID, and OAuth2) solutions. 22
- 23. Use Cases
- 24. Business Process – Chip Manufacturer 24
- 25. Analytic Models – Prologis 25 Web Service (Python Model Scoring)
- 26. Capabilities
- 27. Data as a Service (DaaS) using Microservices 27
- 28. Capabilities for Data Services • Data models (tables, views, stored procedures) available automatically as web services - zero coding required • Available in multiple formats: RESTful (XML, JSON), OData 4, GeoJSON • Support for GraphQL: flexible new format for data services • Automatic documentation (OpenAPI) and integration with Data Catalog • Authentication with modern protocols like OAuth 2.0 • Authorization based on roles with, including column/row restrictions and masking • Workload management: priorities, quotas (queries per hour), restrictions by user/role/IP, etc. • Caching and query acceleration capabilities • Integrates with BPMs, iPaaS and API Management tools • Monitoring, access auditing 28
- 29. GraphQL • Zero code needed to publish a GraphQL interface • No n+1 query issue • All the power of the Denodo Data Virtualization engine underneath • Advanced query capabilities (optional) • Integrated with Denodo's security infrastructure • GraphQL-enabled web applications and frameworks can now talk to Denodo 29
- 30. Key Takeaways
- 31. Key Takeaways 1. Data Virtualization enables reduced time-to- market and improved data asset utilization via APIs in modern data ecosystems 2. Decoupling access and storage is a fundamental concept with APIs and Data as a Service 3. Real-time is especially important when interacting with business processes and analytic models 4. Microservice approaches like REST, OData and GraphQL augment data use 31
- 32. Q&A
- 33. 33 Next Steps Access Denodo Platform in the Cloud! Start your Free Trial today! www.denodo.com/free-trials GET STARTED TODAY
- 34. Thanks! www.denodo.com [email protected] © Copyright Denodo Technologies. All rights reserved Unless otherwise specified, no part of this PDF file may be reproduced or utilized in any for or by any means, electronic or mechanical, including photocopying and microfilm, without prior the written authorization from Denodo Technologies.