Enabling Cloud Data Integration (EMEA)
0 likes91 views
The document outlines the findings from the 2021 Denodo cloud survey, highlighting a significant increase in cloud adoption among organizations, with nearly 80% already utilizing some form of cloud services. Key insights include the preference for hybrid cloud solutions, concerns over limited cloud skills, and the growing importance of data integration and analytics as primary use cases. The Denodo platform is presented as a solution for overcoming challenges in cloud migration and integration, emphasizing data virtualization's role in enhancing productivity and agility in cloud environments.
Enabling Cloud Data Integration (EMEA)
- 1. EMEA WEBINARS Enabling Cloud Data Integration
- 2. Speakers Robin Tandon Product Marketing Director Denodo Sales Engineering Director Denodo Mark Pritchard
- 3. Agenda 1. Key Findings from the 2021 Denodo Cloud Survey 2. Cloud Data Integration, The Journey 3. Denodo Demonstration 4. Case Studies 5. Q & A
- 4. Key Findings from the Denodo Cloud Survey 2021
- 5. 5 Cloud adoption maturity is on the rise, with close to 80% of those surveyed already running some kind of workload in the cloud.
- 6. 6 25% growth in the number of advanced users clearly indicates an upward trend in cloud usage and adoption.
- 7. 7 Close to 25% of participants indicated concerns over the limited cloud skills and ability to manage workload deployments in a multi-cloud environment.
- 8. 8 Hybrid cloud has consistently been the top choice over the past years as organizations migrate workloads to the cloud. More than one-third (36%) of participants are leveraging hybrid cloud while private cloud is still the go-to deployment model for privacy-bound applications or those that safeguard mission critical operations.
- 9. 9 Microsoft Azure, for the second time in a row, has come out as the lead cloud provider, with 34% of participants, with Amazon right behind it. While there could be regional influences, the survey has shown that the EMEA region has a slightly higher preference for Azure compared to the other cloud providers.
- 10. 10 BI and data integration are the top cloud use cases BI and analytics remain the top cloud initiative, while establishing better data integration and expanding data science using AI/ML are tied as the runner up.
- 11. 11 Close to 50% of participants leverage multiple solutions such as data lakes and data warehouses for data management in the cloud. Data virtualization can nicely support data infrastructure modernization while providing core capabilities such as data catalogs and support for handling streams. Demand for data integration in the cloud is driven by data lakes and cloud data warehouses!
- 12. 12 ML/AI and streams processing services see big jumps in usage, close to 50%, while infrastructure usage demonstrates fairly good consumption after analytics.
- 13. 13 Close to 25% of participants indicated concerns over the limited cloud skills and ability to manage workload deployments in a multi-cloud environment.
- 14. 14 45% of our participants see a strong value in leveraging enterprise agreements via marketplaces to close deals faster, and a similar percentage value discount programs, which help exploit budgets from a procurement perspective.
- 15. 15
- 16. 16
- 17. Cloud Data integration – The Journey
- 18. 18 It’s Not “If” or “When”, rather How best to Optimize the journey ! Migrating workloads to cloud ?
- 19. 19 Understanding Cloud Migration PUBLIC CLOUD Move data or applications or both from on-premises to Public cloud HYBRID CLOUD Move partial workloads to the cloud, with some still on premises MULTI-CLOUD Migration of on- prem apps /data to multiple public clouds (use case driven) PRIVATE CLOUD Build an on-premises cloud computing platform
- 20. 20 Common Migration Methods from On-Premises to Public Cloud v
- 21. 21 Denodo Platform 8.0 Architecture DATA CATALOG Discover - Explore - Document DATA AS A SERVICE RESTful / OData GraphQL / GeoJSON BI Tools Data Science Tools SQL CONSUMERS DATA VIRTUALIZATION CONNECT to disparate data in any location, format or latency COMBINE related data into views with universal semantic model CONSUME using BI & data science tools, data catalog, and APIs Self-Service Hybrid/ Multi-Cloud Data Governance Query Optimization AI//ML Recommendations Security LOGICAL DATA FABRIC SOURCES Traditional DB & DW 175+ data adapters Cloud Stores Hadoop & NoSQL OLAP Files Apps Streaming SaaS
- 22. 22 A logical data layer – a “data fabric” – that provides high-performant, real-time, and secure access to integrated business views of disparate data across the enterprise The Denodo Platform • Data Abstraction: decoupling applications/data usage from data sources • Data Integration without replication or relocation of physical data • Easy Access to Any Data, high performant and real-time/ right- time • Data Catalog for self-service data services and easy discovery • Unified metadata, security & governance across all data assets • Data Delivery in any format with intelligent query optimization that leverages new and existing physical data platforms
- 23. 23 Denodo Data Catalog Data Catalog within Data Virtualization to seamlessly integrate data catalog and data delivery Dynamic Catalog of curated, timely, contextual, and reusable information assets and data services. Govern – Fine-grained privilege that governs access to the catalog; both metadata and information assets. Describe – Ability to describe data assets with categorization, tagging, annotations, lineage and other business-oriented metadata. Usage-based metadata – who, when, what, why, and how of data consumption. Lightweight Data Preparation – Ability to transform, refine, and customize data assets for business use. Enhanced UI – Business-friendly user interface geared to roles such as data stewards, data analysts, and citizen analysts
- 24. 24 But There are Challenges with Cloud Adoption/Migration • Silos remain (Cost and Interoperability). Lack of strategy. • Security concerns in the Cloud (GDPR …) • Performance bottlenecks (data across regions, infrastructure) • Business downtime (complexity of migration, apps/data sources) • Learning new skills and resources. 8
- 25. 25 Stages of the Cloud Journey All systems are on-premise. Using traditional databases, etc. – maybe an on-premise Hadoop cluster. Lots of ETL pipelines. Using Denodo for integrated view of data. Systems are now on-premise and in the Cloud – initially hosted by the preferred Cloud provider. The data is balanced across the different environments although the bulk of the data is initially on-premise. ETL-style data movement is often used to move data from on-premise systems to Cloud-based analytical systems. The systems are more complex and users need to be able to find and access data from on- premise and Cloud locations. In reality, this is a hybrid/multi-Cloud environment, with systems in multiple Clouds (AWS, Azure, GCP, Salesforce, etc.) and a few legacy systems still on-premise. The environment is even more complex as workloads can move between Cloud providers to take advantage of new capabilities, cost optimization, etc. Users still need to find and access data in this environment. System modernization initiatives move applications and data to the Cloud. For critical systems, this migration is typically a phased approach over a period of months (or years). (Note: Most organizations skip this stage and go straight to multi-cloud) Systems have moved to the Cloud (although some systems are still on-premise and cannot be moved to the Cloud). The ‘center of gravity’ for data is solidly in the Cloud. More processing and data integration occurs in the Cloud. Data is moved from on-premise systems to the Cloud using ETL. User data access is predominantly from Cloud systems. On-Premise Transition to Cloud Hybrid Cloud Single Cloud Multi-Cloud
- 26. 26 Stages of the Cloud Journey All systems are on-premise. Using traditional databases, etc. – maybe an on-premise Hadoop cluster. Lots of ETL pipelines. Using Denodo for integrated view of data. Systems are now on-premise and in the Cloud – initially hosted by the preferred Cloud provider. The data is balanced across the different environments although the bulk of the data is initially on-premise. ETL-style data movement is often used to move data from on-premise systems to Cloud-based analytical systems. The systems are more complex and users need to be able to find and access data from on- premise and Cloud locations. In reality, this is a hybrid/multi-Cloud environment, with systems in multiple Clouds (AWS, Azure, GCP, Salesforce, etc.) and a few legacy systems still on-premise. The environment is even more complex as workloads can move between Cloud providers to take advantage of new capabilities, cost optimization, etc. Users still need to find and access data in this environment. System modernization initiatives move applications and data to the Cloud. For critical systems, this migration is typically a phased approach over a period of months (or years). On-Premise Transition to Cloud Hybrid Cloud Single Cloud Multi-Cloud (Note: Most organizations skip this stage and go straight to multi-cloud) Systems have moved to the Cloud (although some systems are still on-premise and cannot be moved to the Cloud). The ‘center of gravity’ for data is solidly in the Cloud. More processing and data integration occurs in the Cloud. Data is moved from on-premise systems to the Cloud using ETL. User data access is predominantly from Cloud systems. 1 2 3
- 27. 27 Cloud Migration Options Re-Host – ‘Lift and Shift’ Take existing data and copy it to Cloud “as is” into same database ▪ Good for smaller data sets or data sets with low importance Re-Platform Relocate to new database running on Cloud – everything else stays the same ▪ e.g. move from Oracle 12g to Snowflake Re-Factor/Re-Architect Move to a different database *and* change the data schema ▪ e.g. move from Oracle to Redshift and re-factor data model, partitioning, etc.
- 28. 28 Cloud Migration Using Data Virtualization • Large or critical Cloud migrations are risky • Big Bang approach is not advised • Phased approach is recommended • Select data set to migrate, copy to Cloud • Test and tune data access, then go live • Repeat for next data set and so on • Use Denodo as abstraction layer during migration process • Isolate users from shift of data
- 29. 29 Hybrid Data Integration with a Logical Data Fabric Common access point for both on-premise and cloud sources • Access to all sources as a single schema with no replication: Virtual data lake • Enables combination of data across sources, regardless of nature and location • Allows definition of common semantic model • Single security model and single point of enforcement Active Directory Data Center Cloud
- 30. 30 Multi-Cloud Integration with Logical Data Fabric Amazon RDS, Aurora US East Availability Zone EMEA Availability Zone On-prem data center
- 31. 31 Simplifying Cloud Architectures with Data Virtualization Product Demo
- 32. 32 Scenario Cloud Migration Demo EDW Amazon Redshift Master Data Amazon Aurora RDS On-Premise Data Center Store Sales Date Legacy Master Data Customer Profitability Report Customer (Interface) Customer (Interface) Abstraction Layer
- 33. DEMO
- 35. Business Need Solution Benefits Global industrial real estate company creates a new data architecture and successfully launches data analytics program for cost optimization. Case Study • Create a single governed data access layer to create reusable and consistent analytical assets that could be used by the rest of the business teams to run their own analytics. • Save time for data scientists in finding , transforming and analysing data sets without having to learn new skills and create on data models that could be refreshed on demand. • Efficiently maintain its new data architecture with minimum downtime and configuration management. • The analytics team was able to create business focussed subject areas with consistent data sets that were 30% faster in speed to analytics. • Denodo made it possible for Prologis to quick start advanced analytics projects. • Denodo deployment was as easy as a click of a button with centralized configuration management and easy to upgrade and scale up and down according to the workload. • Denodo was used to create a logical data warehouse that made the data available for analytics. Data Catalog feature used by data scientists to find the data easily. • Denodo was used to leverage the microservices architecture to push enterprise data into the data modals and then get the result set back into Denodo to make them available for consumption. • Terraform used to script a lot of configurations that were running Denodo. CICD pipeline used to automate the Denodo configuration backup. 35 Prologis is the largest industrial real estate company in the world, serving 5000 customers in over 20 countries and USD 87 billion in assets under management. Prologis was ranked the top U.S.company and sixth overall among the 2019 Global 100 Most Sustainable Corporations in the World at the World Economic Forum in Davos.
- 36. 36 Old State Architecture 8 Database 12 Integration 5 Reporting 2 Virtual Desktop 27 Servers 4 Environments
- 37. 37 Current State Architecture wc_monthly_occupancy_rpt_f wc_lease_amendment_d w_day_d wc_property_d MARKET_AVAILABILITY_CURRENT MARKET_AVAILABILITY_FUTURE Prologis SnowFlake API Access Informatica Cloud ShareHouse ODBC JDBC peoplesoft_gl_actuals yardi_unit_leasing p360_property WAF AWS Lambda APIs
- 38. Conclusions
- 39. 39 Data is a critical asset to any organization – and aligning the right architecture is a fundamental step. Data virtualization is core to a data fabric and accelerates a wide range of initiatives; from self- service cloud analytics to data marketplaces to regulatory reporting and compliance in the cloud. Cloud data integration using data virtualization enhances user productivity and time to value. 1 2 3 Key Takeaways
- 40. Q&A
- 41. 41 Get Started Today Try the Denodo Standard 30-day free trial in the cloud marketplaces CHOICE Under your cloud account SUPPORT Community forum AND remote sales engineer OPPORTUNITY 30 minutes free consultation with Denodo Cloud specialist denodo.com/free-trials
- 42. Thanks! www.denodo.com [email protected] © Copyright Denodo Technologies. All rights reserved Unless otherwise specified, no part of this PDF file may be reproduced or utilized in any for or by any means, electronic or mechanical, including photocopying and microfilm, without prior the written authorization from Denodo Technologies.