Build Spark-based ETL Workflows on Cloud in Minutes
0 likes174 views
The document outlines the benefits and challenges of leveraging cloud-based ETL workflows, highlighting the shift towards hybrid cloud deployment. It introduces StreamAnalytix as a solution that simplifies the development of ETL processes with features like graphical interfaces and support for multi-cloud environments. The presentation also discusses a use case involving customer data processing and emphasizes the importance of data quality management.
Build Spark-based ETL Workflows on Cloud in Minutes
- 1. Copyright © 2019 Impetus Technologies, Inc. You are prohibited from making a copy or modification of, or from redistributing, rebroadcasting, or re-encoding of this content without the prior written consent of Impetus. This presentation may include images from other products and services. These images are used for illustrative purposes only. Unless explicitly stated there is no implied endorsement or sponsorship of these products by Impetus. All copyrights and trademarks are property of their respective owners.
- 2. Build Spark-based ETL Workflows on Cloud in Minutes Punit Shah Senior Solutions Architect Saurabh Dutta Product Manager
- 3. Our Mission Enabling a unified, clear and present view of your business.
- 4. Poll
- 5. Our speakers Punit Shah Senior Solutions Architect Saurabh Dutta Product Manager
- 6. Agenda Background of ETL and cloud Features and benefits of cloud Must-haves of a cloud ETL platform Challenges with ETL on cloud Demonstrate an ETL use case on cloud using StreamAnalytix
- 7. Almost three-quarters of businesses now operate partially or fully in the cloud, and that number will exceed 90 percent by 2020. – International Data Group, 2018 Source: https://www.idg.com/tools-for-marketers/2018-cloud-computing-survey/
- 8. One of the biggest shifts in the data integration market is customers asking for hybrid deployment (cloud and on- premises), with the expectation of multi-cloud and cloud-to-cloud integration. – Gartner, 2018 https://www.gartner.com/en/documents/3883264
- 9. Traditional ETL Load Store data into a data warehouse or data mart Extract Get data from the source Transform Filter/map/enrich/ combine/validate/sort
- 10. Why move ETL to cloud?
- 11. Elasticity On-premise cluster Elastic cluster
- 12. Cloud native ETL S3 RDBMS Azure Blob store Azure DW Google cloud bucket Redshift
- 13. IaaS - Infrastructure as a service ETL Job 1 High memory cluster Cloud Infrastructure High CPU cluster ETL Job 2 ETL Job 3 GPU-based cluster
- 14. Total cost of ownership On-premise infrastructure On-demand cloud infrastructure
- 15. However, are these capabilities enough?
- 16. StreamAnalytix combines the must-haves of a modern cloud solution with the ability to simplify and accelerate application development.
- 17. The StreamAnalytix advantage Graphical equivalent of an on-prem solution, in the cloud Build, deploy, and monitor workloads in a single platform Migrate existing ETL processes to cloud visually Deploy in a hybrid or a multi-cloud environment Get higher productivity
- 18. Demo
- 19. Upselling customers with targeted plans Process last one month of CDR data
- 20. Upselling customers with targeted plans Process last one month of CDR data Enrich it with cell tower information
- 21. Upselling customers with targeted plans Process last one month of CDR data Enrich it with cell tower information Apply data quality checks
- 22. Upselling customers with targeted plans Process last one month of CDR data Enrich it with cell tower information Apply data quality checks Combine with customer billing database
- 23. Upselling customers with targeted plans Process last one month of CDR data Enrich it with cell tower information Apply data quality checks Combine with customer billing database Group customers based on certain attributes
- 24. Upselling customers with targeted plans Process last one month of CDR data Enrich it with cell tower information Apply data quality checks Combine with customer billing database Group customers based on certain attributes Analyze customers’ change in call pattern
- 25. Upselling customers with targeted plans Process last one month of CDR data Enrich it with cell tower information Apply data quality checks Combine with customer billing database Group customers based on certain attributes Analyze customers’ change in call pattern Store processed data
- 26. A view into a typical ETL on cloud development process
- 27. Challenges with the manual approach Skilled resources Time-to-market Cluster utilization Orchestration Maintenance nightmare
- 28. Steps in the StreamAnalytix demo Connecting to multiple sources Data quality management Data preparation Enrichment Standardization and unification Storage
- 29. StreamAnalytix — A recap Unified data 360 platform Based on open source Operationalizes ML workflows Application lifecycle management Enterprise grade
- 32. Poll results
- 33. Q&A
- 34. Thank you Visit www.StreamAnalytix.com for a free download or cloud-based trial or contact us at [email protected] for more information or a PoC.
Editor's Notes
- #11: Punit to talk about how must haves alone do not provide the time to market advantage, which is a big challenge in a resource intensive process like ETL. We need features beyond these must haves that can make the ETL process easier faster and far less cumbersome compared than current a pproaches agility
- #12: Punit to talk about how must haves alone do not provide the time to market advantage, which is a big challenge in a resource intensive process like ETL. We need features beyond these must haves that can make the ETL process easier faster and far less cumbersome compared than current a pproaches agility
- #13: Punit to talk about how must haves alone do not provide the time to market advantage, which is a big challenge in a resource intensive process like ETL. We need features beyond these must haves that can make the ETL process easier faster and far less cumbersome compared than current a pproaches agility
- #16: Punit to talk about how must haves alone do not provide the time to market advantage, which is a big challenge in a resource intensive process like ETL. We need features beyond these must haves that can make the ETL process easier faster and far less cumbersome compared than current a pproaches agility
- #18: Add a screenshot of the canvas and show bullets animating on the right side