Harnessing the Power of Distributed Processing: Managing Data Across Clouds and On-Premises
0 likes458 views
The document discusses the challenges organizations face in managing their data across clouds and on-premises due to compliance, scalability, and security issues. It introduces the FME Flow and Remote Engines Service, which facilitates distributed data processing, enabling organizations to process data close to its location for improved performance and security. Additionally, it highlights the importance of adapting to a changing data landscape and offers resources for further engagement and learning.
Harnessing the Power of Distributed Processing: Managing Data Across Clouds and On-Premises
- 1. Harnessing the Power of Distributed Processing: Managing Data Across Clouds and On-Premises
- 2. Richard Mosley FME Server Tech Support Team Lead Don Murray Co-Founder and CEO
- 3. Welcome to Livestorm. A few ways to engage with us during the webinar: Audio issues? Click this for 4 simple troubleshooting steps.
- 4. Poll: What best describes your data landscape?
- 5. Agenda 1 Setting the Stage 2 FME Flow and Remote Engines Service 3 Benefits and CPU Time License Option 4 Demo 1 - Intro to Remote Engines 5 Demo 2 - Multi-Cloud and On-Premises 6 Bonus Demo - Massive Parallelism 7 Conclusion 8 Resources & Next Steps 9 Q&A Agenda
- 7. Organizations today are faced with a constantly changing data landscape.
- 8. Data Landscape Challenges Compliance, Digital Sovereignty Scalability Flexibility and Agility Distributed Data Data Security
- 9. Scalability Data Volumes are exploding Data Variety is increasing Data Velocity is more varied Data Veracity is more important than ever. Data Value is no longer questioned as a business driver. Systems must scale as data volumes grow.
- 10. Distributed Data An Organization's data is spread across ● geographic locations, ● cloud providers, ● on-premises Where do the Applications you use store their data? ● Which clouds? ● Which data centers? ● How much data do you have on-premises? Processing is best close to the data
- 11. Flexibility and Agility Reasons for data movement ● Cost Optimization ● Performance and Scalability ● Disaster Recovery and Backup ● Compliance and Security ● Accessibility Data Integration must ● be where your data is, ● be able to move your data, ● move with your data. Tools must adapt to your changing data landscape
- 12. Data Security Top of mind for every organization Ensure data is stored where it is needed. Don’t move data more than necessary. Restrict data access as much as possible. Number of major data breaches reported
- 13. Digital Sovereignty and Regulatory Requirements Residency Laws Compliance requirements have a growing impact on where and how data is transferred. GDPR Greatly increased regulatory scrutiny. PIPL Data residency rules for China. More Countries are Following Data is important for national security and to protect citizens. Processing of Data must obey sovereignty rules! % of data covered by Compliance and Regulatory Requirements GDPR PIPL
- 14. Processing is best when it is close to the data! ALWAYS!
- 15. 2 FME Flow and Remote Engines Service
- 16. Traditional FME Flow ● Distributed Data Processing: Data processing distributed across machines within one location. ● Parallel Processing: With Automations you can engage parallel processing ● Deployment: Deployment within one firewall. Optimized for one data center, one deployment type, one local area network. ● Scalable: Deploy as much processing as needed within one location. FME Flow is optimized for deployments with all machines in one geographic location.
- 17. FME Flow has a united control plane and processing plane.
- 18. FME Flow control and processing planes integrated FME Flow Deployment Any Cloud, On-premises, Containers, Native, Serverless Product Roadmap FME Flow Control Plane Local FME Engines (local network, within firewall) FME Flow Processing Plane
- 19. FME Remote Engines Service adds a separate processing plane capability to FME Flow.
- 20. FME Flow Processing Planes (FME Remote Engines Service) Leverage separate FME Flow control and processing planes. FME Remote Engines Service Any Cloud, On-premises, Containers, Native, Serverless FME Flow Control Plane Local FME Engines (local network, within firewall)
- 21. With FME Remote Engines you can move the Processing Anywhere
- 22. FME Remote Engines Service
- 23. FME Flow with FME Remote Engines Service ● Distributed Data Processing: Data processing across machines in multiple locations. ● Parallel Processing: Automations orchestrate parallel processing across locations. ● Flexible Deployment: Across Firewalls, data centers, multiple deployment types, and multiple disconnected networks. ● Scalable: Deploy as much processing as need in each location. ● Geographic Distribution: Deploy engines in different geographic locations. Process data where it resides for enhanced performance. ● Enhanced Data Security: Processing is moved to the data honoring digital sovereignty.
- 24. Snowflake in Development ● FME Engines Service inside database instance. ● More systems making this possible. You can’t get closer to the data than that. One exciting Example
- 25. 3 Benefits and CPU Time License Option
- 26. Maintain simplicity with easily replicable nodes
- 27. Increase stability with online and o ine functionality for Guaranteed Delivery
- 28. One Central Control FME Flow Hosted FME Flow
- 29. Different Department, Different Permissions More control over service account permissions Silo your departments permissions with a remote engine
- 30. Cut down on the commute. Install engines near the data Data Lake On Premise Geodatabase File Format Data SQL Database
- 31. Transform your data when you want and where you want. CPU Engines:
- 32. CPU Engines: Never an Idle Engine CPU Time Pool 10 Engines for parallel processing 1 Engine to listen to a stream 7 Engines for nightly processing in Europe 7 Engines for nightly processing in Australia
- 33. Bulk process with unlimited engines with CPU Time
- 34. 4 Demo 1 - Intro to Remote Engines
- 35. Demo
- 36. Demo 1 - Add Remote Engine and Queue
- 37. 5 Demo 2 - Multi-Cloud and On-Premises
- 38. Demo
- 39. Demo 2 - Multi-Cloud and On-Premises 3 Processing Planes ● AWS ● Azure ● On-Premises Orchestration of all processing. ● Parallel and sequential processing. ● Distributing the processing is just handled AWS Azure On-Premises
- 40. Demo 2 - Multi-Cloud and On-Premises
- 42. Parallel Processing Gratuitous Parallelism Split a task and process with 1,000 engines. Massive parallel processing Knowledge Article: Job Orchestration with Automations
- 43. The “Splitter” Workspace Splitter pattern, used to break the large job into 1,000 smaller jobs. This allows for small jobs to be processed across many engines in parallel. Knowledge Article: Getting Started with Enterprise Integration Patterns Automation Writer is key
- 44. The “Worker” Workspace A workspace that completes the work for the data that is passed in. Here for fun we write it out to a web service so we can visualize the process. The“Splitter” and “Worker” combination is a pattern for processing tasks where individual data elements can be processed independently. Knowledge Article: Getting Started with Enterprise Integration Patterns
- 45. The Automation The Automation is very simple. You can see how the Splitter and Worker workspaces work together. The amount of parallelism depends on the number of engines. Here they are all part of one processing plane. Blog: FME Server 2020 & Enterprise Integration Patterns Splitter Worker
- 46. 1 Engine vs. 1,000 Engines
- 47. 7 Conclusion
- 48. Today’s Data Landscape Challenges Compliance, Digital Sovereignty Scalability Flexibility and Agility Distributed Data Data Security
- 49. The ability to adapt processing to match your changing data landscape is critical.
- 50. FME Flow with FME Remote Engines Service built for this
- 51. 8 Safe & FME
- 52. 29+ 27K+ 128 190 20K+ years of solving data challenges FME Community members countries with FME customers organizations worldwide global partners with FME services 30+ 29K+ 128 140+ 25K+ years of solving data challenges FME Community members countries with FME customers organizations worldwide global partners with FME services 200K+ users worldwide Safe & FME
- 53. One platform, two technologies FME Form FME Flow Data Movement and transformations (“ETL”) workflows are built here. Brings life to FME Form workflows FME Flow Hosted Safe Software managed FME Flow fme.safe.com/platform FME Enterprise Integration Platform Safe & FME
- 54. Safe Software is recognized as Customers’ Choice again in the 2024 Gartner Peer Insights ‘Voice of the Customer’: Data Integration Tools report. We are now recognized as Customers’ Choice in North America and Midsize Enterprise segments. GARTNER is a registered trademark and service mark, and PEER INSIGHTS is a registered trademark, of Gartner, Inc. and/or its affiliates in the U.S. and internationally and are used herein with permission. All rights reserved. Gartner Peer Insights content consists of the opinions of individual end users based on their own experiences, and should not be construed as statements of fact, nor do they represent the views of Gartner or its affiliates. Gartner does not endorse any vendor, product or service depicted in this content nor makes any warranties, expressed or implied, with respect to this content, about its accuracy or completeness, including any warranties of merchantability or fitness for a particular purpose. Read Full Report
- 55. 9 Resources
- 56. Get our Ebook Spatial Data for the Enterprise fme.ly/gzc Guided learning experiences at your fingertips academy.safe.com FME Academy Resources Check out how-to’s & demos in the knowledge base support.safe.com Knowledge Base Webinars Upcoming & on-demand webinars safe.com/webinars
- 57. Check out our podcasts on-demand. featuring special guest speakers over at EM360 Resources
- 58. 10 Next Steps
- 59. We’d love to help you get started. Get in touch with us at [email protected] Experience the FME Accelerator Contact Us A world where data is not just a commodity but a catalyst for real change. fme.safe.com/accelerator Next Steps
- 60. ClaimYour Community Badge & Dive into the new Community! ● Get community badges for watching webinars ● community.safe.com ● Today’s code: K3FAG5 Join the Community today! Next Steps
- 61. 11 Q&A
- 62. ThankYou Recap of Next Steps 1 Join the FME Community 2 Contact us 3 Experience the FME Accelerator Please fill out our webinar survey