Stream Me Up, Scotty: Transitioning to the Cloud Using a Streaming Data Platform
2 likes929 views
The document discusses the transition to cloud services using a streaming data platform, highlighting challenges like data sprawl and consistency while proposing the bridge to cloud pattern for effective migration. It details Monsanto's journey towards integrating cloud architecture, managing data replication, and ensuring security through various technologies. Key lessons learned, future plans for multi-cloud integration, and practical use cases are also outlined.
Stream Me Up, Scotty: Transitioning to the Cloud Using a Streaming Data Platform
- 1. Stream Me Up, Scotty Transitioning to the cloud using a streaming data platform
- 2. About us Chrix Finne, Director of Product Management Bob Lehmann, Architect, Data Platform
- 3. We’ll Talk About ● Everyone is moving to the cloud ● Common challenges ● Transition design patterns ● How streaming platform fits in ● Monsanto’s data challenges ● Monsanto’s streaming architecture / use cases ● Security ● Lessons learned ● Future plans
- 4. How Amazon thinks you will move to cloud
- 5. You have a lot to worry about ❏ Do I migrate to cloud-native databases? ❏ Will my apps survive random cloud failures? ❏ Do I need a failure injection test framework? ❏ Dynamic configuration of hostnames ❏ Do I break my monolith into microservices? ❏ Do I want to migrate to more than one cloud? ❏ Can I migrate back?
- 6. At first, this is no big deal…. App App AppApp DWH DB KV App DB DC1 AWS
- 8. Are you kidding? ● This is expensive ● This is a maintenance nightmare ● We may need more than one region! ● We may need more than one cloud!
- 9. We’ve done this before... This... To this...
- 10. There is a better way
- 11. Bridge to Cloud Pattern
- 12. Key Components 1. Distributed Log 2. Big ReLiable Buffer 3. Great Replication 4. Connect Anywhere 5. Management & Monitoring
- 13. Deeper look at bi-directional replication
- 14. Why is this awesome? ● Proven architecture ● Non-stop low-latency ● One throat to choke ● Cost savings
- 15. But wait - there is more! ● Future-proof architecture ○ With Connect - get the data everywhere. ○ Try cloud services without fear of lock-in ○ “Kafka is our escape valve” ● Multi-zone availability ● Multi-Region architecture ● Multi-Cloud architectures ● Microservices ready ● Jump to stream processing!
- 16. Let’s look at how this worked for...
- 17. Monsanto A sustainable agriculture company • Bringing a broad range of solutions to help nourish our growing world • Headquartered in Saint Louis, Missouri • >20,000 employees in 66 countries • A global company with >50% employees based outside of the United States • One of the 25 World’s Best Multinational Workplaces by Great Place to Work Institute Produce with more judicious use of limited natural resources. improve the lives of the world’s farmers. Increase production to meet needs of a growing population. “We succeed when farmers succeed.” -Hugh Grant, Monsanto CEO
- 19. The Monsanto Corn “Galaxy”
- 20. To Boldly Go Where No Man (or Woman) Has Gone Before... CAPTAIN’S LOG STARTDATE 41153.7 (early 2015) Mission: Develop an enterprise information architecture covering BI to machine learning and everything in between. Maximize use of cloud based assets.
- 21. In What Parallel Universe Are You Living?
- 22. Existing Challenges ● Data sprawl ● Data consistency ● Data discovery ● Data latency
- 23. The Cloud - New Challenges ● Increased data sprawl (microservices’ dirty little secret) ● Can’t forklift applications overnight ● Cloud apps need on-prem data ● On-prem apps need cloud data
- 24. The “Aha” Moment The Log: What every software engineer should know about real-time data's unifying abstraction LinkedIn Engineering Blog post by Jay Kreps Dec. 6th, 2013
- 25. Let’s Clean Up This Mess! Courtesy of Jay Kreps
- 26. Replication Is Your Friend ● Uncontrolled, unsynchronized replication IS BAD ● Controlled, synchronized replication IS NECESSARY AND BENEFICIAL...especially in distributed environments
- 27. The Enterprise DataHub EDH Plan - Create Kafka clusters on prem and in AWS - Establish cross datacenter connection - Provide replication between the clusters - Use AVRO schemas - Only alow Apps To interact with the local Cluster IMPORTANT! Use Schema Registry VPN? DIRECT CONNECT? MIRRORMAKER
- 28. Use Cases And Architecture Evolution
- 29. Initial POC - Operations Metrics Collection
- 30. Use Case - Customer 360
- 31. Use Case - Replication Between Dissimilar Databases
- 32. Use Case - Data Warehouse Replication
- 33. Use Case - Data Warehouse Migration Turn this feed off once cloud solution is complete
- 34. Drives adoption of the platform despite these common “concerns”: ● Our data will never be used by anybody else ● We don’t need our data in near real time ● Our volumes are too low ● Our volumes are too high ● Do we really have to use schemas? Cross Datacenter Replication - The Killer App!
- 36. Security Native Clients ❏ SSL certs used for authentication AND authorization ❏ Principal is defined by the cert DN ❏ ACLs are defined based on DN CN=YourProject, OU=YourOrg, O=YourCompany, L=YourLocation, ST=YourState, C=YourCountry CN and OU are the only attributes defined by clients
- 37. Security REST Clients ❏ API endpoint defined in Network Director for each topic ❏ REST client authenticates with SSO platform and then calls API endpoint ❏ Network Director calls REST Proxy as privileged user ❏ Access to topics is controlled by restricting access to API endpoint
- 39. Security Notes ● Some Kafka tools don’t work with SSL and ACLs ● SSL client certs have to be added to truststore on each broker - requires brokers to be bounced. Patched broker code to load certs dynamically. ● Long term roadmap ○ LDAP authentication and role based authorization for native clients. ○ REST Proxy authorization based on OAuth entitlements
- 40. Lessons Learned ❏ Partition By Default ❏ Use “new” consumer - required for SSL anyway ❏ Lock down Zookeeper ❏ Be prepared for AWS instance termination! ❏ Use Associated VPCs in AWS ❏ Use LVM For disks
- 41. Future - Make Ingress/Egress Part Of The Platform
- 42. Future - Multi-Cloud Integration
- 43. THE END