Moving Quickly with Data Services in the Cloud
0 likes165 views
The document discusses the rapid advancements in cloud-based data services, emphasizing the importance of automated provisioning, security, and efficiency in managing databases such as Amazon S3 and Elastic Search. It highlights the significant shift in focus from high availability to time-to-market for startups and outlines best practices for securely and efficiently configuring cloud services. Key recommendations include using infrastructure as code, ensuring secure and reusable code, and leveraging automated systems to minimize human error.
Moving Quickly with Data Services in the Cloud
- 1. Moving Quickly With Data Services In The Cloud Matt Dimich Cloud Architect Thomson Reuters @JobsWithUs #WorkingAtTR #HappyAtTR
- 2. What do you mean by Data Services? Technologies • Aurora MySQL • Amazon S3 • Elastic Search • Aurora PostgreSQL • DMS/SCT • DynamoDB • ElastiCache • Big Data Areas of Concern • Automate Creation/Provisioning • Automate “Schema” Changes • Data Security – Network • Data Security – Access • Data Security – Encryption • High Availability • Cost Monitoring & Reporting • Logging, Monitoring & Alerting • Connection Management • Backup & Restore • Scale & Limits • Retry Logic, Error Handling & Transactional Integrity • Disaster Recovery • Audit Infrastructure • Internationalization • Data Consistency • Maintenance • Support Team
- 3. Data Services Before Cloud… • Provision huge, costly database servers • Elastic Search would provision monster elastic search nodes so they could scale • Lots of human hands would touch each change to the database • Leads to high cost for experiments if you need a database • Large effort to figure out the storage and compute appropriately • (and a big time penalty if you get it wrong) • Focused on High Availability
- 4. What’s changing… • Provisioning takes minutes • Scaling is easier than ever • Start ups suddenly have the power to get an app out with relatively low cost and high scale • Focus has shifted from high availability to time to market.
- 6. Automated Provisioning •How does it work? • AWS CloudFormation creates our database clusters for us. • Consistent • Repeatable • Embedded Standards (i.e. default to encryption at rest) http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-whatis-howdoesitwork.html AWS CloudFormation can update our database clusters as well
- 7. Warning! Mindset Change Required!
- 8. •Don’t create it until you need it
- 9. •Turn off on weekends and nights as possible
- 10. •Start small and scale up later
- 11. Automated Schema Change Pipeline • Reduce human error • Repeatable • Consistent • Same tooling as App Deployment • Lower time to deployment • Build in safeguards
- 12. Multiple Iterations 1. Flyway with Jenkins on Prem 2. Flyway with Spinnaker and Jenkins in AWS 3. Flyway with AWS Code Pipeline in AWS
- 13. DB Change Pipeline with AWS Code Services
- 14. Object Storage
- 15. S3 • Simple Storage Service • Object Storage • Allows hierarchy structure • Has powerful lifecycle rules to expire items or move to items to cheaper storage classes etc. HTTP AWS Command Line AWS Web Console
- 16. Simple Right? It turns out this simple storage service is actually quite complex to set up…correctly.
- 17. How are other companies doing at this?
- 26. But those are already weeks and months old…
- 29. Why is this so hard? • How do you lock down a bucket? • IAM Policy • AWS-managed policy attached to a user, group or role • Self-managed policy attached to a user, group or role • Inline policy for a user, group, or role • Bucket Policy • Bucket ACL • Object ACL
- 30. Oh, and one more thing…
- 31. How to avoid the front page of the news • Spend time creating secure CloudFormation then use that everywhere! • At TR we often use predefined bucket types with built in standards/safeguards • Resources – For your microservice and only your microservice • Infrastructure – What it sounds like, stuff outside the app. • Website – For public facing static content
- 32. AWS Responds Introducing Amazon Macie, Launched August 14, 2017 • “AI/Machine Learning to look for Sensitive Data”
- 33. AWS – “We take care of the undifferentiated heavy lifting”
- 34. Building Blocks…
- 35. How do I restore to a point-in-time with S3? • S3 has versioning for each object • Everything is available on the API • Enter the PIT restore tool • Dry run available • Estimate size • Loops through a bucket, path or just one file and reverts it to the version that was current during the entered point-in-time
- 36. Elastic Search
- 37. Elastic Search • It’s not as easy as saying AWS every time • Platform Elastic Search has automated their Elastic Search distribution on AWS and Azure. • Gives us custom plugins and encryption at rest • A step toward a managed service
- 38. But I Have Experience Running This • Same ES distribution as our Data Centers • Different hardware profile • The team started smaller • But not small enough • Force our applications to ”tip over” the resources before upgrading
- 40. @JobsWithUs #WorkingAtTR #HappyAtTR In Summary 1. Use Infrastructure as code 2. Craft your code securely and carefully 3. Re-use it everywhere So you can… 1. Move Quickly 2. Stay off the front page of the news