eChai Developer Meetup | Cloud Native Learnings with AWS
0 likes31 views
The document outlines cloud-native learnings and implementations by Dhaval Nagar of Appgambit, focusing on cost-effective, low-maintenance serverless solutions using AWS technologies. Key insights include the importance of data lifecycle management, optimizing cloud resource usage, and utilizing serverless architectures to build scalable applications across various environments. Additionally, it emphasizes the need for robust monitoring, governance, and adherence to best practices in cloud development and operations.
eChai Developer Meetup | Cloud Native Learnings with AWS
- 1. Cloud Native Learnings with AWS Dhaval Nagar, AppGambit
- 2. About Me ● Founder - AppGambit, AWS Consulting Partner ● Partner - Vizabli Inc. 🛠 ● AWS Serverless Hero ● AWS Certification Subject Matter Expert ● AWS Community Leader, Surat ● Certified in AWS, Google Cloud, Docker and Kubernetes ● Web3 Enthusiast ⛓ ● Practicing Barista ☕
- 4. Our Recent Implementations Building a Cost-effective, Low-Maintenance Serverless Data Pipeline Optimising Cloud Resources with minimal refactoring and process disruption Building micro-service architecture that can work in heterogeneous environments
- 5. Eco-system of Amazon Web Services ● Modern Compute ○ EC2 - Virutal Machine ○ ECS and EKS - Container Services ○ Lambda - Serverless Compute ● Modern Databases ○ DynamoDB ○ Aurora Serverless (MySQL and Postgres) ○ RDS Proxy for Connection-less Database Interaction ● Storage Service - AWS S3 ● Developer Tools ○ CodeCommit ○ CodeBuild, CodePipeline, CodeDeploy ○ AWS SDKs in various languages ● Cloud Observability and Monitoring ○ CloudWatch Logs, Events ○ CloudTrail
- 7. AGONICS - Optimising Cloud Assets ● Storage is the Most Critical Part for any applications ○ Raw Data, Logs, Media Assets, etc ○ Usual pattern is store and forget ○ Many times data are part of the same VM where the application is running ○ Increasing data means increasing disk size ○ At times, there is no metrics of which data is used when ● Most applications use Multiple Environments ○ Dev, QA, Customer QA, Production ○ On average 33% time the non-Production environments are sittle idle ○ As applications are becoming complicated, it depends on many services that also run with minimal usage ○ Many Cloud Services still charge per-second, irrespective of actual usage
- 8. ● S3 is the primary storage for the their application ● Cloud storage is cheap, but you can still overpay ● As the data grow, it becomes extremely difficult to monitor, unless you have dedicated team
- 9. ● After understanding their core application, we realise the data follows a very standard pattern ● Each process receives input data and generates output data ● Once processed, input data becomes less important ● After a certain period, the input data loses the intrinsic value in the application
- 10. ● Lifecycle rules help to simplify the data transition ● S3 has various Storage Tiers for Hot, Warm and Cold data types
- 12. ● With growing demand and growing team, Agonics started utilising a lot of AWS services ● Many of these services are having low or in-frequent usage ● But these services are high in per-second cost ● We decided to deploy Schedule Rules to scale-in or hibernate resources based on time schedule ● Resource Tags are an important feature through which we can attach metadata to Cloud resources like EC2 Instances, RDS Databases, Load Balancers, etc
- 13. ● Lambda functions along with CloudWatch Events or EventBridge can allow you to trigger Serverless compute processes ● Combined with AWS Resources Tags, we can invoke some complicated scheduling rules that allow their team to operate different environments as per team’s own schedule
- 14. Learnings ● Understand the lifecycle of your application data ● Keep the important data outside of the compute environment (stateless compute) ● Monitor and Measure the data usage ● If required (compliance, backup) to store, Batch and Compress to further optimise ● Your Cloud usage will grow as your application or team grows, but not all usage is CRITICAL ● Identify opportunities to TURN OFF the SERVER LIGHTS
- 16. Listen4Good - Building Serverless Data Pipeline ● 500+ NGOs under Listen4Good in United States ● They use SurveyMonkey as their Survey platform to create and distribute surveys for performance feedback for NGOs ● NGOs create hundreds of surveys with thousands of respondents ● L4G uses the Quantitative Analysis to generate performance reports ● SurveyMonkey is a solid platform for Survey management but Extremely Expensive when it comes to providing custom solutions ● The problem was to extra meaningful data from SurveyMonkey and build the Analysis Reports independently ● Their team wanted a No-Ops solution
- 17. ● We decided to go for Serverless option ● The use case was predictable and has no dependencies on particular hardware or OS level libraries ● Amazon Lambda is the core Serverless service in the Ecosystem of AWS Serverless Stack
- 20. Learnings ● Each problem is different ● Explore different ways to solve the problem in optimal way ● In last 16 months, the solution had 0 incidents and requires no dedicated resource for management. ● AWS is leader in Serverless space and has a growing list of services in the Serverless Ecosystem.
- 22. Vizabli - Designing Hybrid Solution ● Vizabli is a platform for Acute Care Engagement Solution ● The solution includes Hardware Devices along with Software stack ● Due to compliance and data privacy, the solution is preferred to run inside the Hospital’s private network ● Although, the solution is eventually required to run on the AWS Cloud as well ● Our challenge was to design a scalable solution that can work across Heterogeneous Environment easily
- 24. ● Container is a pioneer technology in the software development process ● Docker is still very popular way to build Application Containers ● We PREFER to package our software irrespective of the technology or infrastructure we use ● Writing the whole application as a Coordinated Bunch of Service Containers helped us create a modular solution
- 25. ● Containers can be orchestrated depending on your target environment ● We use Docker Compose for our minimal setup ● And use Kubernetes for our High-Availability Production Setup
- 28. Learnings ● Understand your target environment and design flexible services ● It’s very likely that the same software may be required to run in different environment in future ● AWS Developer Tools is a set of services that helps layout DevOps practices for any type of softwares ● Most of these services are pay-per-use and easy to setup. ● A consistent build and deployment process helps increase developer productivity and software release cycle.
- 29. ● Isolate Each Environments ● Use IAM Roles for each user and resource, prefer least-priviledge permission model ● Separate Dev/Staging vs Production ● Use AWS Organization with Service Control Policies to limit the unwanted usage ● ALWAYS PACKAGE your software ● Practice 12 Factor App Methodology https://12factor.net/ ● Prefer Process/Practice Automation over human interaction ● Keep an eye on your Cloud Spending Cloud Goveranance
- 30. Thank You!