Serverless orchestration and automation with Cloud Workflows
0 likes364 views
The document presents an overview of serverless orchestration and automation using Google Cloud Workflows, highlighting its ability to automate complex processes through HTTP-based service orchestration. It discusses various challenges in service connectivity, practical use cases, and features such as built-in error handling, decision-making capabilities, and integration with Google Cloud services. Additionally, it emphasizes benefits such as ease of use, scalability, and reliability for enterprise applications.
![retries.yaml
Retries
Serverless orchestration and automation with GCP Workflows @martonkodok
Retries HTTP status codes [429, 502, 503, 504], connection error, or timeout](https://image.slidesharecdn.com/serverlessorchestrationandautomationwithgcpworkflows-210513115532/85/Serverless-orchestration-and-automation-with-Cloud-Workflows-20-320.jpg)
Serverless orchestration and automation with Cloud Workflows
- 1. Serverless orchestration and automation with GCPWorkflows May 2021 - Devops Pro Europe, Vilnius Márton Kodok / @martonkodok Google Developer Expert at REEA.net
- 2. ● Among the Top 3 romanians on Stackoverflow 190K reputation ● Google Developer Expert on Cloud technologies ● Crafting Web/Mobile backends at REEA.net ● BigQuery + Redis database engine expert Slideshare: martonkodok Articles: martonkodok.medium.com Twitter: @martonkodok StackOverflow: pentium10 GitHub: pentium10 Serverless orchestration and automation with GCP Workflows @martonkodok About me
- 3. 1. Challenges in connecting services 2. What is Workflows? - HTTP based service orchestration and automation 3. Introduction to Workflows - automate complex processes 4. Practical use cases 5. Automate, orchestrate and provide reliable line-of-business automation 6. Conclusions Agenda Serverless orchestration and automation with GCP Workflows @martonkodok
- 4. ● Connectivity - should be easy, but in reality you need to figure out ● Parse the results ● Decisions and conditional step executions ● No built-in error handling, logging ● Struggling with polling, retries, exponential backoff ● Scaling up and down to zero ● Authentication Challenges in connecting services Serverless orchestration and automation with GCP Workflows @martonkodok
- 5. Meet Workflows https://cloud.google.com/workflows Serverless orchestration and automation with GCP Workflows @martonkodok
- 6. Workflows in Google Cloud portfolio Introduction Orchestrate any 1. Google Cloud API 2. SaaS API 3. External or private APIs. Serverless Compute External API’s Google API’s etc... Workflows - orchestrate & integrate SaaS API’s Private API’s Other Clouds
- 7. Step-Automation-as-a-Service - Serverless HTTP service automation Declarative workflow language (YAML, JSON) Decent pricing (internal: $1/100K steps, external: $2.5/100K) *Dec 2020 Built-in decision and conditional executions expression formulas, operation on var Subworkflows similar to routine in a programming language with input/return var Support for external API calls out of the box support outside of Google Cloud Integrates with any Google Cloud product without worrying about authentication What is GCP Workflows? Serverless orchestration and automation with GCP Workflows @martonkodok
- 8. OAuth, OIDC, Secret Manager integration Enterprise Security Keep your workflows secure X Authenticated Invocations Authenticated calls to Google Cloud services Integration with Secret Manager Encryption at rest and in transit External API 1 External API 2 External API
- 9. Code example @martonkodok - callMyFunction: call: http.get args: url: https://us-central1-project123.cloudfunctions.net/... query: metric: NoOfTrs result: metricResult - saveResult: switch: - condition: ${ metricResult.body.TrNo > 100 } call: http.post args: url: https://mydeployment.notify.... body: Metric: ${ metricResult.body.TrNo } getMetric TrNo>100? End Yes No {“metric”:”NoOfTrs”} Notify
- 10. hello.yaml Anatomy of a Cloud Workflow! Serverless orchestration and automation with GCP Workflows @martonkodok workflowRevisionId: 000001-de1 argument: 'null' startTime: '2020-10-16T20:19:34.448323739Z' name: projects/985596417983/locations/us-central1/workflows/hello/executions/95c99744-c73c- 4065-a696-940bc7658c33 result: '"Hello, Cloud Workflows!"' endTime: '2020-10-16T20:19:34.845536315Z' state: SUCCEEDED Deploying gcloud beta workflows deploy hello --source=hello.yaml Executing gcloud beta workflows execute hello --data={“var”:”value”} Describing gcloud beta workflows executions describe-last
- 11. Sample Workflows Serverless orchestration and automation with GCP Workflows @martonkodok
- 12. IT management automation Combine automation with scheduler Wait for service checks Orchestrate work across Compute Engine, PubSub, Stackdriver and other Google Cloud Products 9 AM trigger Start a Compute Engine VM Log the event App Started? No Wait 60 seconds Notify the team
- 13. E-commerce invoice generation with Workflows Steps orchestration Reliable execution, with error handling and retries Orchestration microservices or other API’s Create an invoice Generate PDF Send PDF via email Receive Order Cloud Run App Engine Cloud Run
- 14. Sending reminders to accounts with overdue payments Process array elements Execute steps conditionally For each customer: Get a list of customers Send reminder overdue ? Yes
- 15. Code Examples Serverless orchestration and automation with GCP Workflows @martonkodok
- 16. 1. HTTP Post 2. Sequence two steps 3. HTTP Post combined with Secret Manager credentials 4. Switch block 5. Working with subworkflows Code Examples Serverless orchestration and automation with GCP Workflows @martonkodok
- 17. http_post.yaml Making an external HTTP POST request Serverless orchestration and automation with GCP Workflows @martonkodok { "archived":false, "created_at":"2020-10-16T17:40:17+0000", "id":"bit.ly/35452TM", "link":"https://bit.ly/35452TM", "long_url":"<truncated>", }
- 18. wikipedia.yaml Sequence two steps to get data from Wikipedia Serverless orchestration and automation with GCP Workflows @martonkodok
- 19. subworkflow.yaml Subworkflow Serverless orchestration and automation with GCP Workflows @martonkodok call call def
- 20. retries.yaml Retries Serverless orchestration and automation with GCP Workflows @martonkodok Retries HTTP status codes [429, 502, 503, 504], connection error, or timeout
- 21. Orchestrate and automate the Cloud Serverless orchestration and automation with GCP Workflows @martonkodok
- 22. Firestore Backups the easy way with Cloud Workflows - Cloud Scheduler - Cloud Workflows - Cloud Firestore - Cloud Storage Full guide article on: martonkodok.medium.com Cloud Workflows Cloud Storage Cloud Scheduler Firebase Cloud Firestore backup Roles and Permissions Cloud IAM authorize
- 23. Using Workflows to load Cloud Storage files into BigQuery - Cloud Workflows - Cloud Storage - BigQuery Full guide article on: martonkodok.medium.com Cloud Storage BigQuery Cloud Workflows Authenticated Invocations foreach
- 24. Run shell commands and orchestrate Compute Engine VMs - Cloud Workflows - Cloud Build - Compute Engine - Identity-Aware Proxy Full guide article on: martonkodok.medium.com Cloud Build Shell command Serverless Secure Connect Exec command Firewall Compute Engine Cloud IAP Tunnel Cloud Workflows
- 25. Run shell commands and orchestrate Compute Engine VMs - Cloud Workflows - Cloud Build - Compute Engine - Identity-Aware Proxy Full guide article on: martonkodok.medium.com Cloud Workflow Steps Cloud Workflows Execution Authorize Cloud Build Roles and Permissions Cloud IAM Shell Service Account Start VM Compute Engine Submit Stop VM Compute Engine Firewall Compute Engine Cloud IAP Tunnel wait wait
- 26. Conclusions Serverless orchestration and automation with GCP Workflows @martonkodok
- 27. Reliable workflow execution - execute workflows for enterprise business apps Low latency of execution - no cold starts Built-in error handling out of the box error handling with configurable retry policies Passing variable built-in JSON parsing and expression-based variable manipulation Rich runtime iterating through an array, embedded steps for readability Secret Manager integration out of the box Cloud Logging out of the box integration with Cloud Logging Reading from Firestore read/write an entry using Yaml syntax Benefits of Cloud Workflows Serverless orchestration and automation with GCP Workflows @martonkodok
- 28. Easy to build/operate Scales out Does not lose state Handles errors/timeouts Out-of-the-box support of Cloud APIs Auditable Developer friendly Serverless orchestration and automation with GCP Workflows @martonkodok
- 29. The possibilities are endless Marketing Retail IndustrialandIoT Developer Event driven marketing workflow execution Relay conversions to customer profiles in external services Workflow based emails, discounts, promotions Order management Inventory chain operations Data gathering and processing Synchronize systems Generate state machines Verify equipment lifecycle Workflow based maintenance needs Digitalization of internal policies Automate the Cloud Shell-script replacement Orchestrate devops workflows @martonkodok
- 30. Thank you. Q&A. Slides available on: slideshare.net/martonkodok Reea.net - Integrated web solutions driven by creativity to deliver projects.