GDG DevFest Romania - Architecting for the Google Cloud Platform
2 likes476 views
The document is a comprehensive presentation on architecting applications for the Google Cloud Platform (GCP), detailing various services like Compute Engine, App Engine, and Cloud Functions. It covers critical architectural elements, such as autoscaling, serverless architecture, and data warehousing with BigQuery, while also discussing practical use cases and steps for creating VM instances. The content emphasizes cost savings, scalability, and the flexibility of GCP for modern application development.
GDG DevFest Romania - Architecting for the Google Cloud Platform
- 1. Architecting for the Google Cloud Platform Márton Kodok / @martonkodok Google Developer Expert at REEA.net - Targu Mures November 2018 - Bucharest, Romania
- 2. ● Geek. Hiker. Do-er. ● Among the Top3 romanians on Stackoverflow 127k reputation ● Google Developer Expert on Cloud technologies ● Crafting Web/Mobile backends at REEA.net ● BigQuery/Redis and database engine expert ● Active in mentoring and IT community Slideshare: martonkodok Twitter: @martonkodok StackOverflow: pentium10 GitHub: pentium10 Architecting for the Google Cloud Platform @martonkodok About me
- 3. REEA.net uses GCP Build on the same infrastructure that powers Google
- 4. Google Cloud Platform (GCP) Compute Big Data BigQuery Cloud Dataflow Cloud Dataproc Cloud Datalab Cloud Pub/Sub Genomics Storage & Databases Cloud Storage Cloud Bigtable Cloud Datastore Cloud SQL Cloud Spanner Persistent Disk Machine Learning Cloud Machine Learning Cloud Vision API Cloud Speech API Cloud Natural Language API Cloud Translation API Cloud Jobs API Data Studio Cloud Dataprep Cloud Video Intelligence API Advanced Solutions Lab Compute Engine App Engine Kubernetes Engine GPU Cloud Functions Container- Optimized OS Identity & Security Cloud IAM Cloud Resource Manager Cloud Security Scanner Key Management Service BeyondCorp Data Loss Prevention API Identity-Aware Proxy Security Key Enforcement Internet of Things Cloud IoT Core Transfer Appliance
- 5. Google Cloud Platform (GCP) Developer Tools Cloud SDK Cloud Deployment Manager Cloud Source Repositories Cloud Tools for Android Studio Cloud Tools for IntelliJ Cloud Tools for PowerShell Cloud Tools for Visual Studio Container Registry Google Plug-in for Eclipse Cloud Test Lab Networking Virtual Private Cloud Cloud Load Balancing Cloud CDN Cloud Interconnect Cloud DNS Cloud Network Cloud External IP Addresses Cloud Firewall Rules Cloud Routes Cloud VPN Management Tools Stackdriver Monitoring Logging Error Reporting Trace Debugger Cloud Deployment Manager Cloud Endpoints Cloud Console Cloud Shell Cloud Mobile App Cloud Billing API Cloud APIs Cloud Router Dedicated Interconnect Container Builder
- 6. 1. Virtual Machines - Compute Engine 2. Autoscaling - Design for resiliency, scalability, and disaster recovery 3. Database layer - Identification of storage needs and mapping 4. Serverless - App Engine platform - Cloud Functions 5. Microservices architecture 6. Task Queues - Cloud Pub/Sub - DataFlow 7. Data warehouse in the Cloud 8. Practical use cases 9. Qwiklabs + Coursera Agenda Architecting for the Google Cloud Platform @martonkodok
- 7. Compute Engine - Virtual Machine in the Cloud Compute Engine VM Cloud Firewall Rules Ephemeral IP Labels/Tags: “http-server” Allow port 80 for: “http-server” VM details Architecting for the Google Cloud Platform @martonkodok
- 8. Compute Engine - General Use Architecting for the Google Cloud Platform @martonkodok iOS Android Web Mobile Services REST API Compute Engine Flexibility for Every Workload ● High-Performance, Scalable VMs ● Custom Machine Types ● Persistent Disks or SSD (local or network based) ● Per-second billing ● Networking ● Firewall ● Load balancing ● Compliance & Security
- 9. Architecting for the Google Cloud Platform @martonkodok Creating a VM instance steps
- 10. Creating a VM instance Architecting for the Google Cloud Platform @martonkodok
- 11. Creating a VM instance - Choosing Region, Instance type Architecting for the Google Cloud Platform @martonkodok
- 12. Creating a VM instance - OS, boot disk, Network Architecting for the Google Cloud Platform @martonkodok
- 13. VM instances Architecting for the Google Cloud Platform @martonkodok
- 14. Virtual Private Network - Firewall rules Architecting for the Google Cloud Platform @martonkodok
- 15. Cost savings! “Preemptible” - spot instances Architecting for the Google Cloud Platform @martonkodok ● Much lower price than normal instances ● For fault-tolerant applications ● Can withstand possible instance losses ● Runs for maximum 24 hours ● Might not be always available ● Instance templates Good for: Nodes (eg: Elasticsearch) Batch jobs Worker async tasks Pricing: ranges 3-4 times lower price
- 16. Architecting for the Google Cloud Platform @martonkodok Autoscaling with Instance Groups
- 17. Creating an Instance Group Architecting for the Google Cloud Platform @martonkodok
- 18. Setting Autoscaling settings Architecting for the Google Cloud Platform @martonkodok
- 19. Instance Group - Autoscaling - Health Check @martonkodok Template VM Startup script Instance tag Instance Group Region + Zone(s) Autoscaling Health check Instance Group Backend Compute Engine Multiple Instances Region 2 - EU eu-west 1-b eu-west 1-a Instance Group Backend Compute Engine Multiple Instances Region 1 - US us-east 1-b us-west 1-a Instance Group Backend Compute Engine Multiple Instances Cloud Load Balancing Cloud Firewall Rules Health Check Allow port 80 for: “http-server” Static IP Failover
- 20. Downscale on no load after 10 minute @martonkodok
- 21. Architecting for the Google Cloud Platform @martonkodok Example Architecture for Autoscaling
- 22. CMS with Autoscaling @martonkodok ZONE B Content Server Compute Engine Autoscaling ZONE A Content Server Compute Engine Autoscaling Static Content Cloud Storage Dynamic Content Cloud SQL Publisher Cloud Load Balancing Cloud DNS Architecting for the Google Cloud Platform
- 23. DB multiple zones and Master/Slave topology Architecting for the Google Cloud Platform @martonkodok Zone 1 us-central 1-a Instance Group Compute Engine Web Applications Master Cloud SQL Zone 2 us-central 1-f Instance Group Compute Engine Web Applications Read Replica Cloud SQL ReplicationCloud Load Balancing
- 24. Architecting for the Google Cloud Platform @martonkodok Data layer design
- 25. Database Mapping Architecting for the Google Cloud Platform @martonkodok
- 26. Architecting for the Google Cloud Platform @martonkodok Serverless
- 27. Meet Serverless Architecting for the Google Cloud Platform @martonkodok
- 28. Meet Serverless serverless data center depicted Architecting for the Google Cloud Platform @martonkodok
- 29. Serverless is about maximizing elasticity, cost savings, and agility of cloud computing. @martonkodok Architecting for the Google Cloud Platform @martonkodok
- 30. Serverless types Architecting for the Google Cloud Platform @martonkodok Platforms Triggered Code
- 31. App Engine - application platform Architecting for the Google Cloud Platform @martonkodok API Compute Engine Virtual Machines API App Engine Cloud Endpoints API Container Engine Kubernetes Services Multiple Clients API Requests Cloud Load Balancing ● Fully managed serverless application platform ● Scale your applications seamlessly from zero to planet scale ● Automatically scales depending on your application traffic ● Traffic Splitting (app versions, A/B tests, incremental rollouts) ● Standard vs Flexible environments Python, Java, Node.js, Go, Ruby, PHP, .NET
- 32. App Engine Architecting for the Google Cloud Platform @martonkodok Instance startup time Minutes Seconds Maximum request timeout 60 minutes 60 seconds Background threads Yes Yes, with restrictions SSH debugging Yes No Writing to local disk Yes, ephemeral (disk initialized on VM startup) No Modifying the runtime Yes (through Dockerfile) No Supports installing third-party binaries Yes No Pricing Based on usage of vCPU, memory, and persistent disks Based on instance hours Flexible Standard
- 33. Architecting for the Google Cloud Platform @martonkodok Message Queue systems: Task Queue Cloud Pub/Sub
- 34. Task Queue - Message Queue systems Architecting for the Google Cloud Platform @martonkodok Static Content Cloud Storage Dynamic Content Cloud SQL Dynamic Content Cloud Datastore Front End App App Engine Autoscaling Task Queues Memcache Batch App App Engine Autoscaling Cloud Load Balancing Cloud DNS Workers Compute Engine Autoscaling
- 35. Task Queues Architecting for the Google Cloud Platform @martonkodok Slides: https://www.slideshare.net/martonkodok Title: Architectural Patterns - Message Queues Push queues ● Put with delays Pull queues ● Ability to “tag” ● Lease multiple eg: gameboard updated, game id as tag.
- 36. Architecting for the Google Cloud Platform @martonkodok Cloud Functions
- 37. Cloud Functions On-Premises Servers ApplicationEvent Sourcing Frontend Platform Services Metrics / Logs/ Streaming Event Triggered Cloud Functions Triggered Code Architecting for the Google Cloud Platform @martonkodok Result
- 38. Cloud Functions - Event-driven - FaaS Architecting for the Google Cloud Platform @martonkodok ● Simplest way to run your code in the cloud - just deploy ● Automatically scales, highly available and fault tolerant ● No servers to provision, manage, patch or update ● Pay only while your code runs ● Connects and extends cloud services (In alpha: Go and Java) ● Node 8.11 ● Support for async/await ● Python 3.7.1 ● Flask microframework
- 39. Functions: your gateway to GCP Services Architecting for the Google Cloud Platform @martonkodok Access 20+ Google services from GCF BigQuery Cloud Pub/Sub Cloud Storage Cloud Bigtable Cloud Datastore Cloud Spanner Cloud Machine Learning Cloud Vision API Cloud Speech API Cloud Natural Language API Cloud Translation API Stackdriver Services Applications Firebase APIs Assistant
- 40. Why is adopting serverless hard today? Architecting for the Google Cloud Platform @martonkodok Dependencies Constrained runtimes, frameworks and packages Multi-Cloud Unable to run your workloads on-prem, in the cloud or on a third party service provider 1 2
- 41. New Serverless containers on GCF (EAP) Architecting for the Google Cloud Platform @martonkodok Serverless containers Fully managed BYO workloads Pay for use Alpha invitations coming later this year Sign up: g.co/serverlesscontainers
- 42. ● No servers to provision or manage ● Scales with usage (ready every time for viral spikes or #BlackFriday) ● Availability and fault tolerance built in ● No orchestration in code ● Never pay for idle ● Decoupled: APIs as contracts (Abstract away the complexity) ● Monitored: Metrics and logging are a universal right ● Think concurrent, stateless, queue, stream based. Serverlessmeans Architecting for the Google Cloud Platform @martonkodok
- 43. Architecting for the Google Cloud Platform @martonkodok Reporting & Analytics moved to the Cloud
- 44. Legacy Reporting System App Cloud Load Balancing NGINX Compute Engine 10GB PD 2 1 Database Service (Master/Slave) Compute Engine 10GB PD 4 1 Compute Engine 10GB PD 4 1 Compute Engine 10GB PD 4 1 Report & Share Business Analysis Scheduled Tasks Batch Processing Compute Engine Multiple Instances Architecting for the Google Cloud Platform @martonkodok
- 45. Serverless Reporting System App Cloud Load Balancing NGINX Compute Engine 10GB PD 2 1 Database Service (Master/Slave) Compute Engine 10GB PD 4 1 Compute Engine 10GB PD 4 1 Compute Engine 10GB PD 4 1 Report & Share Business Analysis Scheduled Tasks Batch Processing Compute Engine Multiple Instances BigQuery Data Studio Report & Share Business Analysis Architecting for the Google Cloud Platform @martonkodok
- 46. Architecting for the Google Cloud Platform @martonkodok
- 47. Analytics-as-a-Service - Data Warehouse in the Cloud Scales into Petabytes on Managed Google Infrastructure (US or EU zone) SQL 2011 + Javascript UDF (User Defined Functions) Familiar DB Structure (table, views, struct, nested, JSON) Integrates with Google Sheets + Cloud Storage + Pub/Sub connectors Decent pricing (queries $5/TB, storage: $20/TB cold: $10/TB) *June 2018 Open Interfaces (Web UI, BQ command line tool, REST, ODBC) What is BigQuery? Architecting for the Google Cloud Platform @martonkodok
- 48. Columnar storage (max 10 000 columns in table) Large files for loading: 5TB (CSV or JSON) UDF in Javascript or SQL Append-only tables prefered (DML syntax available) Day column partitioned tables (select * from t where day=’2018-01-01’) Rich SQL 2011: JSON,IP,Math,RegExp,Geocode,Window functions Modern data types: Record, Nested, Struct, Array BigQuery: Convenience of SQL Architecting for the Google Cloud Platform @martonkodok
- 49. “ Data needs to be processed in multiple services. How can we pipe to multiple places? Architecting for the Google Cloud Platform @martonkodok
- 50. Architecting for The Cloud On-Premises Servers Event Sourcing Frontend Platform Services Analyze Metrics / Logs/ Streaming Cloud Storage Architecting for the Google Cloud Platform @martonkodok Cloud Dataflow Process BigQuery Cloud SQL Stream Batch Data Studio Third-Party Tools
- 51. “ We have our app outside of GCP. How can we use the benefits of BigQuery? Architecting for the Google Cloud Platform @martonkodok
- 52. Data Pipeline Integration at REEA.net Analytics Backend BigQuery On-Premises Servers Pipelines FluentD Event Sourcing Frontend Platform Services Metrics / Logs/ Streaming Development Team Data Analysts Report & Share Business Analysis Tools Tableau QlikView Data Studio Internal Dashboard Database SQL Application ServersServers Cloud Storage archive Load Export Replay Standard Devices HTTPS Architecting for the Google Cloud Platform @martonkodok
- 53. ● SQL language to run BigData queries ● run raw ad-hoc queries (either by analysts/sales or Devs) ● no more throwing away-, expiring-, aggregating old data ● no provisioning/deploy ● no running out of resources ● no more focus on large scale execution plan Our benefits using BigQuery Architecting for the Google Cloud Platform @martonkodok
- 54. Summary: When to use what Architecting for the Google Cloud Platform @martonkodok App Engine Unit of computing is apps HTTP request/response Large scalable backends Cloud Functions Smallest unit of computing Event driven architecture Connect & extend services Serverless add-on Run functions, apps & containers on GKE Full portability of your artifacts Run on your own cluster
- 55. Thank you. Slides available on: slideshare.net/martonkodok Reea.net - Integrated web solutions driven by creativity to deliver projects.