[pgday.Seoul 2022] PostgreSQL with Google Cloud
0 likes459 views
Google Cloud offers several fully managed database services for PostgreSQL workloads, including Cloud SQL and AlloyDB. Cloud SQL provides a fully managed relational database service for PostgreSQL, MySQL, and SQL Server. It offers 99.999% availability, unlimited scaling, and automatic failure recovery. AlloyDB is a new database engine compatible with PostgreSQL that provides up to 4x faster transactions and 100x faster analytics queries than standard PostgreSQL. It features independent scaling of storage and computing resources. Google Cloud aims to be the best home for PostgreSQL workloads by providing compatibility with open source PostgreSQL and enterprise-grade features, performance, reliability, and support across its database services.
[pgday.Seoul 2022] PostgreSQL with Google Cloud
- 1. PostgreSQL with Google Cloud 전병찬 Data Management Specialist Google Cloud August 2022
- 2. That’s why 75% of all databases* are expected to be in the cloud this year Cloud offers organizations agility, cost savings, and differentiated capabilities * Source: Press Release: Gartner Says the Future of the Database Market Is the Cloud
- 3. Spanner Bigtable Datastream Google Cloud: The best place to run your PostgreSQL database workloads Bare Metal Solution Cloud SQL AlloyDB Memorystore MySQL PostgreSQL SQL Server Oracle Redis Memcached Database Migration Service Relational In-memory Document Key Value Firestore PostgreSQL Compatible PostgreSQL Interface Managed third-party database engines Google’s native database engines
- 4. Portability Simple license management Cost effective Enterprise features Extensible architecture Built to scale Millions of users Used in mission critical applications Friendly community Support Why are developers choosing PostgreSQL? Open source Rich functionality Proven Strong community *Stack Overflow Developer Survery 2022
- 5. Enterprise-ready fully managed relational database service for PostgreSQL, MySQL, SQL Server Google Cloud is the best home for your PostgreSQL workloads Cloud SQL PostgreSQL-compatible database ready for enterprise level workloads Unlimited global scale and 99.999% availability with PostgreSQL interface AlloyDB Cloud Spanner
- 6. Google Cloud is the best home for your PostgreSQL workloads Enterprise-ready fully managed relational database service for PostgreSQL, MySQL, SQL Server Cloud SQL PostgreSQL-compatible database ready for top-tier workloads Unlimited global scale and 99.999% availability AlloyDB Cloud Spanner
- 7. Fully Managed & Enterprise Ready Easy to set up, operate, and scale Trusted Enterprise-grade data protection, security and governance Supports PostgreSQL, MySQL and SQL Server Full compatibility with source database engines Developer Friendly Application centric observability and API-first administration Over 90% Of GCP’s top 100 customers use Cloud SQL Cloud SQL Fully managed relational database service
- 8. Managed by customer Managed by Cloud SQL Hardware & Networking Security OS Database Maintenance HA Scalability Application Development Monitoring ● MySQL ● PostgreSQL ● SQL Server Cloud SQL Focus on innovation, not infrastructure with fully managed services
- 9. Key Benefits of Google Cloud SQL for PostgreSQL Compatibility Cloud SQL offers standard Postgres (9.6 -> 14) databases . Use standard connection drivers and built-in migration tools to get started quickly. Simple & Fully Managed Easy to use with no manual software installation, data backup or maintenance. HA option. Integrated monitoring and alerts. Performance & Scale Designed for performance-intensive workloads. Easily scale up to 96 processor cores and more than 620GB of RAM. Create databases up to 64TB in size. Security & Compliance Automatic data encryption at rest and in transit. User controlled network access with firewall protection. Cloud SQL is SSAE 16, ISO 27001, PCI DSS v3.0, and HIPAA compliant.
- 10. Cloud SQL for PostgreSQL is innovating rapidly Logical Replication and Decoding IAM Database Authentication Support for 175+ flags and 50+ extensions In-place upgrades Cloud SQL Insights Cost Recommenders with Active Assist Fully tuned for PostgreSQL
- 11. Cloud SQL for PostgreSQL Query & System insights Query Insights Cloud SQL Insights is a simple, open tool that helps developers quickly understand and resolve database performance issues on Cloud SQL System Insights (preview) Displays metrics about the resources and helps you detect and analyze system performance issues
- 12. Cloud SQL momentum Updated Features Deletion protection (Launched) GA Local user password validation (Launched) GA Cascading replicas, Replica HA, Replication from external server GA Key Access Justification GA Self-Service Maintenance GA In-place major version Upgrades GA Serverless Exports GA PG: Cloud SQL System Insights Preview Plv8, pgrouting, amcheck, pg_anonymizer. Pg_bigm, refint, pg_largeobjects, pg_shadow, decoderbufs, pg_wait_sample GA Reduced Maintenance Downtime (<30s) GA IAM authentication PostgreSQL GA BigQuery to Cloud SQL federation GA Database Migration Service GA Cross region replicas GA
- 13. Why choose Cloud SQL for your PostgreSQL workloads? Open source PostgreSQL Open APIs Easy, consistent experience 99.95% availability SLA Maintenance controls and low downtime (<30s) Cross-region replicas and Point-in-time Recovery (PITR) Encrypted by default Google global VPC Global Google-owned fiber backbone Integrated with Security Command Center Well-integrated with GKE, CloudRun Analytics via BigQuery, Looker SQL Insights Integrations with Open Telemetry Open Trusted for reliability Trusted for security Development velocity
- 14. Consider using Cloud SQL for… Fully compatible PostgreSQL database with broadest support for major and minor releases on an ongoing basis The need for a common control plane for MySQL, PostgreSQL and SQL Server on Google Cloud Lift & shift migrations off of an existing, self managed PostgreSQL database from on premises or other clouds Enterprise-grade managed PostgreSQL at an attractive entry point
- 15. Google Cloud is the best home for your PostgreSQL workloads Enterprise-ready fully managed relational database service for PostgreSQL, MySQL, SQL Server Cloud SQL PostgreSQL-compatible database ready for enterprise level workloads Unlimited global scale and 99.999% availability AlloyDB Cloud Spanner
- 16. PostgreSQL compatibility The best of Google A new open-source compatible database engine ready for top-tier relational database workloads Introducing AlloyDB Preview
- 17. 4x faster than standard PostgreSQL for transactional workloads TPM 400K 1600K 1200K 800K # of vCPUs 64 AlloyDB Postgre SQL Postgre SQL AlloyDB 16 0
- 18. 100x faster for analytical queries than standard PostgreSQL Up to (lower is better) AlloyDB: 0.42 sec PostgreSQL 14: 60.37 sec Example analytical query: SELECT statement with predicates
- 19. Best of Google AI/ML to databases Pre-integrated with Vertex AI for easy inferencing within database Enables high throughput, low latency augmented transactions 지능적인 기능들 Fully compatible with PostgreSQL 14 Over 175 flags supported Over 50 extensions supported Move your existing PostgreSQL application as-is, with no code changes PostgreSQL에 대한 완벽한 호환성 No licensing or opaque I/O charges Great price-performance Right-size instance when needed Pay-for-what-you-use storage 예측 가능한 투명한 가격
- 20. 99.99% SLA, inclusive of maintenance Automatic and fast failure recovery Multi-zone architecture Non-disruptive management operations 신뢰 가능한 고가용성 Linear read scalability at 1000+ vCPUs Linear write scalability up to the largest instance size Horizontal scale out of database- optimized storage 높은 확장성 엔터프라이즈 수준의 서비스
- 21. Clusters ● 클러스터에는 PostgreSQL 배포를 위한 모든 리소스가 포함됨 ● 리소스 관리의 기본 단위로 관리자가 성능을 모니터링하고 여러 인스턴스에서 정책 및 기능을 간단하게 구성할 수 있음 Primary Instance ● 클러스터의 데이터베이스에 대한 읽기/쓰기를 제공하며 모든 클러스터에는 하나의 기본 인스턴스가 있음 ● 데이터베이스를 정의 및 관리하며, 특히 트랜잭션 처리 워크로드에 적합하지만 데이터 분석 워크로드도 지원 Read Pool Instance ● 클러스터의 데이터베이스 데이터에 대한 읽기를 제공하며, 클러스터에 여러 읽기 풀 인스턴스를 생성할 수 있으며 각 인스턴스의 컴퓨팅 용량을 개별적으로 확장할 수 있음 ● 반드시 필요한건 아니지만 기본 인스턴스보다 데이터 분석 워크로드에 대한 지원이 더 좋음 AlloyDB Cluster
- 22. Disaggregation of compute and storage Modern architecture that scales independently at every level of the stack Within the storage layer itself, automatic rebalancing smooths out load and offers predictable, cost-effective performance Database layer with Cache powered compute instances Horizontally scalable intelligent storage Offload IO
- 23. Zone two Only Log Writes Intelligent Database Storage Engine Google’s Distributed File System - Colossus No BLOCK Writes Intelligent database storage designed and optimized for PostgreSQL Powers fast, predictable performance by eliminating I/O bottlenecks and offloading work to storage service Regional storage improves cluster availability with fast, bounded failover and enable slow-lag read replicas Zone one optimized PostgreSQL optimized PostgreSQL Failover replica Cache Analytics Accelerator Zone (any) optimized PostgreSQL Primary Cache Analytics Accelerator Read pool node Cache Analytics Accelerator
- 24. Row Format Columnar Format AI/ML Driven Auto Columnarization DRAM Query Ultra-fast Cache Scale out AlloyDB Storage Fast and predictable performance Intelligent, workload-aware dynamic data organization leverages both row-based and column-based formats. Multiple layers of cache ensure excellent price-performance.
- 25. Life of a write operation
- 26. Life of a read operation
- 27. Easy to manage with advanced Machine Learning Automatic vacuum management Automatic memory management Automatic storage tiering Automatic data columnarization and query rewrite Autopilot
- 28. Why choose AlloyDB for your PostgreSQL workloads? Fully compatible with PostgreSQL 14 Migrate PostgreSQL workloads without impact to applications 2X faster than Amazon’s comparable PostgreSQL-compatible service for transactional workloads Up to 100x faster analytical queries powered by columnar execution engine Disaggregated storage and compute layer scaling independently to provide predictable, cost-effective performance Linear read scalability up to 1000+ vCPUs with read pools that scale up or down 99.99% availability SLA (including maintenance) Zero maintenance windows for reads and <10s for writes Non-disruptive updates for instance resizing and other configuration changes Auto vacuum, automatic data tiering between DRAM, memory management, storage tiering ML enabled adaptive systems for database tuning Automatic failure recovery Compatible Performance Reliable, scalable and highly available Automation
- 29. Consider using AlloyDB for… Modernizing proprietary databases with high license fees and audits to open source compatible databases in the cloud Mixed transactional and analytical operational database workloads Situations where reduction in PostgreSQL administration overhead around tuning parameters, vacuum, memory management and storage tiering, etc is required PostgreSQL compatible database workloads looking for better performance, availability, scalability and manageability characteristics than what is available with open source PostgreSQL
- 30. Google Cloud is the best home for your PostgreSQL workloads Enterprise-ready fully managed relational database service for PostgreSQL, MySQL, SQL Server Cloud SQL PostgreSQL-compatible database ready for top-tier workloads Unlimited global scale and 99.999% availability with PostgreSQL interface AlloyDB Cloud Spanner
- 31. Philosophy of Cloud Spanner Designed for the unpredictable requirements of today's applications 과거 구글도 동일한 고민을 함 ■ 빠른 성장 ■ 다운타임에따른 수익 손실 ■ 복잡한 관리 구조 ■ 요구사항을만족하는데이터베이스가없었음
- 32. &$985487 Relational semantics Schemas, ACID transactions, SQL Horizontal scale 99.999% SLA, fully managed, and scalable + What is Cloud Spanner?
- 33. What is Cloud Spanner? 관계형 ACID transactions, SQL, Schemas 수평 확장성 Distributed RDBMS, Near unlimited scale 완전 관리형 Simplified administration, Enterprise grade 99.999% uptime SLA Automatic sharding Superior price-performance No maintenance downtime Zero-touch global replication Automatic failure recovery RPO =0, RTO = 0 Online, unlimited scaling Security and compliance Strong external consistency Spanner processes over 2 billion requests per second at peak Spanner has more than 6 exabytes of data under management
- 34. Proprietary Regional Cloud Spanner Data is always replicated for durability, availability, and performance Zone A (Replica 1) DB 1 DB 2 Zone B (Replica 2) DB 1 DB 2 Zone C (Replica 3) DB 1 DB 2 Cloud Spanner instance 99.99% SLA Colossus (Distributed storage) Region
- 35. Proprietary Multi-region Cloud Spanner Multi region example (nam3) Zone A RW - Replica US east4 (Default leader) Zone B RW - Replica Zone A RW - Replica US east1 Zone B RW - Replica US central1 (Witness) Zone A Witness ● Two regions with R/W replicas ● One default leader region ● Witness replica in third region (for event that r/w regions go down). ● Witness do not store data, only used for quorum. ● Tolerate zone and region failures ● 99.999% Availability
- 36. “Our primary reason to move out of AWS was because we encountered various limitations of scale with DynamoDB. The other big reason was cost which became untenable at our scale.” Venkatesh Ramaswamy VP Engineering, Sharechat 30% cost reduction over AWS Start small Granular instances start at $65 USD/month Pay only for what you use Pre-allocated infrastructure is billed only for actual consumption Seamless scaling with predictable pricing All inclusive node-based pricing Committed Use Discounts (CUDs) Up to 40% off and can be layered with other incentives Democratizing Spanner
- 37. Democratizing Spanner What will the program offer? ● 1 Free Instance per project for 90 days trial ● 10GB storage free ● Guided tutorials and sample DB for users to start kicking the tires ● Available in selected regions in US, LATAM, Europe and Asia at launch ● Free instance provides all Spanner features essential for developers. It does not support features such as backups, PITR and and multi-region configuration Upgrading to paid instance ● Flexible upgrading options from free to paid ● Customers can upgrade anytime during the trial period ● Customers can also opt-in to auto-upgrade. ● Instances that are not upgraded after 90 days enter a 30 day grace period ● Once upgraded to paid instance, customers can’t change it back to free instance Free trial instance
- 38. Familiar tools and skills Take advantage of Spanner’s unmatched scale, 99.999% availability, strong consistency using skills and tools from PostgreSQL ecosystem Application portability Enhanced application portability with well-defined migration path to other PostgreSQL environments Faster adoption Reduce hiring and training costs by leveraging existing PostgreSQL resources Cloud Spanner Spanner PostgreSQL Democratizing Spanner PostgreSQL interface
- 39. PostgreSQL Compatibility Provisioning Monitoring Ecosystem clients Stored Procedures Triggers SERIAL Sequences Privileges Isolation control Nested transactions Transactional DDL Partial indexes Extensions Foreign data wrappers psql Data types ● TEXT, VARCHAR ● NUMERIC ● BIGINT ● TIMESTAMP ● FLOAT ● DOUBLE ● BOOL ● BYTEA ● ARRAY INFORMATION_SCHEMA Interleaved tables Functions Operators DQL: SELECT DML: INSERT, UPDATE, DELETE DDL: CREATE, ALTER, DROP Optimizer, query plans Statistics Query hints External consistency Coming soon: JSONB, INTERVAL, PostgreSQL drivers/ORMs Coming soon: TTL, change streams Views Default values PostgreSQL interface for Cloud Spanner PostgreSQL Cloud Spanner Spanner Clients ● Java/JDBC ● Go ● Python ● Node.js ● Ruby ● PHP ● C# ● C++
- 40. ● Specify SQL dialect at database creation time ● Run Postgres dialect SQL over existing Spanner interfaces ○ 8 open-source language drivers (Java, Go, Python, etc.), including downstream applications, like DataFlow ○ gcloud CLI ○ GCP Cloud Console UI ● Run Postgres dialect SQL over the Postgres wire protocol ○ Use Postgres community tools as-is, starting with psql ● Provision and monitor with existing Cloud Spanner control plane User Experience Overview PostgreSQL queries over existing Spanner interfaces, plus PostgreSQL wire protocol
- 41. User Experience: Application Development Application code Open-source Spanner client Java, Go, Python, Node.js. Ruby, PHP, C#, C++ Direct connection using existing Spanner clients Client Application PostgreSQL SQL dialect, types, functions, operators API PostgreSQL dialect Cloud Spanner gRPC
- 42. User Experience: psql psql Popular PostgreSQL community REPL Postgres wire protocol translator PostgreSQL dialect over existing Spanner APIs PostgreSQL dialect, types, functions, operators JDBC Adapter Community tooling via the Postgres wire protocol, starting with psql Cloud Spanner Container/JVM API PostgreSQL dialect gRPC PostgreSQL wire protocol
- 43. Why choose Spanner for your PostgreSQL workloads? Granular instance sizing and committed use discounts (CUDs) make it accessible for developers from companies of all sizes Start small, scale seamlessly and starting as low as $40/month Industry leading 99.999% SLA Zero RPO (data loss) and Zero RTO (downtime) No maintenance windows Online scale up/down and schema changes Unlimited scale for reads and writes Strong consistency at any scale Automatic Sharding Built-in global replication Automatic failure recovery Democratized Access Availability Scalability Automation
- 44. Consider using Spanner for… Global scalability & highest availability requirements with very low downtime tolerance Seamless regional and global replication use cases requiring global distribution Situations needing zero RTO and RPO and strong external consistency at global scale Horizontal scaling of writes without customer managed sharding
- 45. Serverless, secure, and offered at no additional charge Supports migrations of MySQL and PostgreSQL databases to Cloud SQL Support for SQL Server migrations & AlloyDB coming soon More than 85% of DMS migrations are underway in less than an hour* *Google-internal data Database Migration Service makes homogeneous migrations to Cloud SQL & AlloyDB easier and faster
- 46. Database Migration Service now supports Oracle to PostgreSQL schema and data migration Helps reduce your costs and dependence on proprietary databases Serverless data movement and monitoring Integration with the Ora2Pg OSS tool for schema conversion Modernize your legacy workloads to the open cloud Preview Migrate your Oracle workloads from on-premises and in the cloud to Cloud SQL or AlloyDB for PostgreSQL
- 47. Accelerate your cloud journey with our Databases Migration Program Take advantage of incentive funding for partner services to help offset part of the cost of the migration of qualified managed databases workloads* Get connected to expert services partners that can guide you from project scoping all the way to implementation and training Leverage assessment tooling and resources to develop strategy and a project plan for your migration *Qualification criteria apply. Database migrations to Google Cloud managed databases for new workloads only - subject to approval To get more information and get started apply here: https://cloud.google.com/resources/database-migration-program
- 48. Summary PostgreSQL is becoming the relational database of choice for the enterprise Google Cloud offers the best place to run all your PostgreSQL databases Learn more about our database services at cloud.google.com/databases and contact sales for more information
- 49. Thank you