PGConf APAC 2018 Keynote: PostgreSQL goes eleven
0 likes166 views
The document discusses PostgreSQL version 11 and future development. It provides a history of PostgreSQL and its predecessors, describing the development process and community. It summarizes key features committed to version 11, including improvements to partitioning, parallelization, performance and logical replication. It also outlines features proposed for future versions, with a focus on continued enhancements to partitioning and query planning.
PGConf APAC 2018 Keynote: PostgreSQL goes eleven
- 1. PostgreSQL Goes to Eleven! Joe Conway [email protected] [email protected] Crunchy Data March 22, 2018
- 2. Community PostgreSQL 11 12 and Beyond History Development Process PostgreSQL Community and Future Development Community History Development Process PostgreSQL 11 Patches committed Work inprocess 12 and Beyond Actively worked items Under active discussion Joe Conway PGConf APAC 2018 2/37
- 3. Community PostgreSQL 11 12 and Beyond History Development Process University of California Berkeley Courtesy: Bruce Momjian Joe Conway PGConf APAC 2018 3/37
- 4. Community PostgreSQL 11 12 and Beyond History Development Process History: INGRES 1974 – 1985: University INGRES INteractive Graphics REtrieval System Prototype of a relational DBMS University of California at Berkeley Prof. Stonebraker Spawned commercial databases: Sybase, MSSQL, NonStop SQL, others 1980 – Present: Commercialization Commercial success Relational Technologies ⇒ Ingres Corp. ⇒ Computer Associates ⇒ Ingres Corp. ⇒ Actian 2006 – Present: Open Source Joe Conway PGConf APAC 2018 4/37
- 5. Community PostgreSQL 11 12 and Beyond History Development Process History: POSTGRES 1986 – 1994: University POSTGRES ”Post Ingres” University of California at Berkeley Prof. Stonebraker Prototype of an object-relational DBMS POSTQUEL query language Spawned commercial databases: Illustra ⇒ Informix, others Joe Conway PGConf APAC 2018 5/37
- 6. Community PostgreSQL 11 12 and Beyond History Development Process History: PostgreSQL 1994 – 1995: Postgres95 University of California at Berkeley Andrew Yu and Jolly Chen Conversion to SQL More liberal license 1996 – present: PostgreSQL Open-source project PostgreSQL Global Development Group Spawned many derived products Joe Conway PGConf APAC 2018 6/37
- 7. Community PostgreSQL 11 12 and Beyond History Development Process Originators Courtesy: Bruce Momjian Joe Conway PGConf APAC 2018 7/37
- 8. Community PostgreSQL 11 12 and Beyond History Development Process First Core Team + Jolly and Andrew Courtesy: Bruce Momjian Joe Conway PGConf APAC 2018 8/37
- 9. Community PostgreSQL 11 12 and Beyond History Development Process PostgreSQL 10 Year Anniversary Summit - 2006 Courtesy: Alvaro Herrera Joe Conway PGConf APAC 2018 9/37
- 10. Community PostgreSQL 11 12 and Beyond History Development Process PostgreSQL Community Diagram Courtesy of Lætitia Avrot Joe Conway PGConf APAC 2018 10/37
- 11. Community PostgreSQL 11 12 and Beyond History Development Process PostgreSQL Development Cycle Note: future dates are estimates Joe Conway PGConf APAC 2018 11/37
- 12. Community PostgreSQL 11 12 and Beyond History Development Process PostgreSQL Versioning Old Versioning scheme Up to version 9.6 Three segments Major X.Y (e.g. 9.6) Minor X.Y.Z (e.g. 9.6.8) New Versioning scheme Starting with version 10 Only two segments Major X (e.g. 10) Minor X.Z (e.g. 10.3) Joe Conway PGConf APAC 2018 12/37
- 13. Community PostgreSQL 11 12 and Beyond History Development Process Commitfests Commitfest App: https://commitfest.postgresql.org/ CF-entries: Patch or set of patches Implements some goal Proposed for inclusion One or more authors/reviewers Examples: new feature, bugfix, refactoring, improved docs Joe Conway PGConf APAC 2018 13/37
- 14. Community PostgreSQL 11 12 and Beyond History Development Process Commitfests Goals: Patches do not get lost Reviewers/committers can quickly find patches deserving attention Single place tracking discussion and documenting state of CF-entries Encourage: people review other people’s patches review patches proportionally to patches submitted Joe Conway PGConf APAC 2018 14/37
- 15. Community PostgreSQL 11 12 and Beyond History Development Process Commitfests CF-entry States: Needs Review Waiting for Author Ready for Committer Committed Rejected Returned with Feedback Moved to next CF Joe Conway PGConf APAC 2018 15/37
- 16. Community PostgreSQL 11 12 and Beyond History Development Process Commitfests Process: Start on certain date submission deadline date is day before start date, AoE (UTC-12) deadline not passed if, anywhere on earth, deadline date has not yet passed must be in Needs Review or Ready for Committer state CF manager: ensures CF-entries in appropriate state, moves process along Reviewers analyze/test patches, provide feedback Authors adapt entries per reviews/discussion When satisfied, reviewer marks CF-entry Ready for Committer Committer either commits or resets state Joe Conway PGConf APAC 2018 16/37
- 17. Community PostgreSQL 11 12 and Beyond History Development Process Commitfests Process: Waiting for Author too long ⇒ Returned with Feedback At end of commitfest: Needs Review ⇒ Moved to next CF Waiting on Author ⇒ Returned with Feedback Joe Conway PGConf APAC 2018 17/37
- 18. Community PostgreSQL 11 12 and Beyond History Development Process Commitfests Last Commitfest: CF before the expected feature freeze date Extra restrictions: No nontrivial CF-entries unless previously submitted to earlier CF CF-entries deemed unlikely to finish in last CF aggressively be moved early Joe Conway PGConf APAC 2018 18/37
- 19. Community PostgreSQL 11 12 and Beyond History Development Process Commitfests Joe Conway PGConf APAC 2018 19/37
- 20. Community PostgreSQL 11 12 and Beyond History Development Process Release Stats Note: as of late Feb 2018 Joe Conway PGConf APAC 2018 20/37
- 21. Community PostgreSQL 11 12 and Beyond History Development Process Release Stats Note: as of late Feb 2018 Joe Conway PGConf APAC 2018 21/37
- 22. Community PostgreSQL 11 12 and Beyond History Development Process Release Stats Note: as of late Feb 2018 Joe Conway PGConf APAC 2018 22/37
- 23. Community PostgreSQL 11 12 and Beyond Committed Proposed Already Committed by the Numbers (CF1-CF3) Bug Fixes - 59 Clients - 22 ECPG - 2 pg dump - 3 pg receivewal - 2 pgbench - 7 psql - 8 Code Comments - 3 Documentation - 16 Miscellaneous - 22 Monitoring and Control - 3 Joe Conway PGConf APAC 2018 23/37
- 24. Community PostgreSQL 11 12 and Beyond Committed Proposed Already Committed by the Numbers (CF1-CF3) Performance - 26 caching - 3 index - 3 memory - 2 miscellaneous - 4 parallel query - 6 partitioning - 2 plan/opt - 6 Procedural Languages - 4 Refactoring - 27 Replication and Recovery - 5 Joe Conway PGConf APAC 2018 24/37
- 25. Community PostgreSQL 11 12 and Beyond Committed Proposed Already Committed by the Numbers (CF1-CF3) Security - 1 Server Features - 13 authentication - 1 miscellaneous - 4 parallel query - 1 partitioning - 6 security - 1 SQL Commands - 4 System Administration - 5 Joe Conway PGConf APAC 2018 25/37
- 26. Community PostgreSQL 11 12 and Beyond Committed Proposed Already Committed - Notable Features (as of 17 March) Partitioning Many bug fixes and miscellaneous improvements Hash Partitioning Default partition Partition-wise JOIN for partitioned tables pg_dump/pg_restore reload through parent Local partitioned indexes UPDATE moves rows between partitions UNIQUE indexes on partitioned tables Joe Conway PGConf APAC 2018 26/37
- 27. Community PostgreSQL 11 12 and Beyond Committed Proposed Already Committed - Notable Features (as of 17 March) Parallelization Many bug fixes and miscellaneous improvements Queries with InitPlans CREATE TABLE AS queries Prepared statements with generic plans. Queries with Append plan nodes Hash joins Btree index builds Joe Conway PGConf APAC 2018 27/37
- 28. Community PostgreSQL 11 12 and Beyond Committed Proposed Already Committed - Notable Features (as of 17 March) Performance Pushdown LIMIT through subqueries to underlying sort, where possible SET STATISTICS on expression indexes Automatic ”prewarm” for pg_prewarm Configurable WAL segment size at initdb time Index-only Bitmap scans Generational Memory Allocator Improve performance of MemoryContext creation Support huge pages on Windows Push down UPDATE/DELETE joins to remote postgres fdw servers Clone extended stats in CREATE TABLE (LIKE INCLUDING ALL) Joe Conway PGConf APAC 2018 28/37
- 29. Community PostgreSQL 11 12 and Beyond Committed Proposed Already Committed - Notable Features (as of 17 March) Logical replication Many bug fixes and miscellaneous improvements Testing Coverage Analysis improvements Additional tests Improved code coverage Authentication Custom search filters for LDAP auth LDAPS support libpq connection parameter scram_channel_binding SCRAM channel bindings tls-unique and tls-server-end-point Joe Conway PGConf APAC 2018 29/37
- 30. Community PostgreSQL 11 12 and Beyond Committed Proposed Already Committed - Notable Features (as of 17 March) Miscellaneous Build with Visual Studio 2017 Arrays over domains Domains over composite types Fewer superuser checks, more GRANT-based Convert documentation to DocBook XML SQL procedures with transaction control Transaction control in PL procedures User-callable SHA-2 functions pg_stat_statements 64 bit queryid Window frame clauses now full SQL:2011 support Allow external command for obtaining passphrases for SSL key files Joe Conway PGConf APAC 2018 30/37
- 31. Community PostgreSQL 11 12 and Beyond Committed Proposed Already Committed - Notable Features (as of 17 March) pgbench Allow non-ASCII characters in variable names Add approximated Zipfian-distributed random generator Add pow(), aka power(), function Improve scripting language in pgbench psql Use PSQL_PAGER in preference to PAGER if set gdesc command Variables to track success/failure of SQL Test for variable existence Joe Conway PGConf APAC 2018 31/37
- 32. Community PostgreSQL 11 12 and Beyond Committed Proposed Patches proposed Partitioning Faster partition pruning Runtime partition pruning (for prepared statements) Partition-wise aggregation Allow updating partition key Support INSERT .. ON CONFLICT Tuple routing for foreign partitions Remove redundant query checks, based on partition Foreign keys on partitioned tables Foreign key arrays Joe Conway PGConf APAC 2018 32/37
- 33. Community PostgreSQL 11 12 and Beyond Committed Proposed Patches proposed - continued Planning Aggregation push-down Convert JOIN OR-clauses into UNIONs Remove LEFT JOINs in more cases Remove useless DISTINCT clauses Indexes Incremental sort Predicate locking in Gist/Hash indexes REINDEX CONCURRENTLY Covering Indexes Joe Conway PGConf APAC 2018 33/37
- 34. Community PostgreSQL 11 12 and Beyond Committed Proposed Patches proposed - continued Administration Allow VACUUM to use more than 1G Make relation extension lock cheaper Generic WAL compression Online enabling of checksums Verify Checksums during Basebackups SQL MERGE SQL/JSON Standard (functions, JSON_TABLE, jsonpath) SQL ASSERTION Performance Just-In-Time (JIT) Compilation expressions and tuple deform ALTER TABLE ADD COLUMN fast DEFAULT Joe Conway PGConf APAC 2018 34/37
- 35. Community PostgreSQL 11 12 and Beyond Actively Being Worked Ongoing Discussions Actively Being Worked 64bit transaction IDs Parallelize more operations (VACUUM?) More improvements in partitioning (automatic creation?) Database Encryption Logical Replication conflict handling Synchronous logical replication Chained transactions Generated columns Alternate to OpenSSL (e.g. GnuTLS) Support Joe Conway PGConf APAC 2018 35/37
- 36. Community PostgreSQL 11 12 and Beyond Actively Being Worked Ongoing Discussions Ongoing Discussions Pluggable Storage Columnar Storage Global Indexes Index-Organized Tables Autonomous Transactions Cron-like Background Worker Building more on logical replication Using FDWs from parallel workers Asynchronous Append with FDWs CREATE VARIABLE Joe Conway PGConf APAC 2018 36/37
- 37. Community PostgreSQL 11 12 and Beyond Actively Being Worked Ongoing Discussions Questions? Thank You! [email protected] [email protected] Joe Conway PGConf APAC 2018 37/37