12 in 12 – A closer look at twelve or so new things in Postgres 12

12 in 12
A closer look at twelve or so new things in Postgres 12
12
2019-08-06
Basil Bourque
LinkedIn: basilbourque
© 2019 Basil Bourque

me
• Basil Bourque
• LinkedIn: basilbourque
• Decades as Software Developer
• Custom-crafted  
database-backed  
client-server apps
• Shipped iOS apps ( Objective-C )
• Micro-startups
• Got an idea? Talk to me.
Java
Vaadin
Postgres
2

• I am no expert on these
features
• I've not veriﬁed if items
have made the cut
• Still in beta, may
change
3

Postgres 12
• Beta 2 released 2019-06-20:
• Press release: https://www.postgresql.org/about/news/1949/
• Release notes: https://www.postgresql.org/docs/12/release.html
• Tip: Bundles PgAdmin 4.9, but 4.10 released 2019-07-04
⑫
4

themes
faster
more
useful
pluggable
storage
engine
💔
5

Agenda
• Pluggable table storage ("AMs")
• Collations, case- & accent-insensitive
queries
• Indexes — smaller, faster, concurrent
• Page checksums, enabled & disabled
• JIT enabled by default
• Common table expressions, inlined
• OIDs: no system column oid
• Statistics for query planner, improved
for non-uniform distributions
• Generated-value column
• Authentication: GSSAPI connections
encrypted, LDAP server discovery
• JSON path queries
• Partitioning improved
• Bonus items
6

Pluggable table storage ("AMs")
• Alternate storage engines to be
plugged into Postgres 12 or 13+
• a.k.a. Access Methods
• Postgres currently has one
particular way to structure a row
(tuple) when written to storage
(drive/disk/SSD)
• Many other structures are
conceivable
⑪⑫
7

flexible by design
• The way an AM actually stores data
is fairly unconstrained
• May or may not use Postgres’
shared buffer cache
• For crash safety, can use Postgres’
WAL, or a custom implementation.
• Current AM known as heap
becomes an example
implementation
• Txns involving different AMs must
integrate with current machinery
(xlog.c code).
• Current constraint: For modifications
or indexes, tuple must have Tuple
Identifier (TID) — a block number &
index-within-block number
8

potential beneﬁts
• Storage to suit particular kinds of data
• Ex: column-oriented versus row-oriented
• Less bloat
• Little or no VACUUM
• Faster access
9

zheap
• New AM being built by EnterpriseDB
• Reduce bloat 
Get away from always creating a
new version of a tuple on update,
to eventually be removed by
periodic vacuuming. Ditto for
deletes.
• Allow in-place updates in
common cases
• Reuse space ASAP in txn with
delete or non-in-place-update.
• Reduce write ampliﬁcation
• Avoid rewrites of heap pages
• Make possible an update touching
indexed columns without updating
every index
• Reduce tuple size
• Shrink tuple header
• Eliminate most alignment padding
10

zheap in a nutshell
• Aims to make changes in-place.
• Old row copied to an Undo Log. 
Then new row overwrites the old, in-place.
• A transaction abort does the reverse.
• Concurrent transactions can ﬁnd old row in the Undo Log.
• Cannot always be done, but often enough to be worthwhile.
• Ex: The block is full, and new row is wider.
12

future-ready
• Alternate storage engines to be plugged into
Postgres 13+
• a.k.a. Access Methods
• Postgres 12 is preparation, a heavy refactoring of
the code
• Building just the interface now 
Change all points in code that assume a storage
engine
• No beneﬁt now, just risk. (a necessary step)
13

interface
• Chapter 60. Table Access Method Interface Deﬁnition 
Postgres 12 manual
• 42 callbacks
• The core of Postgres is agnostic/ignorant as to the Access Method (storage
implementation)
• Pg knows only what is exposed by this interface
14

pluggable
• Plug in one or more implementations via: 
CREATE ACCESS METHOD TYPE TABLE 
DROP ACCESS METHOD TYPE TABLE
• Stored in pg_am system catalog
⑫
⑫ ⑪ …
15

pluggable
• Plug in one or more implementations via: 
CREATE ACCESS METHOD TYPE TABLE 
DROP ACCESS METHOD TYPE TABLE
• Stored in pg_am system catalog
• Already have in Postgres 9 & 10 & 11: 
CREATE ACCESS METHOD TYPE INDEX 
DROP ACCESS METHOD [ IF EXISTS ] [ CASCADE | RESTRICT ]
• See manual: Chapter 61. Index Access Method Interface Deﬁnition
⑪
⑫
⑫ ⑪ …
16

per table
• Specify an Access Method (AM) when creating table
• CREATE TABLE … ( … ) USING
• Currently exactly one AM, known as heap
• CREATE TABLE … USING heap ;
• Not found in PgAdmin 4.10 yet (?)
get current behavior, explicitly
19

example
CREATE TABLE public.bogus_
(
pkey_ uuid NOT NULL,
name_ text,
PRIMARY KEY (pkey_)
)
USING heap
;
ALTER TABLE public.bogus_
OWNER to postgres;
✅⑫
20

error
• ERROR: access method
"canine" does not exist
• SQL state: 42704
CREATE TABLE public.bogus_
(
pkey_ uuid NOT NULL,
name_ text,
PRIMARY KEY (pkey_)
)
USING canine
;
ALTER TABLE public.bogus_
OWNER to postgres;
21

risks
• Bugs introduced in 12 ? ( help test! )
• Confusion amongst newbies learning Postgres
• Troubled history of pluggable storage in MySQL and others
• Cultural issue: 
closed-source and/or commercial storage engines mixed with open Postgres 
( bad thing? good thing? )
22

resources
• Plugable Table Storage in
PostgreSQL - Andres Anarazel
(PgCon 2019)
• DO or UNDO - there is no VACUUM  
posting by Robert Haas
(2018-01-30)
• Postgres 12 highlight - Table Access
Methods and blackholes  
by Michaël Paquier
• zheap: a new storage format for
PostgreSQL (pgsql-hackers email
list) 
by Amit Kapila (2018-03-01)
• zheap wiki page
• zheap project page (GitHub)
23

OIDs
• Object identiﬁers (OIDs)
• Data type
• This type used as IDs internal to Postgres
• Primary keys in various system tables
25

OID problems
• Limited range
• 32-bit integer
• cluster-wide, one sequence across all objects of cluster ( all databases! )
• ill-fated origins in Postgres’ early OOP attempts
• Problematic for pluggable table storage
26

No more oid system column
• Could be added invisibly to your own tables
• CREATE TABLE … WITH OIDS  
CREATE TABLE … WITH ( OIDS = TRUE )
• default_with_oids (boolean) compatibility setting
🚫
🚫
💔
System columns
oid
tableoid
ctid
cmax
cmin
xmax
xmin
⑫
27

OID example
CREATE TABLE public.person_
(
pkey_ uuid NOT NULL DEFAULT uuid_generate_v1(),
name_ text COLLATE pg_catalog."default" NOT NULL,
phone_ text COLLATE pg_catalog."default" NOT NULL
DEFAULT ''::text,
CONSTRAINT person_pkey_ PRIMARY KEY (pkey_)
)
WITH ( OIDS = TRUE )
TABLESPACE pg_default;
ALTER TABLE public.person_ OWNER to postgres
;
28

Other system columns
• tableoid (id of this table) remains as-is
• Useful for select from inheritance hierarchies
• Join on pg_class to obtain table name
• ctid
• Tuple identiﬁer (TID) data type
• Describes physical location:  
(a) block number, (b) tuple index within block
• Changes when row updated, or VACUUM FULL
System columns
oid
tableoid
ctid
cmax
cmin
xmax
xmin
31

error in Pg 12
• ERROR: tables declared WITH OIDS are not supported
• SQL state: 0A000
CREATE TABLE foo
(
…
)
WITH ( OIDS = TRUE )…
💔
⑫
32

data loss
• Restoring from Pg 11 into Pg 12
• oid column on your tables not brought over (data gone)
• If you need this data, copy values into a new column
• Make other changes. Maybe generate an OID value??
💔
⑫
33

SELECT * surprise
• ??? 
“As a concrete consequence, oid will now be visible when running  
select * from the catalogs that have OIDs, as well as when querying
information_schema.columns, or with d inside psql.”
• Not in my experience: oid column dropped when restoring a db
• Perhaps they mean when upgrading-in-place with pg_upgrade tool
💔
⑫
34

resources
• OIDs demoted to normal columns: a glance at the past 
by Daniel Vérité (2019-04-24)
• System Columns 
Postgres 11 manual
35

indexing in the background
• ⑪… CREATE INDEX CONCURRENTLY
• Runs in the background
• Allows concurrent inserts, updates, or deletes on the table
• Avoids a table-write lock
37

reindex
• Need: Remove bloat, or ﬁx corrupted index, and rebuild for feature change
• Exists in Postgres, many versions
• Exclusive lock on table
• no queries, wait for REINDEX to ﬁnish
• ⑫ Progress reporting ( for REINDEX & CREATE INDEX ) 
pg_stat_progress_create_index system view
38

⑫ REINDEX CONCURRENTLY
• Create Phase
• New index
• Named with sufﬁx _ccnew
• Build index in background (may
take a while)
• When built, update index for
parallel activity
• Swap Phase
• Rename new index with old
• Switch dependencies from old to
new
• Invalidate old index
• Drop old index
39

notes
• Each step = txn
• Works on Toast indexes too
• Also in client app: reindexdb --concurrently
40

caveats
• If interrupted, invalid index remains
• Takes up space
• Designed so you can drop it
• Does not work in catalog tables
41

Older tools
• pg_repack (formerly pg_reorg)
• Using new command is cheaper
42

resources
• Postgres 12 highlight - REINDEX CONCURRENTLY  
posting by Michaël Paquier, 2019-04-22
• Waiting for PostgreSQL 12 – REINDEX CONCURRENTLY  
posting by depesz (Hubert Lubaczewski), 2019-03-29
43

index improvements
• B-tree
• Improve performance and space
utilization when many duplicates
• Multi-column smaller
• Insertions faster, with less locking
overhead
• GIST
• supports INCLUDE columns
• Improve the performance of
vacuum scans
• Reduce the WAL write overhead of
GiST, GIN and SP-GiST index
creation
• Nearest-neighbor (KNN) searches
of SP-GiST indexes
44

REINDEX fail ?
• Quote from Release Notes, Beta 2: 
The maximum btree index length is now reduced by eight bytes; a REINDEX
operation could potentially fail.
• I could ﬁnd no further info

other index features
• Partition indexes appear in pg_indexes system view

pg_upgrade
• Dump/Restore omits index content.
• All indexes rebuilt automatically as part of reconstructing new database
• pg_upgrade upgrades between major versions of Postgres while leaving data
ﬁles in place
• Tip: If using pg_upgrade, re-index

electric motorcycle
Zero Motorcycles
• A single moving part
• no messy ﬂuids
• no clutch
• no transmission
48

Common Table Expression (CTE)
• Basically, a different syntax for nested queries
• Standard-SQL WITH … AS clause, including WITH RECURSIVE
• For more readable queries, and often faster queries
• See manual: 7.8. WITH Queries (Common Table Expressions)
49

CTE example (from manual)
WITH regional_sales AS (
SELECT region, SUM(amount) AS total_sales
FROM orders
GROUP BY region
), top_regions AS (
SELECT region
FROM regional_sales
WHERE total_sales > (SELECT SUM(total_sales)/10 FROM regional_sales)
)
SELECT region,
product,
SUM(quantity) AS product_units,
SUM(amount) AS product_sales
FROM orders
WHERE region IN (SELECT region FROM top_regions)
GROUP BY region, product;
virtual
tables
50

CTE — old way
• Previously, treated as “optimization fence” (Postgres planner ignores)
• CTE is processed ﬁrst — WHERE clauses applied later
• Restrictions from outer query not pushed into the WITH virtual-table
• So, virtual-table is materialized  
( results written to a temporary table, literally or conceptually )
51

CTE — old way
• Previously, treated as “optimization fence” (Postgres planner ignores)
• CTE is processed ﬁrst — WHERE clauses applied later
• Restrictions from outer query not pushed into the WITH virtual-table
• So, virtual-table is materialized  
( results written to a temporary table, literally or conceptually )
• Makes sense for:
• Side-effects ( INSERT / UPDATE / DELETE )
• Recursive
• Outer-query calls the WITH query more than once
52

background for example
• pg_class = list of all deﬁned tables &
anything else with columns
• relkind = what kind of table-thingy
• r = ordinary table, i = index, S =
sequence, t = TOAST table, v = view, m =
materialized view, c = composite type, f =
foreign table, p = partitioned table, I =
partitioned index
53

⑪ example (by depesz)
WITH x AS (
SELECT relkind, COUNT(*)  
FROM pg_class  
GROUP BY relkind
)
SELECT *  
FROM x  
WHERE relkind = 'r'
;
returns 4 rows
relkind:
r
v
i
t
restriction to 'r' ignored by WITH query
⑪
54

CTE — new way
• For (a) side-effect-free and b) non-recursive “WITH” virtual tables:
• Remove the “optimization fence”
• ⑫ Not materialized
• ⑫ Inline with the outer query, if called only once
• User-override
• ⑫ MATERIALIZED 
⑫ NOT MATERIALIZED
💔 Performance of
existing WITH-AS
queries changes
by default in 12

⑫ example (by depesz)
WITH x AS (
FROM pg_class  
GROUP BY relkind
)
SELECT *  
FROM x  
WHERE relkind = 'r'
;
returns 1 rows
relkind:
r
⑫
restriction to 'r' applied inside WITH query

WITH x AS NOT MATERIALIZED (
FROM pg_class  
GROUP BY relkind
)
SELECT *  
FROM x  
WHERE relkind = 'r'
;
returns 1 rows
relkind:
r
restriction to 'r' applied inside WITH query
⑫

WITH x AS MATERIALIZED (
FROM pg_class  
GROUP BY relkind
)
SELECT *  
FROM x  
WHERE relkind = 'r'
;
add
MATERIALIZED
to get old v11
behavior
⑫

resources
• Waiting for PostgreSQL 12 – Allow user control of CTE materialization, and
change the default behavior 
by depesz (Hubert Lubaczewski) (2019-02-19)
• Postgres 12 highlight - WITH clause and materialization 
by Michaël Paquier (2019-02-28)

electric scooter
Vespa Elettrica

what is partitioning?
• Splitting what is logically one very large table into smaller physical pieces
• Rule of thumb: table size > RAM
• Serious pros & cons

beneﬁts of partitioning
• Query performance
• Most-accessed rows in few partitions
• Most-accessed parts of index in RAM
• Accessing much of a partition done sequentially rather than index & whole
table
• Seldom-used data in cheaper storage
• Bulk load/delete via attaching/detaching partition

⑨ roll-your-own partitioning
From wiki.postgresql.org: New in postgres 10 > Native Partitioning
father table
daughter table

⑩+ declarative partitioning
CREATE TABLE padre (
id INTEGER GENERATED ALWAYS AS IDENTITY ,
pais INTEGER ,
fch_creado TIMESTAMPTZ NOT NULL DEFAULT NOW()
) PARTITION BY RANGE( fch_creado )
;
CREATE TABLE hija_2017 PARTITION OF padre
FOR VALUES FROM ( '2017-01-01 00:00Z' )
TO ( '2018-01-01 00:00Z' )
;
CREATE TABLE hija_2019 PARTITION OF padre
FOR VALUES FROM ( '2019-01-01 00:00Z' )
TO ( '2020-01-01 00:00Z' )
;

⑩+ declarative partitioning
• CREATE TABLE parent_ PARTITION BY RANGE/LIST …
• CREATE TABLE child_ PARTITION OF parent_ FOR VALUES …

⑫ new with partitioning
• ⑫ Foreign keys referencing partitioned table as child
• So… now have partitioned OLAP (see Roybal)
• ⑫ Dynamic expressions as partition bounds
• Formerly evaluated at table-creation-time.

⑫ new with partitioning
• ⑫ Several performance improvements.
• Ex: Faster COPY into partitioned tables
• ⑫ Tablespace inherited from parent table
• ALTER TABLE … ATTACH PARTITION
• ⑫ No more exclusive lock, so add partition at runtime

⑫ Reporting - psql
• ⑫ New + option to delineate per-partition info: d+ parent_tbl
• ⑫ New P option to list partitioned tables and indexes: dP

⑫ Reporting - commands
• 3 functions:
• ⑫ SELECT pg_partition_tree( 'parent_tbl' ) ;
• ⑫ SELECT pg_partition_ancestors( 'child_tbl' ) ;
• ⑫ SELECT pg_partition_root( 'child_tbl' ) ;
• System view pg_indexes includes partitioned indexes

caveats
• Many caveats remain with partitioning
• Use only as a last-resort, for a pressing reason
• Study thoroughly before proceeding
• Steadily improving, so pay attention to each major release

resources
• Partitioning enhancements in PostgreSQL 12  
post by Kirk Roybal 2019-07-15
• PostgreSQL 12: New partition reporting functions 
post by Daniel Westermann 2019-06-02
• 5.11. Table Partitioning 
maunal

electric kick scooters
Lime (rental)
Glion

collation
• Collation: Rules by which we decide the order of items
• French in France sorts differently than French in Canada

collation
• Collation: Rules by which we decide the order of items
• French in France sorts differently than French in Canada
• Other variants too.
• Ex: Phonebook in German.
• Uppercase ﬁrst or last (you can force this, with option)
• Numeric: A-12n, A-21, A-123, B-100 (even non-Arabic digits)
• Emoji sort ( happy < sad, human faces < cat faces ) (doc)
'it-u-kf-upper'
'it@colCaseFirst=upper'

collation provider
• libc = C standard library (POSIX)
• glibc = Gnu implementation by Free
Software Foundation, for Linux
• BSD libc = BSD impl. for FreeBSD,
NetBSD, OpenBSD, & macOS
• ICU = International Components for
Unicode by the Unicode Consortium
• Deﬁnes collation, and much more

collation provider
• ⑩+ Choice of provider: CREATE COLLATION
• Database-wide default: Limited to libc (Eventually ICU to be offered)
• System catalog pg_collation column collprovider ( d , c , i ) ( doc )
• ICU = International Components for
Unicode by the Unicode Consortium
• Deﬁnes collation, and much more
• libc = C standard library (POSIX)
• glibc = Gnu implementation by Free
Software Foundation, for Linux
• BSD libc = BSD impl. for FreeBSD,
NetBSD, OpenBSD, & macOS

libc
glibc
C Standard Library
$ %
ICU
BSD libc

using collations in Postgres
• CREATE COLLATION c1 (  
provider = libc ,  
locale = 'it_IT.utf8'  
) ;
• CREATE TABLE t1 ( 
id INTEGER PRIMARY KEY , 
x TEXT COLLATE c1 
) ;
Italian language
Italy culture
UTF-8 encoding

using collations in Postgres
locale = 'it_IT.utf8'  
) ;
• CREATE TABLE t1 ( 
id INTEGER PRIMARY KEY , 
x TEXT COLLATE c1 
) ;
lc_collate = 'it_IT' , 
lc_ctype = 'utf8'  
) ;

ignoring case & accents
• ⑫ String-equality smarter:  
comparisons agnostic on case & accents
(diacriticals)
• ICU-based collations can report string
equality when not byte-wise-equal.
• International Components for Unicode
(ICU) added in Pg 10  
(alternative to relying on OS-supplied
internationalization library) 
(see Peter Eisentraut post)

⑪⑩… case-insensitive sort
• Roll-your own:
• SELECT x from t1 WHERE lower( x ) = lower( 'foo' ) ;
• Index on this expression (see doc)
• citext extension (see doc)
• Caveat: lower() does not work in all language cases (ex: Greek)

⑫ non-deterministic collation
• CREATE COLLATION case_insensitive ( 
provider = icu , 
locale = 'und-u-ks-level2' , 
deterministic = false 
) ;
• CREATE COLLATION ignore_accents ( 
provider = icu , 
locale = 'und-u-ks-level1-kc-true' , 
deterministic = false 
) ;
'abc' = 'ABC'
'resumé' = 'resume'
To ignore both: 
'und-u-ks-level1'

caveats of deterministic = false
• Only in ICU ( not libc/glibc )
• Some performance hit
• Hashing changed to hash on sort-key (weighted numbering codes) 
rather than on underlying bits-and-bytes
• Pattern-matching does not work (logic uses entire string, not characters)
• No LIKE
• No regex
solution: swap collation 
SELECT * FROM t1  
ORDER BY x COLLATE …

statistics
• Statistics (used by planner) got smarter
• ⑫ Keeps stats on the collation deﬁned for each column  
(rather than ⑪ default collation assumed for all stats)
• Better optimizer behavior for columns with non-default collations

Linux alert: glibc changing !
See 2019-05 talk by
Peter Eisentrout
Problem OSes:
RHEL 8
Debian testing
Ubuntu 18.10
Fedora 29
glibc version 2.28,
released 2018-08-01

12 in 12 – A closer look at twelve or so new things in Postgres 12

eﬀects of built index not matching
• Query using index scan could fail to ﬁnd data that is indeed present
• Update could insert duplicate data that should be disallowed
• In partitioned table, a query could look in the wrong partition and an update
could write to the wrong partition

issues with glibc change
• Not the ﬁrst such change, but the biggest
• Affects everybody (on Linux)
• Any data with space, period, or hyphen
• Sorting happens in a variety of contexts, including for user output,
merge joins, B-tree indexes, and range partitions
• Collation deﬁnitions in glibc are not versioned ! (ICU is versioned)
PostgreSQL currently
has no way to detect
an incompatible glibc
update

workarounds
• Workaround: amcheck extension (doc)
• Verify the logical consistency of the structure of relations
• Verify … for B-Tree indexes on text, index tuples should be in collated
lexical order
• Workaround: REINDEX after OS update text, varchar, char, & citext
• Workaround: Stick with Long-Term Support (LTS) releases of your OS
• See Postgres wiki page, Locale data changes

not aﬀected
• Databases or columns using C or POSIX locales
• Columns collated with ICU provider

resources
• More robust collations with ICU support in PostgreSQL 10  
post by Peter Eisentraut 2017-05-22
• Collations: Introduction, Features, Problems 
video lecture by Peter Eisentraut 2019-07-12
• Unicode Collation Algorithm (UTS # 10) by Unicode Consortium 
Locale (naming etc.) ICU User Guide 
Demo (list of 209 languages, 501 regions & variants)

generated columns
• a.k.a. “calculated columns”, “virtual columns”, or “generated columns”
• Generate columns based on data in other columns of the same table
• Value generated at time of INSERT / UPDATE
• Physically stored with rest of table data
• Can be indexed
• Read-only, of course.
• Deﬁned in SQL:2003 standard. See SQL:2003 Has Been Published

syntax
• Similar to the replacement of SERIAL: 
GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY[ ( seq_option ) ]
• ⑫  
GENERATED ALWAYS AS ( generation_expression ) STORED

example
CREATE TABLE people (
...,
height_cm numeric,
height_in numeric
);

example
CREATE TABLE people (
...,
height_cm numeric,
height_in numeric GENERATED ALWAYS AS (height_cm / 2.54) STORED
);

uses
• Likely use-cases results in de-normalized data structure, but may be
necessary for performance
• Converted data such as math calculation
• Extended cost for invoice line-item: ( quantity * item_cost )
• Concatenated text
• Full address = street1 + street2 + city + state + zip
• Generate JSON / XML

caveats
• Based on columns of the same table (not related tables)
• Not allowed for partitioning (cannot be part of a partition key)
• Data always written to row, taking space in storage
• Future feature might offer VIRTUAL for values calculated on-the-ﬂy without
storage

caveats
• Based on columns of the same table (not related tables)
• Not allowed for partitioning (cannot be part of a partition key)
• Data always written to row, taking space in storage
• Future feature might offer VIRTUAL for values calculated on-the-ﬂy without
storage
• Single-generation deep (use base column, not another generated column)
• There is no GENERATED BY DEFAULT (you cannot override value)
• Cannot access gen-col in BEFORE trigger
• Functions must be immutable

immutable functions
• 3 kinds of functions: VOLATILE, STABLE, IMMUTABLE ( CREATE FUNCTION )
• Immutable = cannot modify the database, and is guaranteed to return the
same results given the same arguments forever
• Example of non-Immutable: random() or CURRENT_TIMESTAMP
• Example: 
CREATE TABLE … 
gen_ TEXT GENERATED ALWAYS AS ( CURRENT_TIMESTAMP :: TEXT
) STORED , …
• ERROR: generation expression is not immutable 
SQL state: 42P17

resources
• See manual: 5.3. Generated Columns
• Generated columns in PostgreSQL 12  
by Kirk Roybal (2019-07-05)
• PostgreSQL 12: generated columns  
by Daniel Westermann (2019-04-06)

Amtrak
Union Station
Portland OR
Coast Starlight
Cascades

data checksums
• checksum function =  
feed some data, get back a calculated value
• Added in Postgres 9.3 (see wiki) (using CRC32 reduced to 16 bits)
• Maintain a checksum calculation on data pages
• Report corruption of data in storage (on disk/drive/SSD)
• Has some impact on performance
• Theoretically unnecessary with  
error-detecting/correcting RAID or ZFS pool
image via J. Yang / NSF

new with checksums
• ⑪ Previously only set when creating database: initdb … --data-
checksums
• ⑫ Now turned on/off for existing cluster
• Your cluster must be shutdown.
• Switch affects all databases across the cluster.
• CAUTION: Must be done across your replication chain, or recreate

pg_checksums
• ⑫ pg_verify_checksums now called pg_checksums
• Veriﬁes current checksum status: Reports 0 if none, non-zero if any
• ⑫ Enables/disables checksum feature
• ⑫ Reports progress: --progress option
• ⑫ Counter of checksum failures in pg_stat_database

resources
• Postgres 12 highlight - pg_checksums  
by Michaël Paquier

What is JIT?
just-in-time (JIT) compilation is a way of executing computer code that
involves compilation during execution of a program – at run time –
rather than prior to execution

expression evaluator
… WHERE a.col < 10 AND a.another = 3 …
a
col
another

Postgres generates bytecode
From talk by
Andres Freund
at PgCon 2018

bytecode
From talk by
Andres Freund
at PgCon 2018

JIT = Just-In-Time compilation
• Rather than interpreting that bytecode, compile to native code
• See manual: Chapter 31. Just-in-Time Compilation (JIT)
• ⑫ Enabled by default (if built)
⑫⑪

JIT
• Uses:
• Expression evaluation jit = on / off 
( WHERE clause, target lists, aggregates & grouping, projections )
• Tuple deforming jit_tuple_deforming = on / off 
( transforming an on-disk / in-memory tuple to be more accessible ) ( seq.
scan )
• Accelerated by creating a function speciﬁc to the table layout and the
number of columns to be extracted. 
⑫⑪

example of JIT
SELECT COUNT(*) FROM tbl WHERE ( x + y ) > 20 ;
Early pre-release results showed:
- WHERE clause went from taking 56% of execution time to 6%
- Query execution 2x faster
From PgCon 2017 talk:  
JIT-Compiling SQL Queries in PostgreSQL Using LLVM

beneﬁts of JIT
• Primarily for long-running CPU-bound queries
• Postgres makes a choice of when to JIT versus when to run interpreted
• Can inline bodies of small functions ( of types C and internal )
• Optimizing generated code ( LLVM can optimize very well )
• Pluggable
• LLVM is currently the only implementation of JIT-within-Postgres

LLVM
• Postgres’ JIT relies on LLVM technology
• "collection of modular and reusable compiler and toolchain technologies"
• Frontend for any programming language (both static & dynamic) 
(C, C++, Objective-C, Swift, Rust, Kotlin, Haskell, Python, Ruby, Crystal,
Xojo, Delphi, Fortran, Dylan, Lisp, … )
• Intermediate representation (IR) “bitcode” ( like Java bytecode ) 
Low-level programming language similar to assembly
• Backend for any instruction set architecture

cases
• 👍
• CPU bound
• OLAP
• ✅ CPU-intensive aggregates
• Wide relations
• Tuple-deforming, especially
many NON-NULL columns
• ✅ Sequential scans
• 👎
• sub-second query time (check
pg_stat_activity & explain-
analyze)
• I/O bound
• OLTP (client-server to-&-fro)
• ❌ Index scans (ex: nav thru b-tree)
• ❌ Sorting (not yet)
• Joins

caveats
• Build Postgres with --with-llvm option
• Not available on Microsoft Windows (currently)
• Entire query JITed
• Not optimized to JIT only the parts that most beneﬁt
• No caching (yet)
• Ex: Would be great to re-use across runs of a prepared-statement
• Not parallelized (yet): Start interpreted, JIT in b/g, then switch to compiled

optimal
query interpreted
time elapsed, shorter = better
JIT query JITed

too expensive
query interpreted
JIT query JITed
JIT query JITed
time elapsed, shorter = better

Query Planner - costing
• JIT is not always invoked
• Cost of revving up the compiler
may not be worth the bother
• Simple planner cost thresholds,
naïve implementation
• Could possibly show worse
performance with JIT (but not
likely)
• Planner cost constants
• jit_above_cost (100,000)
• jit_optimize_above_cost
(500,000)
• jit_inline_above_cost
(500,000)
• -1 = disabled

JIT commands
• Was my Postgres built to have LLVM?
• pg_config --configure
• Manual: … find out with what options a binary package was built.
• Is JIT enabled currently?
• show jit ;
• Available in this session?
• SELECT pg_jit_available() ;

resources
• Postgres manual, Chapter 32. Just-in-Time Compilation (JIT)
• Just in time compilation in PostgreSQL 
Lecture video, by Andres Freund. Higher overview. Late 2018.
• The State of Postgres JIT 2018 Edition -- PGCon 2018  
Lecture video, by Andres Freund. Very technical. Mid-2018.
• PostgreSQL 11 and Just In Time Compilation of Queries  
Posting about benchmarks, by Dimitry Fontaine, Citus Data, 2018-09-11

Greens – It’s what’s for breakfast
Beet
family
Chard
Beet
greens
Cabbage
family
Kale
(ﬂat, curly,
dino)
(green,
purple)

JSON Path
• Yes, JSON is getting an XPath-like query feature  
(in the needless quest to re-invent XML all over again)
• A query language to select elements within a JSON document
• Support deﬁned in SQL:2016 speciﬁcation
• ⑫ jsonpath data type (doc), works with jsonb type in Postgres
• Postgres 12 highlight - SQL/JSON path  
by Michaël Paquier 2019-06-21
• Manual: 9.15. JSON Functions, Operators, and Expressions

authentication
• ⑪ GSS-API (Kerberos)
• ⑫ Client and server-side encryption
• ⑫ pg_stat_gssapi view (row per connection)
• LDAP
• discovery of LDAP server via DNS SRV records 
(need not specify ldapserver)

authentication denial-of-service ﬁxed
• Bug # 15,182
• ⑫ ﬁxed
• Syndrome discovered by Lloyd Albin
• Deadlock occurs in certain circumstances, preventing further logins
• Requesting a lock on tables for which the user lacks permission
• See email thread

pg_dumpall …except for…
• The pg_dumpall tool dumps all the databases of a cluster
• Now you can exclude one or more
• --exclude-database=PATTERN  
where pattern = a name, or asterisk wildcard.  
Ex: db* or *db*
• See post by Mouhamadou Diaw. (2019-03-03)
• By the way: pg_dump option --on-conflict-do-nothing to avoid conﬂict
failures during restore
• pg_dump option --rows-per-insert

-socketdir for pg_upgrade
• When upgrading into a directory (folder) with a very long pathname
• See post by Mouhamadou Diaw. (2018-12-08)

auto-vacuum
• Setting autovacuum_vacuum_cost_delay
• ⑫ Takes a ﬂoating-point rather than integer
• Fraction means microseconds, whole means milliseconds (?)
• ⑫ Default changes from 20 milliseconds to 2.

geometric
• Functions
• ⑫ Refactored to produce better results than are currently available.
• Types
• point, box, lseg, line, path, polygon, and circle
• ⑫ Restructured to handle NaN, underﬂow, overﬂow and division-by-zero.

parallelization
• Parallelized queries allowed in SERIALIZABLE isolation mode
• More efﬁcient parallel restores with pg_restore
• Decouple order of parallel pg_dump from order in pg_restore

other resources
• Major Features: Postgres 12  
by Bruce Momjian
• Manual: Appendix E. Release
Notes
• PostgreSQL 12 Open Items 
working wiki page
• DBA Stack Exchange 
https://DBA.StackExchange.com/
• Stack Overﬂow 
https://www.StackOverﬂow.com/

photo by Robert Bernier
2019
O’REILLY®
Open Source Award
Lifetime
Achievement Award
PostgreSQL

« ﬁn »
• … and much more
• Summary
• Release Notes
• Feature Matrix (now interactive!)
Basil Bourque
LinkedIn: basilbourque

12 in 12 – A closer look at twelve or so new things in Postgres 12

More Related Content

What's hot (20)

Similar to 12 in 12 – A closer look at twelve or so new things in Postgres 12 (20)

Recently uploaded (20)

12 in 12 – A closer look at twelve or so new things in Postgres 12