Scaling postgres

/
Scaling Postgres
Denish Patel
Database Architect
https://twitter.com/DenishPatel
Wednesday, September 18, 13

OmniTI
1

OmniTI
1
• Helping customers
navigate explosive growth
with technology.

OmniTI
1
with technology.
100MM+ users
$1B+ gross online sales

OmniTI
1
with technology.
100MM+ users
$1B+ gross online sales
Open and closed source thought leaders,
experts and authors

Talk Outline
2

Talk Outline
• Scalability
• Database Scaling needs, costs, methods
• Scaling Postgres
• Vertically
• Horizontally
• Obstacles to Scalability
• Beyond Postgres
2

What is Scalability?
3

What is Scalability?
3
A service is said to be scalable if when we
increase the resources in a system, it results in
increased performance in a manner proportional
to resources added.

Why to scale databases?
4

• Support a higher volume of users
4

• Provide better performance for existing users
4

• Store a larger volume of data
4

• Improve system availability
4

• Improve system availability
• Geographic dispersion
4

Why is database scalability so hard?
5

• Search
5

• Search
• Concurrency
5

• Search
• Concurrency
• Consistency
5

• Search
• Concurrency
• Consistency
• Speed
5

• Search
• Concurrency
• Consistency
• Speed
• Cost
5

• Search
• Concurrency
• Consistency
• Speed
• Cost
• Cost of hardware
5

• Search
• Concurrency
• Consistency
• Speed
• Cost
• Cost deployment effort
5

• Search
• Concurrency
• Consistency
• Speed
• Cost
• Cost deployment effort
• Cost Ongoing maintenance
5

Scaling Needs & Methods
6

• Data growth
6

• Data growth
• Read requests
6

• Data growth
• Read requests
• Write requests
6

• Data growth
• Read requests
• Write requests
• Vertical Scaling
• Horizontal Scaling
6

Vertical Scaling (Scale up)
7

• Pros
7

• Pros
• Simple to implement
7

• Pros
• Ease of maintenance
7

• Pros
• Cons
7

• Pros
• Cons
• Sometimes SPOFs
7

Horizontal Scaling (Scale Out)
8

• Pros
8

• Pros
• Cheaper in hardware cost
8

• Pros
• Flexibility
8

• Pros
• Flexibility
• Higher fault tolerance
8

• Pros
• Flexibility
• Cons
8

• Pros
• Flexibility
• Cons
• Complex to implement
8

• Pros
• Flexibility
• Cons
• Expensive to maintain
8

• Pros
• Flexibility
• Cons
• Bigger footprint in the Data Center
8

• Pros
• Flexibility
• Cons
• Bigger footprint in the Data Center
• No built in support in databases
8

Spec’ing Hardware
9

Spec’ing Hardware
• CPU
9

Spec’ing Hardware
• CPU
• 8+ cores
9

Spec’ing Hardware
• CPU
• 8+ cores
• RAM
9

Spec’ing Hardware
• CPU
• 8+ cores
• RAM
• 64GB+
9

Spec’ing Hardware
• CPU
• 8+ cores
• RAM
• 64GB+
• Disks
9

Spec’ing Hardware
• CPU
• 8+ cores
• RAM
• 64GB+
• Disks
• SSDs
9

Spec’ing Hardware
• CPU
• 8+ cores
• RAM
• 64GB+
• Disks
• SSDs
• RAID 10
9

Spec’ing Hardware
• CPU
• 8+ cores
• RAM
• 64GB+
• Disks
• SSDs
• RAID 10
• Network
9

Spec’ing Hardware
• CPU
• 8+ cores
• RAM
• 64GB+
• Disks
• SSDs
• RAID 10
• Network
• min Gigbit, 10Gigbit
9

10

Tune Postgres/memory parameters
10

• shared_buffers
10

• shared_buffers
• effective_cache_size
10

• shared_buffers
• checkpoint_completion_target
10

• shared_buffers
• checkpoint_segments
10

• shared_buffers
• max_connections
10

• shared_buffers
• max_connections
• work_mem
10

• shared_buffers
• max_connections
• work_mem
• maintenance_work_mem
10

• shared_buffers
• max_connections
• work_mem
• maintenance_work_mem
http://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Server
10

11

Tune Postgres/logging parameters
11

• logging_collector => 'on'
11

• log_destination => 'stderr'
11

• log_filename => 'postgresql-%Y-%m-%d_%H%M%S.log'
11

• log_line_prefix => '%m [%r] [%p]: [%l-1] user=%u,db=%d,e=%e '
11

• log_min_duration_statement => 1000ms
11

• log_min_duration_statement => 1000ms
• log_autovacuum_min_duration => '0'
11

12

• log_lock_waits => 'on'
12

• log_temp_files => '0'
12

• log_checkpoints => 'on'
12

• log_connections => 'on'
12

• log_disconnections => 'on'
12

• log_min_error_statement => 'warning'
12

• log_min_messages => 'warning'
12

• log_min_messages => 'warning'
• log_statement => 'ddl'
12

13

Optimize Queries/report slow queries
13

14

Optimize Queries/Extensive monitoring
14

15

Optimize Queries /Explain Analyze
15

Optimize Queries /Explain Analyze
explain (analyze,buffers) select col1,col2 from demo_ios where col2 between 0.01 and 0.02;
QUERY PLAN
-----------------------------------------------------------------------------------------------------------------------------------------------
Index Only Scan using idx_demo_ios on demo_ios (cost=0.00..35330.93 rows=993633 width=16) (actual time=58.100..3250.589
rows=1000392 loops=1)
Index Cond: ((col2 >= 0.01::double precision) AND (col2 <= 0.02::double precision))
Heap Fetches: 0
Buffers: shared hit=923073 read=3848
Total runtime: 4297.405 ms
15

16

Optimize Queries /track functions
16

• track_functions = pl # none, pl, all
16

• reload online
16

• reload online
• select * from pg_stat_user_functions;
16

17

Partitioning
17

Partitioning
• As table size grows, queries eventually slows down, even with indexing
17

Partitioning
• Allows data added, removed and queried fast
17

Partitioning
• Partitioning pruning queries
17

Partitioning
• Partitioning pruning queries
• Manage partitions
17

18

Partitioning /Postgres
18

• Postgres partitioning
18

• Trigger based
18

• Trigger based
• Rule based
18

• Trigger based
• Rule based
• Lack of built-in Postgres partition management
18

• Trigger based
• Rule based
• Postgres partition management extension: pg_partman
18

• Trigger based
• Rule based
• http://pgxn.org/dist/pg_partman/doc/pg_partman.html
18

• Trigger based
• Rule based
• Tomorrow’s session at 11:30AM (When Postgres can’t....)
18

19

Partitioning /functional
19

Configuration Data Transaction data Session data
Configuration Tools Reporting Tools Monitoring Tools
Web Applications Other Applications
19

• Partition data based on functionality
19

• Separate Postgres clusters
19

• Start with Separate schemas
19

• No relationship between data
19

• Help to spread the load across server
19

• Help to spread the load across server
• Less complex compare to sharding!
19

20

pgbouncer
20

pgbouncer
• A lightweight connection pooler
20

pgbouncer
• Helps to reduce # of newly created connections on DB server
20

pgbouncer
• Abstracts DBs from App
20

pgbouncer
• Helps to instrument smooth and easy failover
20

pgbouncer
• Connection pooling Options
20

pgbouncer
• Session, Transaction, Statement pooling options
20

pgbouncer
• Session, Transaction, Statement pooling options
• Beware! Transaction pooling doesn’t support prepared transactions
20

21

Caching
21

Caching
• Memcached
21

Caching
• Memcached
• Open source, High-performance distributed memory object
caching system
21

Caching
• Memcached
caching system
• Speeds up dynamic web applications by alleviating database load.
21

Caching
• Memcached
caching system
• An in-memory key-value store for small chunks of arbitrary data
21

Caching
• Memcached
caching system
• Redis
21

Caching
• Memcached
caching system
• Redis
• Open source, advanced key-value store.
21

Caching
• Memcached
caching system
• Redis
• Open source, advanced key-value store.
• Works with an in-memory & persistent dataset
21

22

Replication /built-in
22

• Cluster Level Replication (Binary)
22

• Streaming Replication
22

• WAL-only replication
22

• Hybrid replication
22

Master
22

Master
Failover• Cluster Level Replication (Binary)
22

Master
Failover
Read
Salve 1
22

Master
Failover
Read
Salve 1
Streaming Replication
22

Master
Failover
Read
Salve 1
PITR!
22

Master
Failover
Read
Salve 1
PITR!
PITR!
22

Master
Failover
Read
Salve 1
PITR!
PITR!
PITR!
22

• Pros:
Master
Failover
Read
Salve 1
PITR!
PITR!
PITR!
22

• Pros:
• Built-in
Master
Failover
Read
Salve 1
PITR!
PITR!
PITR!
22

• Pros:
• Built-in
• Allows to open replicated database in read-only mode
Master
Failover
Read
Salve 1
PITR!
PITR!
PITR!
22

• Pros:
• Built-in
• Cons:
Master
Failover
Read
Salve 1
PITR!
PITR!
PITR!
22

• Pros:
• Built-in
• Cons:
• All or none
Master
Failover
Read
Salve 1
PITR!
PITR!
PITR!
22

• Pros:
• Built-in
• Cons:
• All or none
• Doesn’t allow write on replicated database
Master
Failover
Read
Salve 1
PITR!
PITR!
PITR!
22

• Pros:
• Built-in
• Cons:
• All or none
• Doesn’t work across major version
Master
Failover
Read
Salve 1
PITR!
PITR!
PITR!
22

• Pros:
• Built-in
• Cons:
• All or none
• Postgres 9.2 (primary) does NOT replicate to 9.3 (secondary)
Master
Failover
Read
Salve 1
PITR!
PITR!
PITR!
22

• Pros:
• Built-in
• Cons:
• All or none
• Postgres 9.2.1 (primary) can replicate to 9.2.4 (secondary)
Master
Failover
Read
Salve 1
PITR!
PITR!
PITR!
22

• Pros:
• Built-in
• Cons:
• All or none
• Postgres 9.2.1 (primary) can replicate to 9.2.4 (secondary)
Master
Failover
Read
Salve 1
PITR!
PITR!
PITR!
https://wiki.postgresql.org/wiki/Binary_Replication_Tools
22

23

Read
Salve 1
23

Read
Salve 1
DW
System
23

Read
Salve 1
DW
System
postgres_fdw
23

• postgres_fdw
Read
Salve 1
DW
System
postgres_fdw
23

• postgres_fdw
• Postgres 9.3 feature
Read
Salve 1
DW
System
postgres_fdw
23

• postgres_fdw
• Allows to access data stored in external PostgreSQL
servers
Read
Salve 1
DW
System
postgres_fdw
23

• postgres_fdw
servers
• cross version queries
Read
Salve 1
DW
System
postgres_fdw
23

• postgres_fdw
servers
• Postgres 9.3 could query Postgres 9.1
Read
Salve 1
DW
System
postgres_fdw
23

• postgres_fdw
servers
• Application
Read
Salve 1
DW
System
postgres_fdw
23

• postgres_fdw
servers
• Application
• Run query remotely on slave db
Read
Salve 1
DW
System
postgres_fdw
23

• postgres_fdw
servers
• Application
• Run query remotely on slave db
• Data warehouse data refreshes
Read
Salve 1
DW
System
postgres_fdw
23

24

Replication /third-party tools
24

• Table level Replication Tools (Trigger based)
24

• Slony
24

• Slony
• Bucardo
24

• Slony
• Bucardo
• Mimeo: http://pgxn.org/dist/mimeo/
24

• Slony
• Bucardo
• Pros:
24

• Slony
• Bucardo
• Pros:
• Allows to open replicated database in read-
write mode
24

• Slony
• Bucardo
• Pros:
write mode
• Allows table/database level replication
24

• Slony
• Bucardo
• Pros:
write mode
• Allows rolling upgrade
24

• Slony
• Bucardo
• Pros:
write mode
• cross version replication is allowed
24

• Slony
• Bucardo
• Pros:
write mode
• Multi-master replication
24

• Slony
• Bucardo
• Pros:
write mode
• Cons:
24

• Slony
• Bucardo
• Pros:
write mode
• Cons:
• Complicated to setup
24

• Slony
• Bucardo
• Pros:
write mode
• Cons:
• Complicated to setup
• Unknown territory
24

25

Sharding
25

Sharding
• Sharding is the process of splitting up your data so it resides in
different tables or often different physical databases.
25

Sharding
• Application aware sharding
25

Sharding
• Application aware sharding
• Application transparent sharding
25

26

Application aware sharding
26

• http://instagram-engineering.tumblr.com/
post/10853187575/sharding-ids-at-
instagram
26

instagram
• Postgres allows “logical” shards through
Schema
26

instagram
• Postgres allows “logical” shards through
Schema
• Easy to move to “physical” shard later
26

27

Application transparent sharding
27

28

Sharding Challenges
28

Sharding Challenges
• Reliability
28

Sharding Challenges
• Reliability
• Distributed queries
28

Sharding Challenges
• Reliability
• Cross-shard join
28

Sharding Challenges
• Reliability
• Auto-increment key management
28

Sharding Challenges
• Reliability
• Choosing shard key
28

Sharding Challenges
• Reliability
• Choosing shard key
• Shard schemes
28

29

Obstacles for Scaling Postgres
29

• Postgres table bloat
29

• FKs relationships
29

• Insufficient logging
29

• Insufficient Caching
29

• Insufficient Monitoring and Metrics
29

• ORMs
29

• ORMs
• Single Point of Failure
29

• ORMs
• Single Point of Failure
• Lack of communications between teams
29

30

Beyond Postgres
30

Beyond Postgres
• Avoid serialization in application code
30

Beyond Postgres
• Feature Flags
30

Beyond Postgres
• Feature Flags
• Browse only mode (Read only mode)
30

Beyond Postgres
• Feature Flags
• Don’t use database for Queuing
30

Beyond Postgres
• Feature Flags
• RabbitMQ
30

Beyond Postgres
• Feature Flags
• RabbitMQ
• Reconsider options for Full Text Search
30

Beyond Postgres
• Feature Flags
• RabbitMQ
• tsearch provided by Postgres
30

Beyond Postgres
• Feature Flags
• RabbitMQ
• tsearch provided by Postgres
• Solr, Lucene
30

31

Scaling postgres

More Related Content

Viewers also liked (19)

Similar to Scaling postgres (20)

More from Denish Patel (13)

Recently uploaded (20)

Scaling postgres