![mkfs / mount options
● ext4, xfs
– mkfs.ext4 E stripewidth=256 /dev/sda1
– mkfs.xfs d su=512k,sw=1 l su=512k f /dev/sda1
– mount: defaults,noatime,discard,nobarrier
● btrfs
– mkfs.btrfs l 8192 L pgdata /dev/sda1
– mount: defaults,noatime,ssd,discard,nobarrier
[compress=lzo]
● zfs
– zpool create pgpool /dev/sda1
– zfs create pgpool/pgdata
– zfs set recordsize=8k pgpool/pgdata
– zfs set atime=off pgpool/pgdata](https://image.slidesharecdn.com/milano-filesystems-150513140654-lva1-app6891/85/PostgreSQL-on-EXT4-XFS-BTRFS-and-ZFS-49-320.jpg)
![ext4 xfs btrfs btrfs (lzo) zfs zfs (lz4)
0
1000
2000
3000
4000
5000
6000
TPC-DS load duration
on EXT4, XFS, BTRFS and ZFS
data indexes
duration[seconds]](https://image.slidesharecdn.com/milano-filesystems-150513140654-lva1-app6891/85/PostgreSQL-on-EXT4-XFS-BTRFS-and-ZFS-50-320.jpg)
![ext4 xfs btrfs btrfs lzo zfs zfs (lz4)
0
100
200
300
400
500
600
700
TPC-DS query performance
EXT4, XFS, BTRFS and ZFS
duration[seconds]](https://image.slidesharecdn.com/milano-filesystems-150513140654-lva1-app6891/85/PostgreSQL-on-EXT4-XFS-BTRFS-and-ZFS-51-320.jpg)
![ext4 xfs btrfs btrfs lzo zfs zfs (lz4)
0
10
20
30
40
50
60
70
TPC-DS space used
on EXT4, XFS, BTRFS and ZFS
size[GB]](https://image.slidesharecdn.com/milano-filesystems-150513140654-lva1-app6891/85/PostgreSQL-on-EXT4-XFS-BTRFS-and-ZFS-52-320.jpg)
PostgreSQL on EXT4, XFS, BTRFS and ZFS
- 1. PostgreSQL on EXT3/4, XFS, BTRFS and ZFS comparing modern (Linux) file systems Tomas Vondra <[email protected]>
- 2. Linux file systems ● plenty of choices, with different – goals, features, tuning options – maturity level, reliability – ext3/4, XFS – traditional, design from the 90s – improving over time, reasonably “modern” ● BTRFS, ZFS – next-generation, new architecture / design ● other (not included in this talk) – log-organized file systems, distributed, clustered, ...
- 4. EXT3, EXT4, XFS - history ● ext3 (2001) / ext4 (2008) – evolution of original Linux filesystem (ext, ext2, ...) – continuous improvements / fixes ● XFS (2002) – originally from SGI Irix 5.3 (1994) – 2000 released under GPL – 2002 merged into 2.5.36 ● both are – reliable journaling file systems – proven by time on many deployments
- 5. EXT3, EXT4, XFS - features ● traditional design with journal ● not handling – multiple devices – volume management – snapshots – ... ● need additional layers for those things – hardware RAID – software RAID (dm) – LVM / LVM2
- 6. EXT3, EXT4, XFS - evolution ● conceived in times of rotational storage – mostly work with SSD – stop-gap for future storage (NVRAM, ...) ● evolution, not a revolution (mostly) – fixing bugs (some real, some imaginary) – adding features (e.g. TRIM, barriers, ...) – scalability improvements (metadata, ...) – be careful when reading old articles / benchmarks – be vary of anecdotal evidence (without context) – synthetic benchmarks are misleading
- 7. EXT3, EXT4, XFS - sources ● Linux Filesystems: Where did they come from? (Dave Chinner @ linux.conf.au 2014) https://www.youtube.com/watch?v=SMcVdZk7wV8 ● Ted Ts'o on the ext4 Filesystem (Ted Ts'o, NYLUG, 2013) https://www.youtube.com/watch?v=2mYDFr5T4tY ● XFS: There and Back … and There Again? (Dave Chinner @ Vault 2015) https://lwn.net/Articles/638546/ ● XFS: Recent and Future Adventures in Filesystem Scalability (Dave Chinner, linux.conf.au 2012) https://www.youtube.com/watch?v=FegjLbCnoBw ● XFS: the filesystem of the future? (Jonathan Corbet, Dave Chinner, LWN, 2012) http://lwn.net/Articles/476263/
- 8. BTRFS, ZFS
- 9. BTRFS, ZFS - goals ● ideas – integrate the layers – design for commodity hardware (expect failures) – design for huge data volumes ● so that we get … – flexible management – built-in snapshotting – compression, deduplication – checksums – ...
- 10. BTRFS, ZFS - history ● BTRFS – merged in 2009, but considered “experimental” – on-disk format “stable” (1.0) – some claim it’s “stable” but I doubt that … – (What are the criteria for filesystem to be “stable”?) ● ZFS – originally from Solaris, but got Oracled :-( – today a bit fragmented development – available on other BSD systems (FreeBSD) – “ZFS on Linux” project (CDDL vs. GPL)
- 11. Tuning options
- 12. Generic tuning options ● TRIM (discard) – enable / disable TRIM on SSDs – impacts garbage collection / wear leveling ● write barriers – prevent disk from optimizing order of writes – still may loose data, but no filesystem corruption – write cache + battery => disable barriers ● SSD alignment – alignment on SSDs matter (pages, blocks, …) – not dedicated tuning options (can use stripe unit / width)
- 13. BTRFS tuning options ● nodatacow (BTRFS) – disable copy on write – still can do snapshots (will do necessary COW) – disables checksums (needs full COW) ● zfs_arc_max – limit the size of ARC cache – should be released automatically, but ...
- 14. BTRFS tuning options ● recordsize=8kB – match the fs page with PostgreSQL page ● ashift=13 (8kB) – align the writes to SSD pages ● primarycache=metadata – prevent double buffering (shared buffers) http://open-zfs.org/wiki/Performance_tuning
- 15. file systems
- 16. ● ext3 (default) ● default ● ext4 ● default ● discard, nobarrier, stripe-width ● xfs ● default ● LVM ● LVM + snapshot ● discard, nobarrier ● discard, nobarrier, agcount, sunit/swidth
- 17. ● btrfs ● default ● nodatacow ● nodiscard (+fstrim) ● zfs ● default ● recordsize=8k, ashift=13, primarycache=metadata (open-zfs) ● recordsize=8k, ashift=13, max_arc_size=5GB (custom)
- 18. benchmarks
- 19. pgbench (TPC-B) ● transactional benchmark – small queries (access by PK, ...) ● modes – read-only – read-write ● scales – small (~200MB) – medium (~50% RAM) – large (~200% RAM)
- 20. TPC-DS ● warehouse, analytical – large amounts of data – queries processing a lot of data ● complex queries – aggregations – joins – CTEs – … ● successor to TPC-H – more elaborate / realistic
- 21. System ● PostgreSQL 9.4.1 ● Gentoo with kernel 3.17 ● CPU: Intel i5-2500k – 4 cores @ 3.3 GHz (3.7GHz) – 6MB cache – 2011-2013 ● 8GB RAM (DDR3 1333) ● SSD Intel S3500 100GB (SATA)
- 23. btrfs btrfs-nodatacow btrfs-nodiscard-fstrim ext3 ext4 ext4-discard-nobarrier-stripe xfs xfs-discard-lvm-snapshot xfs-discard-nobarrier xfs-lvm xfs-tuned-agcount-su-sw zfs zfs-tuned zfs-tuned-2 0 10000 20000 30000 40000 50000 60000 pgbench / small (150MB) / read-only transactions per second
- 24. btrfs btrfs-nodatacow btrfs-nodiscard-fstrim ext3 ext4 ext4-discard-nobarrier-stripe xfs xfs-discard-lvm-snapshot xfs-discard-nobarrier xfs-lvm xfs-tuned-agcount-su-sw zfs zfs-tuned zfs-tuned-2 0 10000 20000 30000 40000 50000 60000 pgbench / medium (50% RAM) / read-only transactions per second
- 25. btrfs btrfs-nodatacow btrfs-nodiscard-fstrim ext3 ext4 ext4-discard-lvm-snapshot ext4-discard-nobarrier-stripe xfs xfs-discard-lvm-snapshot xfs-discard-nobarrier xfs-lvm xfs-tuned-agcount-su-sw zfs zfs-tuned zfs-tuned-2 0 5000 10000 15000 20000 25000 30000 35000 40000 45000 pgbench / large (200% RAM) / read-only transactions per second
- 27. btrfs btrfs-nodatacow btrfs-nodiscard-fstrim ext3 ext4 ext4-discard-nobarrier-stripe xfs xfs-discard-lvm-snapshot xfs-discard-nobarrier xfs-lvm xfs-tuned-agcount-su-sw zfs zfs-tuned zfs-tuned-2 0 1000 2000 3000 4000 5000 6000 7000 8000 pgbench / small (150MB) / read-write transactions per second
- 28. btrfs btrfs-nodatacow btrfs-nodiscard-fstrim ext3 ext4 ext4-discard-nobarrier-stripe xfs xfs-discard-lvm-snapshot xfs-discard-nobarrier xfs-lvm xfs-tuned-agcount-su-sw zfs zfs-tuned zfs-tuned-2 0 1000 2000 3000 4000 5000 6000 pgbench / medium (50% RAM) / read-write transactions per second
- 29. btrfs btrfs-nodatacow btrfs-nodiscard-fstrim ext3 ext4 ext4-discard-lvm-snapshot ext4-discard-nobarrier-stripe xfs xfs-discard-lvm-snapshot xfs-discard-nobarrier xfs-lvm xfs-tuned-agcount-su-sw zfs zfs-tuned zfs-tuned-2 0 500 1000 1500 2000 2500 3000 3500 4000 4500 5000 pgbench / large (200% RAM) / read-write transactions per second
- 39. EXT / XFS conclusions EXT4 ● good “default” choice ● disable barriers (with protected write cache) ● tune alignment to match the SSD ● very “smooth” results XFS ● does not outperform ext4 (in this test) ● not much worse, if properly tuned ● disable write barriers, tune alignment to SSD ● more anomalies than ext4 (sudden performance drops, ...)
- 40. BTRFS & ZFS
- 48. TPC-DS
- 49. mkfs / mount options ● ext4, xfs – mkfs.ext4 E stripewidth=256 /dev/sda1 – mkfs.xfs d su=512k,sw=1 l su=512k f /dev/sda1 – mount: defaults,noatime,discard,nobarrier ● btrfs – mkfs.btrfs l 8192 L pgdata /dev/sda1 – mount: defaults,noatime,ssd,discard,nobarrier [compress=lzo] ● zfs – zpool create pgpool /dev/sda1 – zfs create pgpool/pgdata – zfs set recordsize=8k pgpool/pgdata – zfs set atime=off pgpool/pgdata
- 50. ext4 xfs btrfs btrfs (lzo) zfs zfs (lz4) 0 1000 2000 3000 4000 5000 6000 TPC-DS load duration on EXT4, XFS, BTRFS and ZFS data indexes duration[seconds]
- 51. ext4 xfs btrfs btrfs lzo zfs zfs (lz4) 0 100 200 300 400 500 600 700 TPC-DS query performance EXT4, XFS, BTRFS and ZFS duration[seconds]
- 52. ext4 xfs btrfs btrfs lzo zfs zfs (lz4) 0 10 20 30 40 50 60 70 TPC-DS space used on EXT4, XFS, BTRFS and ZFS size[GB]
- 53. TPC-DS summary ● EXT4, XFS, BTRFS – about the same performance ● compression is nice – uncompressed: 60GB – compressed: ~30GB ● mostly storage capacity, queries not faster ● ZFS much slower :-(