Honey I Shrunk the Database
Download as PPTX, PDFâ˘0 likesâ˘1,010 views
This document discusses strategies for shrinking production databases to use in test and development environments. It presents a case study of how Paperless Post creates a composite slice of their production data including application schemas, static content tables, a subset of user data, and mutated email addresses. The strategies allow for daily fresh data while protecting sensitive information and limiting resource usage on non-production machines.
![PG Tools â Vertical SliceFlexibility at Source (Production)pg_dumpInclude data only [-a --data-only]Include table schema only [-s --schema-only]Select tables [-t table1 table2 --table table1 table2]Select schemas [-nschema --schema=schema]Exclude schemas [-N schema --exclude-schema=schema]](https://image.slidesharecdn.com/honeyishrunkthedatabasefinal-110915142754-phpapp01/85/Honey-I-Shrunk-the-Database-18-320.jpg)
![PG Tools â Vertical SliceFlexibility at Destination (Staging, Development)pg_restoreInclude data only [-a --data-only]Select indexes [-iindex --index=index]Tune processing [-jnumber-of-jobs --jobs=number-of-jobs]Select schemas [-nschema --schema=schema]Select triggers[-T trigger --trigger=trigger]Exclude privileges [-x --no-privileges --no-acl]](https://image.slidesharecdn.com/honeyishrunkthedatabasefinal-110915142754-phpapp01/85/Honey-I-Shrunk-the-Database-19-320.jpg)
Honey I Shrunk the Database
- 1. Honey, I Shrunk the DatabaseFor Test and Development EnvironmentsVanessa HurstPaperless PostPostgres Open, September 2011
- 3. User Data
- 4. Why Shrink?AccuracyYou donât truly know how your app will behave in production unless you use real data.Production data is the ultimate in accuracy.
- 5. Why Shrink?AccuracyFreshnessNew data should be available regularly.Full database refreshes should be timely.
- 6. Why Shrink?AccuracyFreshnessResource LimitationsStaging and developer machines cannot handle production load.
- 7. Why Shrink?AccuracyFreshnessResource LimitationsData ProtectionLimit spread of sensitive user or client data.
- 8. Why Shrink?AccuracyFreshnessResource LimitationsData Protection
- 9. Case Study: Paperless PostRequirementsFreshness â Daily, On command for non-developersShrinkage â Slices, Mutations
- 10. Case Study: Paperless PostRequirementsFreshness â Daily, On command for non-developersShrinkage â Slices, MutationsResourcesSource â extra disk space, RAM, and CPUsDestination â limited, often entirely un-optimizedDevelopment -- constrained DBA resources
- 11. Shrink StrategiesCopiesRestored backups or live replicas of entire production database
- 12. Shrink StrategiesCopiesSlicesSelect portions of exact data
- 13. Shrink StrategiesCopiesSlicesMutationsSanitized, anonymized, or otherwise changed data
- 14. Shrink StrategiesCopiesSlicesMutationsAssumptionsSeed databases, fixtures, test data
- 16. SlicesVertical SliceDifficult to obtain a valid, useful subset of data.Example: Include some entire tables, exclude others
- 17. SlicesVertical SliceDifficult to obtain a valid, useful subset of data.Example: Include some entire tables, exclude othersHorizontal SliceDifficult to write and maintain.Example: SQL or application code to determine subset of data
- 18. PG Tools â Vertical SliceFlexibility at Source (Production)pg_dumpInclude data only [-a --data-only]Include table schema only [-s --schema-only]Select tables [-t table1 table2 --table table1 table2]Select schemas [-nschema --schema=schema]Exclude schemas [-N schema --exclude-schema=schema]
- 19. PG Tools â Vertical SliceFlexibility at Destination (Staging, Development)pg_restoreInclude data only [-a --data-only]Select indexes [-iindex --index=index]Tune processing [-jnumber-of-jobs --jobs=number-of-jobs]Select schemas [-nschema --schema=schema]Select triggers[-T trigger --trigger=trigger]Exclude privileges [-x --no-privileges --no-acl]
- 21. MutationsExternal Data ProtectionHIPAA RegulationsPCI ComplianceAPI Terms of Use
- 22. MutationsExternal Data ProtectionHIPAA RegulationsPCI ComplianceAPI Terms of UseInternal Data ProtectionProtecting your usersâ personal dataProtecting your users from accidents, e.g. staging emailsYour Terms of Service
- 23. User Data
- 24. Case Study: Paperless PostComposite Slice includingVertical Slice â All application object schemasVertical Slice â Entire tables of static contentHorizontal Slice â Subset of users and their dataMutation â Changed user email addresses
- 25. Case Study: Paperless PostComposite Slice includingVertical Slice â All application object schemaspg_dump --clean --schema-only --schema public db-01 > slice.sql
- 26. Case Study: Paperless PostComposite Slice includingVertical Slice â All application object schemaspg_dump --clean --schema-only --schema public db-01 > slice.sqlVertical Slice â Entire tables of static contentpg_dump --data-only --schema public -t cards db-01 >> slice.sql
- 27. Case Study: Paperless PostComposite Slice includingVertical Slice â All application object schemaspg_dump --clean --schema-only --schema public db-01 > slice.sqlVertical Slice â Entire tables of static contentpg_dump --data-only --schema public -t cards db-01 >> slice.sql Horizontal Slice â Subset of users and their dataMutation â Changed user email addresses
- 28. Case Study: Paperless PostCREATE SCHEMA staging;
- 29. Case Study: Paperless PostHorizontal SliceCustom SQLSELECT * INTO staging.usersFROM usersWHERE EXISTS (subset of users);
- 30. Case Study: Paperless PostHorizontal SliceCustom SQLSELECT * INTO staging.usersFROM usersWHERE EXISTS (subset of users);Dynamic relative to full data set or newly created sliceSELECT * INTO staging.stuffFROM stuffWHERE EXISTS (stuff per staging.users);
- 31. Case Study: Paperless PostHorizontal SliceCustom SQLDynamic relative to full data set or newly created sliceMutationsEmail AddressesUse regular expressions to clean non-admin addressese.g. [email protected] => [email protected] DataClear cached short link from link-shortening API
- 32. Case Study: Paperless PostComposite Slice includingVertical Slice â All application object schemaspg_dump --clean --schema-only --schema public db-01 > slice.sqlVertical Slice â Entire tables of static contentpg_dump --data-only --schema public -t cards db-01 >> slice.sql Horizontal Slice â Subset of users and their dataMutation â Changed user email addressespg_dump --data-only --schema staging db-01 >> slice.sql
- 33. Case Study: Paperless PostRebuildPrepare new database as standbyGracefully close connectionsRotate by renaming databasesSecurity Dedicated database build userMembership in application user roleApplication user role & privileges remain
- 34. Case Study: Paperless PostRebuild$ bzcat slice.sql.bz2 | psql db-newStaging schema has not been created, so all data loads to default schema
- 35. Case Study: Paperless PostWe hacked our rebuild by importing across schemas!Now our sequences are wrong, causing duplicate data errors every time we try to insert into tables.
- 36. Secret Weapon --Updates all serial sequences for ID columns onlyBEGINFOR table_record IN SELECT pc.relname FROM pg_class pc WHERE pc.relkind = 'r' AND EXISTS (SELECT 1 FROM pg_attribute pa WHERE pa.attname = 'id' AND pa.attrelid = pc.oid) LOOPtable_name = table_record.relname::text; EXECUTE 'SELECT setval(pg_get_serial_sequence(' || quote_literal(table_name) || ', ' || quote_literal('id')::text || '), MAX(id)) FROM ' || table_name || ' WHERE EXISTS (SELECT 1 FROM ' || table_name || ')';END LOOP;
- 37. Case Study: Paperless PostRebuild$ bzcat slice.sql.bz2 | psql db-newStaging schema has not been created, so all data loads to default schemaecho âselect 1 from update_id_sequences();â >> slice.sqlVacuumReindex
- 38. Case Study: Paperless PostSecurity Database build userCREATE DB privilegesMember of Application user roleApplication user remains database ownerApplication user privileges remain limitedBuild only works in predetermined environments
- 39. Case Study: Paperless PostRequirementsFreshness â Daily, On command for non-developersShrinkage â Slices, MutationsResourcesSource â extra disk space, RAM, and CPUsDestination â limited, often entirely un-optimizedDevelopment -- constrained DBA resources
- 40. Questions?Vanessa HurstPaperless Post@DBNessPostgres Open, September 2011
- 41. More ToolsCopies -- LVMSnapshotsSee talk by Jon Erdman at PG Conf EUGreat for all readsData stays virtualized & doesnât take up space until changedIdeal for DDL changes without actual data changes
- 42. More ToolsCopies, Slices-- pg_staging by dmitrihttp://github.com/dimitri/pg_stagingSimple -- pauses pgbouncer & restores backupEfficient -- leverage bulk loadingFlexible -- supports varying psql filesCustom -- limitedSlices -- replicate by rtomayko of Github http://github.com/rtomayko/replicateSimple - Preserves object relations via ActiveRecordInefficient -- Creates text-based .dumpInflexible -- Corrupts id sequences on data insertCustom -- highly
Editor's Notes
- #2: I am Vanessa Hurst and I lead Data and Analytics at Paperless Post, a customizable online stationery startup in New York. I studied Computer Science and Systems and Information Engineering at the University of Virginia. I have experience in databases ranging from a few hundred megabyte CMSes for non-profits to terabytes of financial data and high traffic consumer websites. I've worked in data processing, product development, and business intelligence. I am happy open-source convert and lone data wrangler in a land of web developers using Ruby on Rails.
- #3: Static Data
- #8: This may include external, legal regulations or internal regulations such as terms of service.Data protection can also include mitigating risk or proactively screening before data is even available.HIPAA RegulationsPCI ComplianceAPI Terms of Use
- #9: Any other reasons?
- #10: RequirementsSlice -- significantly less space, power, & memory in staging and dev environments, need smaller data setMutation -- protect user data in highly personal communications, prevent staging use of customer emailsDaily RefreshResourcesSource -- production server w/ample space, power, & memoryDestination -- weak, shared staging infrastructure across several servers, local machine development infrastructureExpertise -- flexible infrastructure automation tools, many application developers, limited DBA time
- #11: RequirementsSlice -- significantly less space, power, & memory in staging and dev environments, need smaller data setMutation -- protect user data in highly personal communications, prevent staging use of customer emailsDaily RefreshResourcesSource -- production server w/ample space, power, & memoryDestination -- weak, shared staging infrastructure across several servers, local machine development infrastructureExpertise -- flexible infrastructure automation tools, many application developers, limited DBA time
- #12: Quick vocabularyBackup & restore, trigger-based replication, there are plenty of options that are all straight forward, but donât give you a lot of leeway on resources.
- #13: Most common case
- #16: If youâre doing Business Intelligence, you need a copy of your production database. Figure it out.
- #17: Vertical -- difficult to keep data valid & usable -- valid units of space are not always valid in an applicatione.g. WAL logs, Pages 1-16 => smaller, finite size, not usableHorizontal -- requires application logic, highly customized but usable e.g. Users with ids 1-50, Users who joined before July 4 Users who are admins, any SQL logic
- #18: Vertical -- difficult to keep data valid & usable -- valid units of space are not always valid in an applicatione.g. WAL logs, Pages 1-16 => smaller, finite size, not usableHorizontal -- requires application logic, highly customized but usable e.g. Users with ids 1-50, Users who joined before July 4 Users who are admins, any SQL logic
- #19: http://www.postgresql.org/docs/current/static/app-pgdump.htmlOptions to: DumpOIDs in case your app uses them Leave out ownership commands (if staging environments run as different users)
- #20: http://www.postgresql.org/docs/current/static/app-pgdump.htmlOptions to: DumpOIDs in case your app uses them Leave out ownership commands (if staging environments run as different users)
- #21: Static Data
- #29: Dedicated schema preserves all table, index, sequence names, etc
- #34: Only the build process is staging-specific, all other privileges and settings match production
- #35: Only the build process is staging-specific, all other privileges and settings match production
- #38: Only the build process is staging-specific, all other privileges and settings match production
- #39: Only the build process is staging-specific, all other privileges and settings match production
- #40: RequirementsSlice -- significantly less space, power, & memory in staging and dev environments, need smaller data setMutation -- protect user data in highly personal communications, prevent staging use of customer emailsDaily RefreshResourcesSource -- production server w/ample space, power, & memoryDestination -- weak, shared staging infrastructure across several servers, local machine development infrastructureExpertise -- flexible infrastructure automation tools, many application developers, limited DBA time
- #43: http://github.com/rtomayko/replicate