![Configuration files ;; silvoes2lx backup config ;; September 2010 -- Nicholas Paun [APPS] source = "root@silvoes2lx:/media/nss/APPS/" dest = "/bkup/silvoes2lx/APPS" opt = "-e 'ssh -p7774'" delete_after = "+30" [HOME] source = "root@silvoes2lx:/media/nss/HOME/" dest = "/bkup/silvoes2lx/HOME" opt = "-e 'ssh -p7774'" delete_after = "+30"](https://image.slidesharecdn.com/rsyncbackup2-110718214608-phpapp01/85/How-we-setup-Rsync-powered-Incremental-Backups-25-320.jpg)
How we setup Rsync-powered Incremental Backups
- 1. TTP Conference 2011 Celebrating 25 years of supporting Novell Solutions
- 2. Rsync Backups How we set up rsync powered incremental backups for Linux. Nicholas and Nick Paun
- 3. The Plan Because our obsolete tape backups were very slow, unreliable and hard to manage, we decided to go hard disk.
- 4. We needed to be able to keep the backups for at least 30 days.
- 5. The only way we could do this is by using incremental backups.
- 6. After ruling out large, expensive proprietary systems, we chose rsync .
- 7. About rsync Using -- link-dest , rsync can make incremental backups look like full ones without wasting any disk space by using hard links. This makes for easy restoring!
- 8. Because of rsync 's algorithm we do not transmit the full data over the network (except for the first time.)
- 9. Rsync is very reliable and rarely breaks due to file changes.
- 10. Version 1: The Scripts Our scripts were based on some heavily tweaked examples on the rsync site.
- 11. These simple scripts had no error checking, but they worked very well!
- 12. We needed to backup Netware servers too, so we used NCP mounting , which was very slow.
- 13. The Scripts echo "Here we rsync DATA volume on CASADMLX server to /bkup/casadmlx/DATA/ folder on CASARKLX server" rsync -av --link-dest= /bkup/casadmlx/DATA/current root@casadmlx:/media/nss/DATA/ /bkup/casadmlx/DATA/ $date rm /bkup/casadmlx/DATA/current ln -s /bkup/casadmlx/DATA/ $date /bkup/casadmlx/DATA/current rm -r /bkup/casadmlx/DATA/ `date --date='30 days ago' +%F`
- 14. The Scripts echo "Here we rsync DATA volume on CASADM server to /bkup/casadm/DATA/ folder on CASARKLX server" ncpmount -S casadm -A casadm -U .bkup.cs.users.selkirk -P provo /mnt/casadm/ rsync -av --link-dest= /bkup/casadm/DATA/current /mnt/casadm/DATA/ /bkup/casadm/DATA/ $date rm /bkup/casadm/DATA/current ln -s /bkup/casadm/DATA/ $date /bkup/casadm/DATA/current rm -r /bkup/casadm/DATA/ `date --date='30 days ago' +%F` umount /mnt/casadm/
- 15. The Scripts echo "Here we rsync INSTRUCTORS volume on FS1 server to /bkup/fs1/INSTRUCTORS/ folder on CASARKLX server" mount.cifs //fs1/INSTRUCTORS /mnt/fs1/ -o user=Administrator,pass=provo rsync -av -e 'ssh -p7774' --link-dest= /bkup/fs1/INSTRUCTORS/current /mnt/fs1/INSTRUCTORS/ /bkup/fs1/INSTRUCTORS/ $date rm /bkup/fs1/INSTRUCTORS/current ln -s /bkup/fs1/INSTRUCTORS/ $date /bkup/fs1/INSTRUCTORA/current rm -r /bkup/fs1/INSTRUCTORS/ `date --date='30 days ago' +%F` umount /mnt/fs1/
- 16. Version 2: alpine-backup Eventually we decided that we needed a proper backup system.
- 17. We wanted to have good error checking, replication, reporting and central control.
- 18. We also wanted to actually understand what was going on and we wanted a system that would fit our exact needs. So we wrote our own.
- 19. Version 2: alpine-backup After learning PHP programming and experimenting for awhile, I wrote alpine-backup , our current backup solution.
- 20. We decided to implement only the bare-minimum that we needed.
- 21. The result was a 100-line script that supports custom mount drivers and performs an incremental backup using rsync .
- 22. alpine-backup function backup ( $opt , $src , $dest , $date , $name ) { $_opt = OPTIONS ; passthru ( "rsync $_opt $opt --link-dest $dest /current $src $dest / $date " , $return ); if ( $return != 0 ) { echo( "* Backup Failure ( $name ): rsync did not finish sucessfully.\n" ); die( 1 ); } unlink ( " $dest /current" ); symlink ( " $dest / $date " , " $dest /current" ); echo( "* Backup Successful: $name \n" ); } Website: http://www.icebergsystems.ca/
- 23. Google Code: http://alpine-backup.googlecode.com
- 25. Configuration files ;; silvoes2lx backup config ;; September 2010 -- Nicholas Paun [APPS] source = "root@silvoes2lx:/media/nss/APPS/" dest = "/bkup/silvoes2lx/APPS" opt = "-e 'ssh -p7774'" delete_after = "+30" [HOME] source = "root@silvoes2lx:/media/nss/HOME/" dest = "/bkup/silvoes2lx/HOME" opt = "-e 'ssh -p7774'" delete_after = "+30"
- 26. Mount Drivers function mountfs ( $user , $server , $command ) { $return = passthru ( "ssh $user @ $server ' $command stop'" ); return(! $return ); } function umountfs ( $user , $server , $command ) { $return = passthru ( "ssh $user @ $server ' $command start'" ); return(! $return ); }
- 27. Logging
- 28. Experiments We setup offsite mirroring
- 29. Experimented with rsync daemons
- 30. We used batch files for replication
- 31. We ran rsync on cygwin (too slow.)
- 32. Backed up Windows servers via CIFS mounting
- 33. We setup Groupwise backups
- 34. We also started iPrint backups
- 35. Syncing using Batch Files Batch files allow you to send the same file list to multiple backup servers at once. ie. mirroring, replication, etc. We used them to sync our backups between 2 campuses.
- 36. This was our first attempt at offsite backups.
- 37. It had a drawback of losing sync if files had changed in between batches.
- 38. Syncing using Batch Files The first command uses --write-batch to create a batch file.
- 39. The second one uses --read-batch to perform the backup.
- 40. rsync -av --write-batch=SCRATCH.batch --link-dest=/bkup/caslablx/SCRATCH/current /mnt/caslablx/SCRATCH/ /bkup/caslablx/SCRATCH/$date
- 41. ssh n10arklx rsync -av --read-batch=- --link-dest=/bkup/caslablx/SCRATCH/current /bkup/caslablx/SCRATCH/$date <SCRATCH.batch
- 42. Offsite Backups The college bought 5 QNAPs, for offsite backups.
- 43. The QNAPs are NASs running some mystery meat Linux, that was the source of many headaches.
- 44. Initially, we added more configuration files that would copy the backups from the main backup server to the offsite one, but this grew quite complicated.
- 45. Selkirk College Backup System
- 46. No-change Syncing So, we came up with a better idea: the configuration files would not be modified, and still point at the real backup server.
- 47. But, when a QNAP reads the configuration files, it would run in slave mode and pull the backups from the backup server, instead of the live server.
- 48. This change removed the need for half of our configuration files.
- 51. GroupWise Backups Because dbcopy was slow and inefficient for backups, we decided to snapshot our GroupWise volumes.
- 52. This means that we can apply rsync 's deduplication ( --link-dest ) to GroupWise too.
- 53. We save 90-95% disk space and it is 10 times faster compared to a full backup.
- 54. GroupWise Backups As an added benefit, GroupWise and FS backups can now be handled by the same tool.
- 55. To snapshot NSS volumes we used Dean Giles' perl scripts (found on Cool Solutions).
- 56. If we ran GroupWise on an LVM volume, we could use LVM snapshots instead.
- 57. GroupWise Mount Driver function mountfs ( $user , $server , $command ) { $return = passthru ( "snapup.sh" ); return(! $return ); } function umountfs ( $user , $server , $command ) { $return = passthru ( "snapdn.sh" ); return(! $return ); }
- 58. Snapup.sh ssh casgrplx perl /usr/local/bin/AddPoolSnapshot.pl /dev/evms/CASGRPLXPOOL sdb 200000000 CASGRPLXPOOL_S1 ssh casgrplx perl /usr/local/bin/ActivatePoolSnapshot.pl CASGRPLXPOOL_S1 ssh casgrplx mount GRPWISE_SV /media/nss/GRPWISE_SV -t nssvol -o rw,name=GRPWISE_SV
- 59. Demo: GroupWise PO Restore
- 60. Demo: GroupWise PO Restore
- 61. Demo: GroupWise PO Restore
- 62. MySQL Backups We wanted to incrementally backup a database, but still use native dumping tools.
- 63. So we added a MySQL agent to our tools.
- 64. To gain maximum space savings, we split up the SQL files into one file per table.
- 65. Now we can use rsync 's --link-dest feature on our DBs too.
- 66. MySQL Backups This is the the command we use to backup databases mysql -p password db -e 'show tables' | grep -v 'Tables_in_' #show all tables in database mysqldump -p password --skip-dump-date db $table > $dumpdir/$table #create a dump of each individual table (with no timestamp, for linking) find $dumpdir -exec touch -d 2010-04-01 {} \; #touch dumps to a consistent date (for linking)
- 67. Demo: Drupal DB Restore
- 68. Demo: Drupal DB Restore
- 69. Demo: Drupal DB Restore
- 70. Demo: Drupal DB Restore
- 71. Oracle Backups One day, we got a user request to set up Oracle backups on a Windows server.
- 72. We had never seen an Oracle database before, so we did some searching and figured out how to use expdp.
- 73. First we logged in to the database and set up a directory handle:
- 74. CREATE DIRECTORY sgrcdev AS '/g/sgrcdev' ;
- 75. Oracle Backups Next, we use cygwin to run expdp on the server:
- 76. expdp system/$passwd@$db directory=$db full=y dumpfile=$db.dmp logfile=$db.log
- 77. … And then, we just copy the backups over and reset dump directory.
- 78. Oracle Backups
- 79. Conclusion In 2007, we didn't have a real backup, even if we bought an enterprise solution.
- 80. Now, we can control all of our backups (FS, GroupWise and Database) from one place, using one lightweight tool.
- 81. We save a lot of disk space by using rsync .
- 82. Most importantly, we achieved peace of mind.
- 83. TTP Conference 2011 Celebrating 25 years of supporting Novell Solutions
Editor's Notes
- #2: TTP Conference 2011 – Celebrating 25 years of supporting Novell Solutions
- #5: According to the man page, --link-dest hard links to files in the link destination when the backup is the same as the source. (ie. identical files).
- #7: We set our link dest to current, a symbolic link always pointing to the current backup, then preform the backup. Next we set our current link to the latest backup and delete backups older than 30 days.
- #8: This is how we backed up a Netware server using NCP.
- #9: ..and a Windows one, using CIFS.
- #12: This is a screenshot of alpine-backup's code. As you can see, it is quite similar to our bash scripts, but adds error checking and is written in PHP. If you want to have a look at alpine-backup, the URLs are on the bottom.
- #13: This what our backup directory looks like. As you can see, only the first backup is large, the rest are quite small, only about 200MB large.
- #14: This an example of the configuration file format alpine-backup uses. It is an INI file. (I used this format because PHP had a built-in parser for it.)
- #15: This what our mount drivers look like. This particular one actually stops and restarts services for backup. (We use it for iPrint backups.)
- #16: Here is a screenshot of the mail we get from cron. As you can see from the summary, the backups are all successful!
- #21: This is a diagram of Selkirk College's organizational structure. All our servers are backed up in at least two locations. As you can see, there are two islands: the Nelson-centric one and the Castlegar-centric one. This is due to the fact that there are no fibre links between Castlegar and Nelson.
- #22: ...about half. (we sometimes use different parameters for our main and offsite backups.)
- #23: At this point I'd like to show you our first demo: How no-change syncing works. QNAP doing a Live Backup
- #24: QNAP in Slave Mode
- #27: Well, this does not explain much, let's have a look at snapup.sh
- #28: The first two commands SSH into our GroupWise server and use the perl scripts to create and activate a snapshot. Then we mount the snapshot and perform the backup via SSH.
- #29: Now, I'm going to move on to our next demo. Live Post Office
- #30: We start up the POA using an a backup.
- #31: ..and now: some old mail.
- #32: alp-dbrestore does not like dashes in DB Names.
- #34: Next, I'm going to show you how we restore our MySQL backups. First, we create test article.
- #35: Then, we create a new DB with an old version of the site. (Avoid dashes in database name)
- #36: We switch Drupal to the restore DB.
- #37: The article has now been rolled back.
- #40: This is a screenshot of our Oracle backups in action. Since we're running early, I'm going to show you what alpine-backup's code is like.
- #42: Any Questions? TTP Conference 2011 – Celebrating 25 years of supporting Novell Solutions