WalB: Real-time and Incremental Backup System for Block Devices

Editor's Notes

• #2: Thank you for attending to this presentation. I’ll talk about WalB, a fast and low latency backup system for block devices. OK, let’s start.
• #3: My name is Kota Uchida. I’m a site reliability engineer at Cybozu incorporated. I’m a WalB developer. I have deployed WalB backup system on our production environment.
• #4: Do you know Cybozu? Cybozu is a large cloud service vendor in Japan. We have the largest market shares in the field of “groupware” or “collaborative software”, like online calendar, workflow, bulletin board system, and so on. Our mission is to enhance teamworks all over the world. We serve web applications on our own cloud platform, not a public cloud like AWS. One of our applications is “kintone”. “kintone” is a low-code business application platform. You can create business applications with little or no code.
• #5: Over nineteen thousands companies are using our services. One ninety millions accesses per day. About Twenty five tibibytes data are written to storages everyday.
• #6: Let me explain our service level objective. Our services are twenty-four seven, nonstop services. We target, and almost achieve, four-nine availability. We backup user data everyday and keep them for fourteen days. We also send the data to a remote site once a day for disaster recovery.
• #7: This diagram shows the architecture of our cloud platform. It is basic architecture. An user request goes through several components, such as layer seven load balancers, application servers, a database server and a blob server. User data will be written to two storage servers. They replicate data each other by software RAID I. Data written to the storage servers will be backed up to a backup server, and copied to a remote site once a day. In a storage server, we use dm-snap to create snapshots and back them up. I’ll show how we do that in the next slide.
• #8: This is detailed architecture of dm-snap. The above figures, indicated as logical structure, show how volumes and snapshots of dm-snap look like from the point of view of users and applications . Let’s consider there is a disk which consists of 5 blocks, numbered zero to four. We assume we took a snapshot when the content of block 1 was A, and block 3 was B. Then write requests came to block 1 and 3. As a result these blocks were overwritten, while the snapshot image was not changed. The below figures are physical structure of a volume with a snapshot. When you create a snapshot, dm-snap prepares a snapshot area with initialized mapping information. When you submit a write IO request at block 1 with A-dash data, dm-snap copies block 1 to the snapshot area first, then the block will be overwritten by A-dash data. In the next slide, I’ll show you how to backup data using dm-snap.
• #9: There are two snapshots 0 and 1. We assume snapshot 1 is newer than snapshot 0. To backup a snapshot taken by dm-snap, two snapshots are required. We scan all blocks of the two snapshots, then we compare them block by block, finally we can get a diff image. We need massive amount of reads for incremental backup, using dm-snap. Q: Like dm-thin, is it possible to extract diff images using mapping indexes without full scan dm-snap? A: Yes it is possible, but it must be very slow because it requires two snapshots at the same time. The overhead of copy-on-write is much larger than that of with dm-thin.
• #10: So we backup user data at night, when accesses are decreased. The above graph shows read I/O throughput of one volume. There is a large spike, about seven hundred MB per second, at midnight. The below graph shows the number of user requests on our platform. Many people don’t use our services at night. # グラフデータは read throughput については tyss-221 と tyss-222 の合算。 response については serviceset:ty13 の全て # Q. 何故 daytime に read がほとんどないのか # A. ほとんどのデータはキャッシュに載っているから Because almost all of data are on cache memory.
• #11: This graph shows a time series of user response time in milliseconds . Data consist of several update operations in a storage server in production. During full-scan, user experience seems to be worse than usual, because ninety percentile of response time exceeds 1 second . # 結局 ty22 の kintone add.json のみのグラフ。10分間隔でのデータ。8:00JST-19:00JST
• #12: And now, we have no more nights. Because we try to provide our services to worldwide customers, we have to handle accesses from multiple timezones. So we must be able to backup user data any time without affecting user experience. With dm-snap, it cannot be achieved. We need a new solution.
• #13: We have researched other backup solutions which satisfy 2 requirements: no I/O spikes and short backup time. There are two candidates, dm-thin and WalB. We compared them.
• #14: dm-thin provides thin-provisioning volume management to share same data among volumes and reduce disk usage using snapshots. dm-thin is included in the mainline kernel. In the following slides, I will explain how dm-thin implements snapshot feature, and how to use it for backup.
• #15: This figures show how dm-thin provides snapshot management feature. Please look at the above figure. From user’s point of view, a snapshot of dm-thin can be considered as a normal volume. Next, look at the below figure. This is little bit complicated. Since dm-thin is not the essential part of this talk, you may ignore this diagram. At first, there is one tree expressing the latest image. Intermediate nodes have only meta data, and leaf nodes have user data.
• #16: When you take a snapshot of a volume, dm-thin copies the root node for the latest and the snapshot tree. At this point, the “latest tree” node refers the original intermediate nodes.
• #17: When a write request comes, dm-thin copies a corresponding leaf and its ancestor nodes, and modify the link of the root node. The latest tree has been updated while the snapshot tree remains unchanged.
• #18: In this slide, I’ll explain how to backup a volume using dm-thin. In the above figure, there are two snapshots. We assume snapshot 1 is newer than snapshot 0. Difference between snapshot 0 and 1 are two blocks, A and B. Its physical structure is pictured as the below figure. There are two trees representing two snapshots. You can get a diff image for incremental backup by comparing structure of two snapshot trees. In this example, the diff image consists of A’ and B’.
• #19: So far I have explained dm-thin. From here, let me introduce WalB. WalB is a real-time and incremental backup system. It has been developed at Cybozu Labs. Using WalB, there seems to be no I/O spikes.
• #20: This slide explains architecture of a WalB device. WalB backup system uses special block devices, called WalB devices. A WalB device, shown in this picture, is a virtual block device that consists of two ordinary block devices, a data device and a log device. A data device stores user data. Its block addresses are mapped linearly to WalB device. A log device stores write IO logs. This area is used as a ring buffer. Write I/O requests are handled at both the data and log devices. Read I/O requests are handled at the data device only. WalB device driver preserves consistency appropriately.
• #21: In this slide, I will explain how WalB takes a backup continuously. Imagine there is a block device with 5 blocks. Block 1 has data A, and block 4 has data B.
• #22: When a write request comes, a WalB device writes it to a data device and a log device. WalB tools read the log device and generate a diff image.
• #23: Another write request will be treated in the same way. Diff images generated by WalB tools will be sent to a backup server.
• #24: We conducted an experiment to see performance of dm-snap, dm-thin, and WalB. In the experiment, we executed a workload during a backup. The workload and the backup will affect each other, so we did these concurrently. We measured two metrics, IO latencies of the workload and the backup time.
• #25: We used server machines with the same spec as ones in our production environment. CPUs have total twelve cores. One-ninety-two gibibytes memory. RAID six storage. Ten gigabits ethernet. Latest upstream kernel. We created one hundred gibibytes volumes for each backup solution, dm-snap, dm-thin, and WalB. And we executed four kibibytes random writes onto the volumes. During the workload running, we executed an incremental backup also.
• #26: Now let’s check again how to backup with dm-snap, dm-thin, and WalB. With dm-snap, we have to scan full area of the snapshot to create a diff image for incremental backup. Those read I/Os are nearly sequential. With dm-thin, first we get information about structures of snapshot trees; then we calculate which blocks are modified; finally we read modified blocks to create a diff image. Tree traversal is tend to be random read. In fact we did an emulation of backup for dm-snap and dm-thin. A real backup has to read a volume, calculate differences and send them to a backup server. In this experiment, for dm-snap, we just scanned full of the latest snapshot. For dm-thin, we just read modified blocks using metadata information. They are the dominant part of the backup time, so we treated them as their backup time.
• #27: For WalB, we employed a real backup system, which is used in our production environment. It’s not an emulation. WalB tools extract write IO logs from the log device and send them to the backup server. The logs extracted are converted to diff images at the backup server. Then the corresponding snapshot becomes restorable. Because WalB tools continuously extract and send I/Os logs, written data will be backed up at almost the same time as the data have been written to a volume. Backup time is defined as a latency from when a snapshot is set at the WalB device as a mark on the latest IO log, to when the snapshot becomes restorable at the backup server. Our 10 gigabit datacenter network does not become a bottleneck, so the backup time with WalB will be a few seconds or less.
• #28: This is the result of our experiment. This graph shows I/O latencies of workloads, 4KiB random writes during a backup. Base line is a normal logical volume without backup, labelled no-backup. Its latency is stably at 5 milliseconds. Latency of dm-snap is about six times of no-backup, thirty milliseconds. Existence of snapshot and a lot of read I/Os affects write I/O latency. Latency of dm-thin is much worse than that of dm-snap! In the first forty seconds, we can see a large spike of dm-thin. We can see an overhead of WalB, but it is very small: about zero point five milliseconds. WalB does not generate spikes. These characteristics are important for 24/7 cloud service. Now let’s look another aspect of this experiment.
• #29: This graph shows backup times for the three solutions. Backups are executed under random write workload. dm-snap takes about one thousand seconds. dm-snap is the solution we were using before WalB. dm-thin takes twice as long as dm-snap. If user data we have grow larger, dm-thin may not be able to back them up within one day. It cannot be tolerated because our objective declares daily backup. WalB takes only 1 point 2 seconds. This is short enough to satisfy the declaration of daily backup. In addition to that, this duration, 1 point 2 seconds, is independent of amount of data. It depends amount of write IOs. Our service will glow much more, but backup time shall be no longer a problem. Let me conclude my talk.
• #30: Taking backup with dm-snap takes a long time and makes user experience worse. Then I compared two solutions: dm-thin and WalB. dm-thin takes also a long time to backup once and causes spikes of I/O latency, while WalB takes a few seconds and has no spikes. So we chose WalB for our backup system.
• #31: WalB is of course an open source software, it is developed on GitHub. There are vagrantfiles for Ubuntu and CentOS. Detailed tutorial documentation is linked in readme of the vagrantfile. Please try WalB.
• #38: Thank you for coming to this presentation. Now questions are welcomed.