Distributed File System
- 1. Lecture 2 – Distributed Filesystems 922EU3870 – Cloud Computing and Mobile Platforms, Autumn 2009 2009/9/21 Ping Yeh ( 葉平 ), Google, Inc.
- 2. Outline Get to know the numbers
- 4. Distributed file systems Basic (example: NFS)
- 5. Shared storage (example: Global FS)
- 9. Fault-tolerant and Parallel (example: dCache) The Google File System
- 10. Homework
- 11. Numbers real world engineers should know L1 cache reference 0.5 ns Branch mispredict 5 ns L2 cache reference 7 ns Mutex lock/unlock 100 ns Main memory reference 100 ns Compress 1 KB with Zippy 10,000 ns Send 2 KB through 1 Gbps network 20,000 ns Read 1 MB sequentially from memory 250,000 ns Round trip within the same data center 500,000 ns Disk seek 10,000,000 ns Read 1 MB sequentially from network 10,000,000 ns Read 1 MB sequentially from disk 30,000,000 ns Round trip between California and Netherlands 150,000,000 ns
- 12. The Joys of Real Hardware Typical first year for a new cluster: ~0.5 overheating (power down most machines in <5 mins, ~1-2 days to recover) ~1 PDU failure (~500-1000 machines suddenly disappear, ~6 hours to come back) ~1 rack-move (plenty of warning, ~500-1000 machines powered down, ~6 hours) ~1 network rewiring (rolling ~5% of machines down over 2-day span) ~20 rack failures (40-80 machines instantly disappear, 1-6 hours to get back) ~5 racks go wonky (40-80 machines see 50% packetloss) ~8 network maintenances (4 might cause ~30-minute random connectivity losses) ~12 router reloads (takes out DNS and external vips for a couple minutes) ~3 router failures (have to immediately pull traffic for an hour) ~dozens of minor 30-second blips for dns ~1000 individual machine failures ~thousands of hard drive failures slow disks, bad memory, misconfigured machines, flaky machines, etc.
- 13. File Systems Overview System that permanently stores data
- 14. Usually layered on top of a lower-level physical storage medium
- 15. Divided into logical units called “files” Addressable by a filename (“foo.txt”) Files are often organized into directories Usually supports hierarchical nesting (directories)
- 16. A path is the expression that joins directories and filename to form a unique “full name” for a file. Directories may further belong to a volume
- 17. The set of valid paths form the namespace of the file system.
- 18. What Gets Stored User data itself is the bulk of the file system's contents
- 19. Also includes meta-data on a volume-wide and per-file basis: Volume-wide: Available space Formatting info character set ... Per-file: name owner modification date physical layout...
- 20. High-Level Organization Files are typically organized in a “tree” structure made of nested directories
- 21. One directory acts as the “root”
- 22. “links” (symlinks, shortcuts, etc) provide simple means of providing multiple access paths to one file
- 23. Other file systems can be “mounted” and dropped in as sub-hierarchies (other drives, network shares)
- 24. Typical operations on a file: create, delete, rename, open, close, read, write, append. also lock for multi-user systems.
- 25. Low-Level Organization (1/2) File data and meta-data stored separately
- 26. File descriptors + meta-data stored in inodes (Un*x) Large tree or table at designated location on disk
- 27. Tells how to look up file contents Meta-data may be replicated to increase system reliability
- 28. Low-Level Organization (2/2) “Standard” read-write medium is a hard drive (other media: CDROM, tape, ...)
- 29. Viewed as a sequential array of blocks
- 30. Usually address ~1 KB chunk at a time
- 31. Tree structure is “flattened” into blocks
- 32. Overlapping writes/deletes can cause fragmentation: files are often not stored with a linear layout inodes store all block numbers related to file
- 33. Fragmentation
- 34. Filesystem Design Considerations Namespace: physical, logical
- 35. Consistency: what to do when more than one user reads/writes on the same file?
- 36. Security: who can do what to a file? Authentication/ACL
- 37. Reliability: can files not be damaged at power outage or other hardware failures?
- 38. Local Filesystems on Unix-like Systems Many different designs
- 39. Namespace: root directory “/”, followed by directories and files.
- 40. Consistency: “sequential consistency”, newly written data are immediately visible to open reads (if...)
- 41. Security: uid/gid, mode of files
- 42. kerberos: tickets Reliability: superblocks, journaling, snapshot more reliable filesystem on top of existing filesystem: RAID computer
- 43. Namespace Physical mapping: a directory and all of its subdirectories are stored on the same physical media. /mnt/cdrom
- 44. /mnt/disk1, /mnt/disk2, … when you have multiple disks Logical volume: a logical namespace that can contain multiple physical media or a partition of a physical media still mounted like /mnt/vol1
- 45. dynamical resizing by adding/removing disks without reboot
- 46. splitting/merging volumes as long as no data spans the split
- 47. Journaling Changes to the filesystem is logged in a journal before it is committed. useful if an supposedly atomic action needs two or more writes, e.g., appending to a file (update metadata + allocate space + write the data).
- 48. can play back a journal to recover data quickly in case of hardware failure. (old-fashioned “scan the volume” fsck anyone?) What to log? changes to file content: heavy overhead
- 49. changes to metadata: fast, but data corruption may occur Implementations: xfs3, ReiserFS, IBM's JFS, etc.
- 50. RAID Redundant Array of Inexpensive Disks
- 51. Different RAID levels RAID 0 (striped disks): data is distributed among disk for performance.
- 52. RAID 1 (mirror): data is mirrored on another disk 1:1 for reliability.
- 53. RAID 5 (striped disks with distributed parity): similar to RAID 0, but with a parity bit for data recovery in case one disk is lost. The parity bit is rotated among disks to maximize throughput.
- 54. RAID 6 (striped disks with dual distributed parity): similar to RAID 5 but with 2 parity bits. Can lose 2 disks without losing data.
- 55. RAID 10 (1+0, striped mirrors)
- 56. Snapshot A snapshot = a copy of a set of files and directories at a point in time. read-only snapshots, read-write snapshots
- 57. usually done by the filesystem itself, sometimes by LVMs
- 58. backing up data can be done on a read-only snapshot without worrying about consistency Copy-on-write is a simple and fast way to create snapshots current data is the snapshot.
- 59. a request to write to a file creates a new copy, and work from there afterwards. Implementation: UFS, Sun's ZFS, etc.
- 60. Should the file system be faster or more reliable?
- 61. But faster at what: Large files? Small files? Lots of reading? Frequent writers, occasional readers?
- 62. Block size Smaller block size reduces amount of wasted space
- 63. Larger block size increases speed of sequential reads (may not help random access)
- 65. Distributed Filesystems Support access to files on remote servers
- 66. Must support concurrency Make varying guarantees about locking, who “wins” with concurrent writes, etc...
- 67. Must gracefully handle dropped connections Can offer support for replication and local caching
- 68. Different implementations sit in different places on complexity/feature scale
- 69. DFS is useful when... Multiple users want to share files
- 70. Users move from one computer to another
- 71. Users want a uniform namespace for shared files on every computer
- 72. Users want a centralized storage system – backup and management
- 73. Note that a “user” of a DFS may actually be a “program”
- 74. Features of Distributed File Systems Different systems have different designs and behaviors on the following features. Interface: file system, block I/O, custom made
- 75. Security: various authentication/authorization schemes.
- 76. Reliability (fault-tolerance): can continue to function when some hardware fail (disks, nodes, power, etc).
- 77. Namespace (virtualization): can provide logical namespace that can span across physical boundaries.
- 78. Consistency: all clients get the same data all the time. It is related to locking, caching, and synchronization.
- 79. Parallel: multiple clients can have access to multiple disks at the same time.
- 80. Scope: local area network vs. wide area network.
- 81. NFS First developed in 1980s by Sun
- 82. Most widely known distributed filesystem.
- 83. Presented with standard POSIX FS interface
- 84. Network drives are mounted into local directory hierarchy Your home directory at CINC is probably NFS-driven
- 85. Type 'mount' some time at the prompt if curious client client client server
- 86. NFS Protocol Initially completely stateless Operated over UDP; did not use TCP streams
- 87. File locking, etc, implemented in higher-level protocols
- 88. Implication: All client requests have enough info to complete op example: Client specifies offset in file to write to
- 89. one advantage: Server state does not grow with more clients Modern implementations use TCP/IP & stateful protocols
- 90. RFCs: RFC 1094 for v2 (3/1989)
- 91. RFC 1813 for v3 (6/1995)
- 92. RFC 3530 for v4 (4/2003)
- 93. NFS Overview Remote Procedure Calls (RPC) for communication between client and server
- 94. Client Implementation Provides transparent access to NFS file system UNIX contains Virtual File system layer (VFS)
- 95. Vnode: interface for procedures on an individual file Translates vnode operations to NFS RPCs Server Implementation Stateless: Must not have anything only in memory
- 96. Implication: All modified data written to stable storage before return control to client Servers often add NVRAM to improve performance
- 97. Server-side Implementation NFS defines a virtual file system Does not actually manage local disk layout on server Server instantiates NFS volume on top of local file system Local hard drives managed by concrete file systems (ext*, ReiserFS, ...)
- 98. Export local disks to clients
- 99. NFS Locking NFS v4 supports stateful locking of files Clients inform server of intent to lock
- 100. Server can notify clients of outstanding lock requests
- 101. Locking is lease-based: clients must continually renew locks before a timeout
- 102. Loss of contact with server abandons locks
- 103. NFS Client Caching NFS Clients are allowed to cache copies of remote files for subsequent accesses
- 104. Supports close-to-open cache consistency When client A closes a file, its contents are synchronized with the master, and timestamp is changed
- 105. When client B opens the file, it checks that local timestamp agrees with server timestamp. If not, it discards local copy.
- 106. Concurrent reader/writers must use flags to disable caching
- 107. Cache Consistency Problem: Consistency across multiple copies (server and multiple clients)
- 108. How to keep data consistent between client and server? If file is changed on server, will client see update?
- 109. Determining factor: Read policy on clients How to keep data consistent across clients? If write file on client A and read on client B, will B see update?
- 110. Determining factor: Write and read policy on clients
- 111. NFS: Summary Very popular
- 112. Full POSIX interface means it “drops in” very easily.
- 113. No cluster-wide uniform namespace: each client decides how to name a volume mounted from an NFS server.
- 114. No location transparency: data movement means remount
- 115. NFS volume managed by single server Higher load on central server, bottleneck on scalability
- 116. Simplifies coherency protocols Challenges: Fault tolerance, scalable performance, and consistency
- 117. AFS (The Andrew File System) Developed at Carnegie Mellon University since 1983
- 118. Strong security: Kerberos authentication richer set of access control bits than UNIX: separate “administer”/“delete” bits, application-specific bits. Higher scalability compared with distributed file systems at the time. Supports 50,000+ clients at enterprise level.
- 119. Global namespace: /afs/cmu.edu/vol1/..., WAN scope
- 120. Location transparency: moving files just works.
- 121. Heavily use client side caching.
- 122. Support read-only replication. client client client server replica
- 123. Morgan Stanley's comparison: AFS vs. NFS Better client/server ratio NFS (circa 1993) topped out at 25:1
- 124. AFS: 100s Robust volume replication NFS servers go down, and take their clients with them
- 125. AFS servers go down, nobody notices (OK, RO data only) WAN file sharing NFS couldn't do it reliably
- 126. AFS worked like a charm Security was never an issue (kerberos4) http://www-conf.slac.stanford.edu/AFSBestPractices/Slides/MorganStanley.pdf
- 127. Local Caching File reads/writes operate on locally cached copy
- 128. The modified local copy is sent back to server when file is closed
- 129. Open local copies are notified of external updates through callbacks but not updated Trade-offs: Shared database files do not work well on this system
- 130. Does not support write-through to shared medium
- 131. Replication AFS allows read-only copies of filesystem volumes
- 132. Copies are guaranteed to be atomic checkpoints of entire FS at time of read-only copy generation (“snapshot”)
- 133. Modifying data requires access to the sole r/w volume Changes do not propagate to existing read-only copies
- 134. AFS: Summary Not quite POSIX Stronger security/permissions
- 135. No file write-through High availability through replicas and local caching
- 136. Scalability by reducing load on servers
- 137. Not appropriate for all file types ACM Trans. on Computer Systems, 6(1):51-81, Feb 1988.
- 138. Global File System
- 139. Lustre Overview Developed by Peter Braam at Carnegie Mellon since 1999 then Cluster File System in 2001, acquired by Sun in 2007
- 140. Goal: removing bottlenecks to achieve scalability Object-based: separate metadata and file data metadata server(s): MDS (namespace, ACL, attributes)
- 141. object storage server(s): OSS
- 142. client read/write directly with OSS Consistency: Lustre distributed lock manager
- 143. Performance: data can be striped
- 144. POSIX interface client client client OSS MDS OSS OSS MDS OSS OST
- 145. Key Design Issue : Scalability I/O throughput How to avoid bottlenecks Metadata scalability How can 10,000’s of nodes work on files in same folder Cluster Recovery If sth fails, how can transparent recovery happen Management Adding, removing, replacing, systems; data migration & backup
- 147. Metadata Service (MDS) All access to the file is governed by MDS which will directly or indirectly authorize access.
- 148. To control namespace and manage inodes
- 149. Load balanced cluster service for the scalability (a well balanced API, a stackable framework for logical MDS, replicated MDS)
- 150. Journaled batched metadata updates
- 151. Object Storage Targets (OST) Keep file data objects
- 152. File I/O service: Access to the objects
- 153. The block allocation for data obj., leading distributed and scalability
- 154. OST s/w modules OBD server, Lock server
- 155. Obj. storage driver, OBD filter
- 156. Portal API
- 157. Distributed Lock Manager For generic and rich lock service
- 158. Lock resources: resource database Organize resources in trees High performance node that acquires resource manages tree Intent locking: utilization dependent locking modes write-back lock for low utilization files
- 159. no lock for high contention files, write to OSS by DLM
- 160. Metadata
- 161. File I/O
- 162. Striping
- 163. Lustre Summary Good scalability and high performance Lawrence Livermore National Lab BlueGene/L has 200,000 clients processes access 2.5PB Lustre fs through 1024 IO nodes over 1Gbit Ethernet network at 35 GB/sec. Reliability Journaled metadata in metadata cluster
- 164. OSTs are responsible for their data Consistency
- 165. Other file systems worth mentioning Coda: post-AFS, pre-Lustre, from CMU
- 166. Global File System: shared disk file systems
- 167. PVFS by IBM
- 168. ZFS by Sun
Editor's Notes
- #4: memory: 1 GHz bus * 32 bit * 1B/8b = 4 GB/s 1MB / 4 GBps = 1/4 ms = 250 micro-seconds network: 1 Gbps = 100 MB/s 1MB / 100 MBps = 0.01 s = 10 ms
- #41: 1.NAL-Network Abstraction Layers 2.Drivers: ext2 OBD, OBD filter makes other file systems recognizable such as XFS, JFS, and Ext3 3.NIO portal API provides for interoperation with a variety of network transports through NAL 4.When a Lustre inode represents a file, the metadata merely holds reference to the file data obj. stored on the OST’s