• View
• Edit
• History
• Print

Distributed File Systems

A Distributed Computing Lecture by Steven Choy

Introduction

• Distributed file systems (DFS) support the sharing of information in the form of files throughout the intranet.
• A distributed file system enables programs to store and access remote files exactly as they do on local ones, allowing users to access files from any computer on the intranet.
• Clients should view a DFS the same way they would a centralized FS; the distribution is hidden at a lower level.
• Performance is concerned with throughput and response time.

Design Requirements and Issues

• Access transparency
• Location transparency
• Mobility transparency
• Performance transparency
• Scaling transparency
• Concurrent file updates is protected
• File replication to allow performance.
• Hardware and operating system heterogeneity.
• Fault tolerance
• Consistency
  • one-copy update semantics may be difficult to achieve in DFS.
• Security
• Efficiency

Transparencies

Access Transparency

• Client programs should be unaware of the distribution of files.

Location transparency

• Client program should see a uniform namespace. Files should be able to be relocated without changing their path name.
  • Users of a distributed system should not have to be aware of where a resource is physically located.

Mobility transparency

• Neither client programs nor system admin program tables in the client nodes should be changed when files are moved either automatically or by the system admin.
  • Relocation transparency - Should a resource move while in use, this should not be noticeable to the end user.

Performance transparency

• Client programs should continue to perform well on load within a specified range.

Scaling transparency

• Increase in size of storage and network size should be transparent

Concurrent transparency

• While multiple users may compete for and share a single resource, this should not be apparent to any of them.
• Concurrent file updates is protected (record locking).

Replication transparency

• If a resource is replicated among several locations, it should appear to the user as a single resource.

Failure transparency

• Always try to hide any failure and recovery of computing entities and resources.

Some Brief Definitions

• A DFS is a classical model of a file system distributed across multiple machines.
• The purpose is to promote sharing of dispersed files.
• The resources on a particular machine are local to itself. Resources on other machines are remote.
• A file system provides a service for clients.
• The server interface is the normal set of file operations: create, read, etc. on files.
• Clients, servers, and storage are dispersed across machines.
• Configuration and implementation may vary
  • Servers may run on dedicated machines,
  • Servers and clients can be on the same machines.
• Clients should view a DFS the same way they would a centralized FS; the distribution is hidden at a lower level.
• Performance is concerned with throughput and response time.

Naming

Naming is the mapping between logical and physical objects.

• Example: A user filename maps to <cylinder, sector>.
• In a conventional file system, it's understood where the file actually resides; the system and disk are known.
• In a transparent DFS, the location of a file, somewhere in the network, is hidden.
• File replication means multiple copies of a file; mapping returns a SET of locations for the replicas.

Location Transparency

• The name of a file does not reveal any hint of the file’s physical storage location.
• File name still denotes a specific, although hidden, set of physical disk blocks.
• This is a convenient way to share data.
• Can expose correspondence between component units and machines.

Location Independence

• The name of a file doesn’t need to be changed when the file’s physical storage location changes. Dynamic, one-to-many mapping.
• Better file abstraction
• Promotes sharing the storage space itself.
• Separates the naming hierarchy from the storage devices hierarchy.

Many distributed file systems are:

• Support location transparent systems
• Do NOT support migration; (automatic movement of a file from machine to machine.)
• Files are permanently associated with specific disk blocks.

Example: Andrew File System

• AFS as an Example
  • Is location independent.
  • Supports file mobility.
  • Separation of FS and OS allows for disk-less systems. These have lower cost and convenient system upgrades. The performance is not as good.

FS Naming Schemes

Three main approaches to naming files

• (1) Files are named with a combination of host and local name.
  • This guarantees a unique name.
    • Not location transparent nor location independent.
  • Same naming works on local and remote files. The DFS is a loose collection of independent file systems.
• (2) Remote directories are mounted to local directories.
  • So a local system seems to have a coherent directory structure.
  • The remote directories must be explicitly mounted. The files are location independent.
  • SUN NFS is a good example of this technique.
• (3) A single global name structure spans all the files in the system.
  • The DFS is built the same way as a local filesystem.
  • Location independent.

Implementation Techniques

• Can Map directories or larger aggregates rather than individual files.
• A non-transparent mapping technique:

• A transparent mapping technique:

• So when changing the physical location of a file, only the file identifier need be modified. This identifier must be "unique" in the universe.

Client Caching

• Reduce network traffic by retaining recently accessed disk blocks in a cache, so that repeated accesses to the same information can be handled locally.
• If required data is not already cached, a copy of data is brought from the server to the user.
• Perform accesses on the cached copy.
• Files are identified with one master copy residing at the server machine,
• Copies of the file are scattered in different caches.
• Cache Consistency Problem
  • Keeping the cached copies consistent with the master file.

Client Cache Location

• Caching is a mechanism for maintaining disk data on the local machine. This data can be kept in the local memory or in the local disk. Caching can be advantageous both for read ahead and read again.
• The cost of getting data from a cache is a few hundreds instructions; disk accesses cost thousands of instructions.
• The master copy of a file doesn’t move, but caches contain replicas of portions of the file.
• Caching behaves just like "networked virtual memory".
• What should be cached (blocks or files)?
  • Bigger sizes give a better hit rate
  • Smaller give better transfer times
• Caching on disk gives
  • Better reliability
• Caching in memory gives
  • The possibility of diskless work stations
  • Greater speed

Cache Update Policy

• Write Through
  • has good reliability
  • but the user must wait for writes to get to the server
  • used by NFS (Network File Systems)
• Delayed Write
  • write requests complete more rapidly
  • data may be written over the previous cache write, saving a remote write.
  • poor reliability on a crash
• Flush sometime later tries to regulate the frequency of writes.
• Write on close delays the write even longer.
• Which would you use for a database file? For file editing?

Cache Consistency

• How to determine that the client-cached data is consistent with what’s on the server?
• Client-initiated approach
  • The client asks the server if the cached data is OK. What should be the frequency of "asking"? On file open, at fixed time interval, ...?
• Server-initiated approach
  • Possibilities: A and B both have the same file open. When A closes the file, B "discards" its copy. Then B must start over
  • The server is notified on every open. If a file is opened for writing, then disable caching by other clients for that file.
  • Get read/write permission for each block; then disable caching only for particular blocks.

Caching vs. Remote Service

Stateful vs. Stateless Services

• Stateful: A server keeps track of information about client requests.
  • It maintains what files are opened by a client; connection identifiers; server caches.
  • Memory must be reclaimed when client closes file or when client dies.
• Stateless: Each client request provides complete information needed by the server (i.e., filename, file offset ).
  • The server can maintain information on behalf of the client, but it’s not required.
  • Useful things to keep include file information for the last N files touched
• Performance is better for stateful.
  • Don’t need to parse the filename each time, or "open/close" file on every request.
  • Stateful can have a read-ahead cache.
• Fault Tolerance: A stateful server loses everything when it crashes.
  • Server must poll clients in order to renew its state.
  • Client crashes force the server to clean up its encached information.
  • Stateless remembers nothing so it can start easily after a crash.

File Replication

• Duplicating files on multiple machines improves availability and performance.
• Placed on failure-independent machines
• The main problem is consistency
  • When one copy changes, how do other copies reflect that change?
  • Often there is a tradeoff: consistency versus availability and performance.
  • Atomic and serialized invalidation isn’t guaranteed (message could get lost / machine could crash.)

Extra Materials for Probing Further

• Beyond the File System - A presentation by Cal Henderson (the Flickr guy) about designing large scale file storage and serving. Slides as PDF's
• Lustre - Lustre® is a scalable, secure, robust, highly-available cluster file system. It is designed, developed and maintained by Cluster File Systems, Inc.
• List of File Systems - From Wikipedia, the free encyclopedia. "Distributed File Systems are also called network file systems. Normally many implementations have been made, they are location dependent and they have access control lists (ACLs).
• Andrew File System (AFS) - It is scalable and location independent, has a heavy client cache and uses Kerberos for authentication. Implementations include OpenAFS.
• Network File System (NFS) - It is the standard in UNIX-based networks. NFS may use Kerberos authentication and a client cache.
• Server Message Block (SMB) - It is originally from IBM but the most common version is modified heavily by Microsoft. It is the standard in Windows-based networks. SMB is also known as Common Internet File System (CIFS) or Samba file system. SMB may use Kerberos authentication.
• MogileFS - MogileFS is an open source distributed filesystem with some good properties and features such as "no single point of failure", "automatic file replication", and "flat Namespace".

Thanks for Reading

If you would rather like to have this lecture note in printed format, please click the print action link in the top right corner.

If you find any problem in this lecture note, please feel free to reach Steven by steven@findaway.hk