Distributed File Systems

Why a Distributed File System? exam

Imagine a lab of fifty computers. A simulation job on one machine produces results that a plotting job on another machine needs, and every student wants their home directory to follow them to whichever computer they sit at. If each computer only had its own disk, files would have to be copied around by hand, and nobody would know which copy is the latest. A distributed file system solves this by letting every computer use files that live elsewhere through the ordinary file calls it already knows.

Three reasons make a DFS worth building. Each one is something a single local disk cannot give you.

The catch is that a familiar interface does not make the network disappear. Three problems come with it. Network delays make a remote read slower than a local one. Partial failures happen, where one machine or link fails while the others keep running, so a client may be left waiting on a server that has crashed. And concurrent updates happen, where two clients change the same file at the same time. These three problems drive the requirements in 8.3.

Keep the scope in mind too. The DFS in this Part is a basic one: it only emulates a local file system for remote clients. It keeps no persistent replicas of files and gives no streaming guarantees (for video, say).

Storage systems and their properties exam

A DFS is one kind of storage system among many. CDK Fig. 12.1 compares them on four properties, and the comparison shows what is special about a DFS. The four columns ask simple questions:

• Sharing: can processes on different computers use the same data?
• Persistence: does the data survive after the program (or the machine) stops?
• Distributed cache/replicas: are copies of the data kept at more than one place? A cache is a nearby copy kept for speed; a replica is a full extra copy.
• Consistency maintenance: once the data has been updated, does some mechanism keep the copies in agreement?

In the consistency column, 1 means strict one-copy consistency (everyone sees exactly what they would see if only one copy existed), ✓ means slightly weaker guarantees, and 2 means considerably weaker guarantees.

Notice that the DFS row is ticked in every column: it is shared, persistent, cached at many places, and still kept (nearly) consistent. That combination is what makes it hard to build.

The consistency column needs a little unpacking. Where there is only one copy, as in RAM or a local file system, consistency is strict for free, because there is nothing to disagree with. DFSs such as NFS and AFS cache parts of files at the clients, so there are many copies, and they use mechanisms that approximate one-copy consistency. The Web also caches pages, in browsers and in proxies, but it never tells those caches when a page changes, so a client must check explicitly. That is fine for browsing, but useless for cooperative applications that need to see each other's updates. Remote objects, finally, are shared but neither persistent nor replicated; with a single copy, their consistency is strict.

Check yourselfWhy does the Web have no consistency maintenance, while a DFS does? (Self-test Q11)

Web caches, in browsers and proxies, are never notified of updates, so a client must re-check explicitly whenever it cares. A DFS, by contrast, approximates one-copy semantics with a mechanism: NFS validates cached blocks with timestamps (8.9), and AFS uses callbacks from the server (8.11).

Check yourselfWhich storage system in CDK Fig. 12.1 is shared, persistent and replicated, but gives only considerably weaker consistency (a "2")?

1. Distributed shared memory
2. Peer-to-peer storage, e.g. OceanStore
3. Remote objects, e.g. CORBA
4. A distributed file system, e.g. Sun NFS

B. Peer-to-peer storage is the only "2". DSM (A) is not persistent and scores ✓; remote objects (C) are not replicated and score 1; the DFS (D) scores ✓ in every column.

Check yourselfRemote objects are shared by many clients. Why do they nevertheless get strict consistency (1)?

Because they are neither persistent nor replicated: there is only a single copy of each object, at its server. With one copy, every client sees the same state, so consistency is strict without any extra mechanism.

File System Basics

Before distributing a file system, we need the vocabulary of an ordinary one: what a file is, how names work, and which calls programs make. The details matter, because the most important design decision in a DFS turns out to be which parts of the UNIX interface to keep and which to change.

Files, attributes and directories

A file has two parts. Its data is a sequence of bytes (data items) that can be read and written at any position. Its attributes describe the file and are kept in an attribute record. Some fields of the record are maintained by the system and cannot be updated by users: the file length, the timestamps and the reference count. Others can be updated by the owner: the owner itself, the file type and the access control list, for example the UNIX permissions rw-rw-r-- (owner and group may read and write, everyone else may only read).

A directory is a special file that maps text names to internal file identifiers. Because a directory can contain other directories, they form a hierarchy (a tree). A pathname such as /home/alice/notes.txt is a multi-part name that traces a route through that hierarchy, starting from the root or from the current directory. One file can have several names: the link call adds another, and the file's reference count records how many names it has. Names exist for users; internal identifiers exist so that the system can locate the object. Finally, metadata means everything stored to manage files: attributes, directories and any other persistent management data.

File system modules

CDK Fig. 12.2 splits a file system into six modules stacked in layers. Each module depends only on the modules below it, so each can be understood (and replaced) on its own. A DFS needs every one of them, plus two more things a local system never needed: client–server communication and distributed naming and location of files.

The UNIX file operations exam

A file descriptor (filedes) is a small integer that the kernel hands back when a process opens a file; the process then uses it, rather than the name, in every later call. The read-write pointer is the current position in the open file, which the kernel keeps for you. The standard UNIX interface is this:

UNIX is stateful, and why that matters

The interface above hides a lot of state, meaning information the kernel remembers between calls. For each process the kernel keeps an open-file list holding a read-write pointer for each open file. And access rights are checked once, at open; the result is remembered until close. Watch the hidden state at work in the copy program from the slides: it never mentions a position, yet every read continues where the last one stopped.

/* copy file1 to file2: a sketch of the slide's program */
fd1 = open("file1", READ);
fd2 = open("file2", WRITE);
while ((n = read(fd1, buf, 1024)) > 0)
    write(fd2, buf, n);
close(fd1);
close(fd2);

That last point is the whole lesson. If a DFS server copied this model, it would need per-client state: an open-file table with pointers and remembered access checks for every client. As 8.4 shows, the file service architecture deliberately avoids this.

Check yourselfA process calls read(fd, buf, 100) twice in a row on the same open file. Do the two calls return the same bytes? Why?

No (unless the file is shorter than 100 bytes and both hit the end). Each read advances the hidden read-write pointer by the count transferred, so the second call returns the next 100 bytes. The call does not name a position, so repeating it does something different. This is exactly why UNIX read is not idempotent (8.4).

Check yourselfWhen does UNIX check whether a process may read a file: on every read, or once? And what is the reference count for?

Once, at open; the result is remembered until close. The reference count records how many names (links) the file has; unlink deletes the file only when its last name is removed.

DFS Requirements exam

What exactly must a DFS promise so that remote files really do feel local, despite delays, failures and other users? CDK lists eight requirements. The first, transparency, comes in five forms, and those five are a favourite short-answer question (Tutorial 8 asked for them).

1 · Transparency

Transparency means hiding something from the user or programmer, here the fact that files are remote. Because the file service is usually the most heavily loaded service in an intranet, transparency has to be balanced against complexity and performance: each form costs something to provide. The five forms are these.

• Access transparency: there is one set of operations for local and remote files, so programs written for local files work unchanged on remote ones.
• Location transparency: there is a uniform name space. A file's name doesn't reveal where it is stored, and files can be relocated without changing their pathnames.
• Mobility transparency: files or whole volumes can move between servers without changing client programs or the client administration tables (such as a client's list of what is mounted where).
• Performance transparency: performance stays satisfactory while the load varies within a specified range.
• Scaling transparency: the service can grow incrementally, with more users, more files and a bigger network.

2 to 8 · The other requirements

2 · Concurrent file updates. One client's changes to a file must not interfere with other clients using the same file. Full concurrency control (the kind databases use) is costly, so most file services settle for UNIX-style advisory or mandatory file or record locking: a client locks a file, or a region of it, while it updates it. (An advisory lock only works if every program checks it; a mandatory lock is enforced by the system.)

3 · File replication. Keeping several copies of a file gives load sharing (requests spread over servers, which helps scalability) and fault tolerance (if one copy is lost, another remains). The price is that updates to the copies must be coordinated. Few systems replicate fully, but most cache files or blocks at clients, which is a limited form of replication.

4 · Hardware and OS heterogeneity. The service interfaces are defined so that clients and servers can run on different operating systems and hardware. This is a matter of openness: anyone can implement either side.

5 · Fault tolerance. The service must keep working through client, server and network failures without corrupting files. Two RPC styles from Part VI make this possible. With at-most-once semantics, a request is carried out once or not at all, never twice. With at-least-once semantics, the client simply resends until it gets a reply, so a request may run more than once; that is only safe if the operations are idempotent, meaning that doing one twice has the same effect as doing it once. Stateless servers help too, because a server that remembers nothing about its clients can restart after a crash with no recovery. Tolerating disconnection, though, needs replication.

6 · Consistency. UNIX gives one-copy update semantics: every process sees the contents it would see if only a single copy of the file existed, so a read after a write always sees that write. Caching or replication delays the spread of updates, so a DFS may deviate from this. Whatever model it promises, both cached data and cached attributes (length, timestamps) must follow it.

7 · Security. Like any file system, a DFS needs access control lists. Over a network it needs more: authentication of requests, so that the server checks the real identity of the user behind each request, and protection of messages by digital signatures and optional encryption (Part X).

8 · Efficiency. A DFS should offer at least the power, generality and performance of a local file system, and it should be convenient to administer.

Check yourself · Tutorial 8, Q1 · official answerName and explain the transparencies a DFS should address.

The tutors' official answer (examinable):

• Access: client programs don't know whether a file is local or remote.
• Location: client programs don't know where a file is stored, and files can be relocated without changing their pathnames.
• Mobility: neither client programs nor the system administration tables on client nodes need to change when files move.
• Performance: performance stays acceptable while the load varies within a specified range.
• Scaling: the service can be expanded without loss of performance.

Check yourself · Ed practice quiz 1A file is moved from server A to server B overnight, and clients use the same pathname with no configuration changes. Which transparency is this?

1. Access transparency
2. Location transparency
3. Performance transparency
4. Scaling transparency

B, location transparency, the best fit among the options: the pathname stays the same after relocation. The fact that no change to client administration tables is needed is also a mark of mobility transparency, but that option is not offered.

Check yourself · Ed practice quiz 2open() and read() work identically on local and remote files. Which transparency is this?

Access transparency: one set of operations for local and remote files.

Check yourself · Ed practice quiz 3A home directory moves to another server, and client mount tables and administration settings need no change. Which transparency is this?

Mobility transparency: the move needs no change to client programs or to client administration tables. That second part is what separates it from location transparency.

Check yourselfRequirement 5 offers two routes to fault tolerance. What are they, and what extra property makes the second one safe?

At-most-once semantics (the server makes sure a request is never carried out twice), or at-least-once semantics (the client keeps resending) combined with idempotent operations, so that a duplicate execution does no harm. A stateless server helps either way, because it restarts with nothing to recover.

The File Service Architecture exam

How should a file service be split up so that its server is simple, survives crashes, and still lets programs use the familiar UNIX calls? CDK's answer is an abstract model, the file service architecture, that underlies both NFS and AFS. It divides the work among three components, which separates concerns and, crucially, allows a stateless server. The official 2026 practice exam asks you to draw it.

The three components

Flat file service performs operations on file contents and attributes, and it identifies files only by a UFID. It allocates a fresh UFID on each Create, and it knows nothing about names. "Flat" means exactly that: no names, no directories, no hierarchy, just numbered files.

Directory service maps text names to UFIDs, and creates and updates directories. It is itself a client of the flat file service, because the directory files are stored there like any other file.

Client module runs on each client computer and presents one familiar API to applications, for example by emulating the UNIX calls, so applications see an ordinary file system. Behind that API it resolves pathnames by iterative Lookup calls (one name component at a time, 8.5), keeps file descriptors and read-write offsets locally, knows the network locations of the servers, and caches recently used blocks.

The flat file service interface

Six operations, every one naming its file by UFID. Notice that Read and Write carry an explicit position i; there is no open, no close and no file descriptor.

Differences from UNIX, made for fault tolerance exam

The flat file interface departs from UNIX in four ways, all of them for the sake of fault tolerance:

Idempotent operations and at-least-once RPC

An operation is idempotent if performing it twice (or many times) has the same effect as performing it once. Idempotent operations make at-least-once RPC safe: if a request times out, the client just sends it again, without needing to know whether the first copy was executed. The slide example is Write(f, 100, "ABC"): doing it twice leaves bytes 100–102 holding "ABC", exactly as doing it once does (assuming no other writes come in between). A blind retry of Create, by contrast, makes an extra file.

Stateless servers

Stateless means the server keeps no per-client state between requests: no open-file table, no offsets, no remembered access checks. Every request carries everything needed to carry it out. As a result, a server that crashes can restart with nothing to rebuild, and clients just carry on sending requests. Stateless does not mean the server stores nothing: file contents, directories and attributes are of course stored persistently on its disks.

Access control without server state

Statelessness makes access control harder. In UNIX the check happens once, at open, and the result is remembered; a stateless server cannot remember it. Meanwhile the server's RPC interface is an open entry point that anyone on the network can call, and identities can be forged. That leaves two stateless options:

1. Capability. Access is checked once, when a name is mapped to a UFID, and the server returns a capability: an unforgeable token that says "the holder may do these operations on this file". The client presents the capability with each later request.
2. Identity on every request. The user's identity travels with every request, and the server checks the access rights on every operation. This is the more common option, and it is the one used by NFS and AFS.

Worked exam question

Check yourself · Self-test Q2Why does the flat file service have no open/close? Which of its operations are idempotent?

It needs no open or close because UFIDs are used directly and every Read and Write carries an explicit position, so there is no server-side pointer or open-file table. The server is therefore stateless, and a crash and restart need no recovery.

Read, Write, Delete, GetAttributes and SetAttributes are idempotent, so at-least-once retries are safe: Write(f, 100, "ABC") done twice has the same effect as done once. Create is not: a retry makes an extra file. UNIX read and write are not idempotent either, because a repeat advances the hidden pointer.

Check yourself · Self-test Q3How is access control enforced by a stateless server?

A stateless server cannot store access-check results, so it has two options. It can issue a capability when a name is mapped to a UFID, which the client presents with later requests; or the user ID is sent with every request and checked every time, which is what NFS and AFS do. Neither stops forged IDs on its own; that requires authentication, such as Kerberos.

Check yourselfWhich statement about a "stateless" file server is correct?

1. It stores no data on disk, only in memory.
2. It keeps no per-client state, such as open files, offsets or access checks, between requests.
3. It never caches anything.
4. Its operations are all idempotent, including Create.

B. Stateless refers only to per-client session state. The server certainly stores files persistently (A is wrong), may cache disk blocks for its own speed (C), and Create is the one non-idempotent flat file operation (D).

Check yourselfA client sends Delete(f), the reply is lost, and it retries. What happens, and is Delete idempotent?

The file is already gone after the first execution; the retry leaves it gone. The effect on the stored files is the same as doing it once, so the guide counts Delete among the idempotent operations. (The retry may report differently; NFS's version of this, remove, is handled with a duplicate-request cache in 8.7.)

The Directory Service and Pathnames

The flat file service only understands UFIDs, but people use names like /home/tawfiq/results.csv. Something has to turn one into the other. That is the job of the directory service, working together with the client module, which walks a pathname one name at a time.

The directory service interface

The directory service has four operations. Each takes a directory, Dir, as its first argument; Dir is itself a UFID, because a directory is just a file.

Compare these with UNIX: AddName plays the part of link, and UnName the part of unlink, including deleting the file when its last name goes.

Resolving a pathname, step by step

Because Lookup translates only one name, a full pathname is resolved one component at a time, starting from the root directory's well-known UFID, R (every client module knows it in advance). To resolve /home/tawfiq/results.csv and read its first 4096 items, the client module makes four RPCs.

The Java sketch below shows the same loop. It is short, but every line matters.

import java.io.IOException;

interface DirectoryService {
    long lookup(long dirUfid, String name) throws IOException; // one-step RPC
}

class ClientModule {
    static final long ROOT_UFID = 1L;            // well-known UFID of "/"
    private final DirectoryService ds;
    ClientModule(DirectoryService ds) { this.ds = ds; }

    // Resolve a pathname by iterative single-component lookups
    long pathLookup(String path) throws IOException {
        long current = ROOT_UFID;
        for (String part : path.split("/")) {
            if (part.isEmpty()) continue;        // skip leading "/" and "//"
            current = ds.lookup(current, part);  // next step uses previous UFID
        }
        return current;
    }
}

• interface DirectoryService describes the remote directory service as the client sees it. Its lookup is the one-step RPC: one directory UFID and one name in, one UFID out. It throws IOException because any remote call can fail.
• ROOT_UFID = 1L is the root's well-known UFID: hard-wired, so resolution has somewhere to start. (UFIDs are modelled as long here; real ones are longer.)
• current = ROOT_UFID starts the walk at /.
• path.split("/") breaks /home/tawfiq/results.csv into "", "home", "tawfiq", "results.csv". The leading slash produces an empty first piece, and a doubled slash would produce another, so if (part.isEmpty()) continue; skips them.
• current = ds.lookup(current, part) is the heart of the method: each RPC is made in the directory found by the previous one, exactly as in the stepper.
• The returned current is the file's UFID, ready for Read or Write on the flat file service.

The hierarchic file system

The directories form a tree whose root sits at a well-known UFID. A file gets multiple names through AddName, with the reference count tracking how many it has. A type attribute marks each object as either a file or a directory, which lets resolution check that every component of a pathname except the last is a directory: you cannot Lookup inside a plain file.

File groups

One server cannot hold every file in a large organisation, so the service must spread files over many servers, and clients must know where to send each request. The tool for this is the file group: a collection of files located on one server, much like a UNIX filesystem (volume) that can be mounted. Groups can move between servers, but a file never changes group.

Each UFID contains a group ID, so the client module can tell from the UFID alone which group a file is in, and route the request to the server holding that group. Group IDs must be globally unique. One way to achieve that is to combine the 32-bit IP address of the host that created the group with a 16-bit date. But that IP address must not be used to locate the group, because the group may since have moved. Instead, the service keeps a separate map from groups to servers.

Check yourselfHow many directory service calls does the client module make to resolve /a/b/c/d.txt, starting from the root's UFID? What does each use as its first argument?

Four Lookup calls, one per component: Lookup(R,"a"), then Lookup(A,"b"), Lookup(B,"c") and Lookup(C,"d.txt"). Each uses the UFID returned by the previous call as its directory argument; the first uses the root's well-known UFID.

Check yourselfA file has names in two directories. Someone calls UnName on one of them. Is the file deleted?

No. UnName removes that entry and decrements the reference count from 2 to 1. The file is deleted only when the count reaches zero, i.e. when its last name is removed.

Check yourselfA group ID contains the IP address of the host that created the group. Why not just send requests for the group to that address?

1. IP addresses are too short to be unique.
2. The group may have moved to another server since it was created.
3. The date field would be ignored.
4. Clients are not allowed to see the UFID.

B. Groups can move between servers, so the creator's address only serves to make the ID unique. Location comes from the separate group-to-server map.

Sun NFS: Architecture exam

The abstract model shows the shape of a good file service; Sun NFS (Network File System) is the real system built on that shape. The problem it had to solve is practical: make remote files work with unchanged UNIX programs, on many kinds of computer, from a server that survives crashes. Its answer has three key parts: a kernel layer called the VFS, opaque file handles, and a stateless server.

NFS appeared in 1985 and was the first file service designed as a product. Its interfaces are public, version 3 is an Internet standard (RFC 1813), and it runs on almost every operating system. The NFS protocol is a set of RPCs for operating on a remote file store. The roles are symmetric: any host can be a server, exporting its own files, and a client, mounting other hosts' files, at the same time. NFS follows the abstract model of 8.4 closely; CDK describes the UNIX implementation of version 3.

The NFS client and server modules both sit in the kernel. They communicate by Sun RPC, which was built for NFS, over either UDP or TCP. A port mapper on each host lets clients bind to RPC services by name: a client asks it which port, say, the NFS service is listening on. The RPC interface is open: any process can send requests, and valid requests with valid credentials are executed. Signed credentials and encryption are optional extras.

The VFS and v-nodes

The virtual file system (VFS) is a kernel layer that integrates local and remote file systems into one name space. It is how NFS achieves access transparency: every file call goes through the VFS, which routes it to the right module (the UNIX file system, the NFS client, or another file system) and translates between NFS file identifiers and local ones.

The VFS keeps two kinds of record. There is one VFS structure per mounted file system, linking that file system to the local directory it is mounted on. And there is one v-node per open file (a "virtual i-node"), flagged as either local, in which case it holds a reference to the file's i-node (the UNIX on-disk record of a file), or remote, in which case it holds the file's NFS file handle.

File handles

The file handle is NFS's UFID: the identifier the server uses for a file. It is opaque to clients, meaning clients never look inside it; they just hand it back with each request. In UNIX it is built from three fields.

Where do handles come from? A client gets its first handle when it mounts a file system (8.8). After that, lookup, create and mkdir return further handles, and every other operation takes a handle as an argument.

Why the NFS client lives in the kernel

The NFS client module could have been a user-level library linked into programs. It is in the kernel instead, for three reasons:

The client caches remote blocks in the same buffer cache as local disk I/O. But many hosts may cache the same file, and that raises a consistency problem, which 8.9 deals with.

Check yourself · Self-test Q4Explain the VFS and v-nodes. What is in an NFS file handle?

The VFS integrates local and remote file systems, routes each call to the right module, and in doing so gives access transparency. It keeps one VFS structure per mount and one v-node per open file; a local v-node refers to an i-node, while a remote one holds a file handle.

An NFS file handle consists of the filesystem identifier, the i-node number and the i-node generation number. It is opaque to clients, and the generation number stops reused i-nodes from being reached by stale handles.

Check yourselfThe VFS lets unchanged programs open remote files with the usual system calls. Which transparency does that provide?

1. Location transparency
2. Access transparency
3. Mobility transparency
4. Replication transparency

B, access transparency: the same operations work on local and remote files. Location transparency (A) is a separate matter that depends on how clients mount file systems (8.10), a classic near-miss.

Check yourselfGive two operations that hand the client a new file handle, and say where the very first handle comes from.

lookup, create and mkdir all return handles. The very first handle comes from mounting: the mount protocol returns the handle of the mounted directory.

NFS Operations and Security

What can an NFS client actually ask the server to do, and how does a server that remembers nothing decide whether to allow it? This section lists the server interface, shows where NFS bends the "everything is idempotent" rule, and follows the identity check from plain UIDs to Kerberos.

The server interface (version 3, simplified)

Unlike the abstract model, NFS combines directory and file operations in one service. For example, a single create(dirfh, name, attr) → newfh, attr both creates a file and enters its name in a directory (dirfh is the directory's file handle). Most operations correspond roughly to one in the abstract model:

Idempotency, and where it breaks

As in the abstract model, explicit offsets keep read and write idempotent, so the client can simply retry them after a timeout. The slides add a nuance, though: some operations, such as remove and rename, are not naturally idempotent. A retried remove whose first attempt succeeded would report "no such file", so the client would be told that an operation failed when it actually worked. Servers therefore keep a duplicate-request cache: a record of recent requests and the replies sent, so a retransmitted request gets the original reply instead of being executed again.

Access control and authentication

Because the server is stateless, it cannot remember an access decision from an earlier call. So it checks the user's identity against the file's permissions on every request, the "identity on every request" option from 8.4. To make this possible, Sun RPC sends the user's UNIX UID (user ID) and GID (group ID) with each call.

The loophole is that nothing proves those numbers are true. Any process that can send RPCs can put whatever UID it likes in a request, and the server cannot tell the difference.

The fixes were DES-encrypted credentials and, later, Kerberos (Part X), a system in which a trusted server issues users cryptographic tickets that prove who they are. But Kerberized NFS does not send a full Kerberos ticket with every request: doing so for every stateless request was too costly. It uses a hybrid scheme instead.

Check yourselfNFS read and write are idempotent. Why does an NFS server still need a duplicate-request cache?

Because some operations, such as remove and rename, are not naturally idempotent. A retried remove whose first attempt succeeded would fail with "no such file". The duplicate-request cache lets the server recognise the retransmission and return the original reply.

Check yourselfHow does a stateless NFS server enforce access control, and what is the loophole?

Sun RPC carries the user's UID and GID with every call, and the server checks them against the file's permissions on every request. The loophole: any process can forge the UID in a request. The fixes are DES-encrypted credentials and, later, Kerberos.

Check yourselfKerberized NFS does not attach a Kerberos ticket to each request. What does it do instead, and under what assumption is that safe?

Full Kerberos authentication happens once, at mount time. The server records the user's ID and the client's address with the mount, and accepts later requests only if their UID and sender address match. Per-request tickets were too costly for stateless requests. It is safe if only one user at a time logs in to each client.

Mounting and Pathname Translation exam

Before a client can use any remote file, it needs a starting point: the file handle of some remote directory, glued into its own directory tree. That is what mounting does. Once a remote directory is mounted, the client still has to turn a long pathname into a file handle, and NFS makes it do that one component at a time. This section covers both.

The mount service

Each server runs a user-level mount service. Its configuration file, /etc/exports, lists the file systems the server is willing to export and which hosts may mount each one. The slide example gives every host on the 192.168.1.x subnet read-write access to /store:

/store   192.168.1.0/255.255.255.0(rw)

On the client, a modified mount command names three things: the remote host, the remote directory (which can be any subtree of an exported file system) and the local mount point, the local directory where the remote one should appear. The mount protocol, an RPC to the mount service, returns the remote directory's file handle, and this is passed to the VFS and the NFS client. From then on, programs open files under the mount point without knowing they are remote.

Hard mounts and soft mounts exam

A remote file system can be mounted in one of two ways. They differ only in what happens when the server is unavailable. Hard mounts favour transparency, while soft mounts let programs handle the failure themselves.

Check yourself · Sample exam 2026 Q1(9) · worked answer, not officialMultiple choice: applications should receive an error after a few retries when the NFS server is unreachable, rather than wait. How should the remote file system be mounted?

Soft-mounted (option (a) on the paper). With a hard mount, the process blocks while the client retries until the server is reachable again, which is exactly the "wait" the question rules out.

The automounter

Mounting everything at boot is wasteful, and it fails if a server happens to be down. The automounter mounts a remote directory on demand, when an empty mount point is first referenced. It holds a table of mount points, each listing one or more servers.

1. A program references the empty mount point /usr/lib for the first time.
2. The automounter looks up the mount point in its table and probes all the listed servers.
3. Server B is the first to respond.
4. The automounter mounts from B. After several minutes without use, it unmounts the directory again.

If the servers listed for a mount point hold identical copies (for example of /usr/lib), this gives read-only replication, plus some fault tolerance (a dead server simply doesn't answer) and some load balancing (a busy server answers late).

Pathname translation: one lookup per component

Why can't the client just send the whole path /usr/students/jon/notes.txt to a server and get back a handle? Because the path may cross a client mount point (or follow a symbolic link) into another server, and only the client knows its own mounts. The server has never heard of the client's /usr.

The client therefore translates the path iteratively, sending one lookup per component, each using the handle returned by the previous one. Caching lookup results hides most of the cost. When nothing is cached, though, /usr/users/jim/code/xyz.c needs 5 lookups. Here is the iteration for a file under the mounted directory, followed by a read.

Check yourself · Self-test Q8Why is NFS pathname translation iterative at the client? How many lookups does /usr/users/jim/code/xyz.c need?

The path may cross client mount points or symbolic links onto other servers, and only the client knows its mounts, so the server cannot translate it in one go. The client sends one lookup per component, each using the previous handle, which for this path means 5 lookups (usr, users, jim, code, xyz.c) when nothing is cached. Caching lookup results limits the cost, but lookup is still about 50% of server calls.

Check yourself · Self-test Q5Hard mounts vs soft mounts?

With a hard mount, the process blocks and the client retries forever. This is transparent, since processes carry on after the server restarts, but applications hang during outages. It is the common choice.

With a soft mount, an error is returned after a few retries. Applications can then recover, but many don't check errors.

Check yourselfThe automounter lists three servers holding identical copies of /usr/lib. What does that give, and what does it not give?

It gives read-only replication, with some fault tolerance and some load balancing, because the client mounts from whichever server answers first. It does not give replication of updates: nothing keeps writable copies consistent.

Caching in NFS exam

Without caching, every read and write would cross the network, and NFS would be far slower than a local disk. With caching, there are copies of the same data in many places, and two different problems appear. At the server, the question is persistence: when the server says "written", is the data safe on disk if the server then crashes? At the clients, the question is consistency: does a client's cached copy still match what other clients have written? Keep the two apart; mixing them up is a classic way to lose marks.

Server caching

UNIX background. NFS server caching builds on ordinary UNIX caching. The kernel keeps a buffer cache of disk pages in memory. Read-ahead fetches the pages after those just read, betting that the program will want them next. Delayed write writes a modified page to disk only when its buffer is evicted (thrown out of the cache to make room). And sync flushes the whole cache to disk every 30 s. All of this works safely because there is only one cache and nothing bypasses it.

On an NFS server, caching reads causes no problems. Caching writes needs care, because clients must be sure their data is persistent if the server crashes. Why does that matter so much? Because of independent failures: the client keeps running after the server crashes, and it may act on writes it believes are safe (deleting its own copy, say, or telling the user "saved"). There are two policies. Write-through gives the guarantee on every reply but is slow; commit, added in NFS version 3, keeps the benefit of the cache but delays the guarantee.

Check yourself · Sample exam 2026 Q1(10) · worked answer, not officialMultiple choice: once the NFS server has replied to a write, the data must survive a server crash, even if the client never sends a commit request. Which server caching policy is this?

Write-through (option (d) on the paper), which writes the data to disk before replying. Under the commit policy the data is safe only after a commit request, and a delayed write gives no guarantee at all.

Check yourself · Self-test Q7Compare write-through and commit server caching.

With write-through, the data is on disk before the server replies. This is safe, but slow, with heavy disk I/O. With commit, the data stays cached until the client commits it, which it does on close. This is fast, but needs extra messages, and uncommitted data can be lost. Either way, the policy is needed because client and server fail independently.

Client caching exam

The NFS client caches the results of read, write, getattr, lookup and readdir. That creates the consistency problem from 8.3: when one client writes a file, other clients' cached copies are not updated. And reads are affected too: if another client writes between two of your reads, your second read may return stale data from your cache. NFS has no callbacks (the server never tells clients that something changed), so clients poll the server to check their cached data.

To make polling cheap, each cache entry carries two timestamps:

• Tc is the time when the entry was last validated (checked against the server) by this client.
• Tm is the time when the block was last modified at the server. The client holds the value it saw, Tm_client; the server holds the current value, Tm_server.

An entry is valid at time T if:

Validation happens on every use of an entry, in a fixed order. The cheap test comes first:

1. The client tests T − Tc < t locally, with no network I/O. If it is true, the client uses the entry.
2. Otherwise, it sends a getattr to fetch Tm_server. (The server keeps one Tm for a file's data blocks and another for its attributes.)
3. If Tm_client = Tm_server, the entry is valid, and the client sets Tc = T (the entry has just been re-validated).
4. If they differ, the client invalidates the entry and re-fetches the data from the server.

A worked example (from the slides, t = 10 s)

Client A has cached a value, and client B then updates it at the server. The moment to watch is A's read at 6 s.

So staleness is bounded by t, and choosing t trades consistency against traffic. A small t gives near one-copy semantics, at the cost of more getattr calls. Solaris adapts t per file between 3 and 30 s, depending on how often the file is updated, and uses 30–60 s for directories, which carry a lower risk of concurrent update.

NFS cuts the validation traffic in three more ways:

• A new Tm_server is applied to all cached blocks of the file, not only the block being checked, so one getattr re-validates (or invalidates) the whole file.
• Attributes are piggy-backed on every operation's reply (recall that read returns attributes), so every reply refreshes Tm for free.
• As just described, t adapts per file.

Writes and bio-daemons

Writes work differently. A modified page is marked dirty and flushed to the server asynchronously when the file is closed or at a client sync. This mimics local write-back; it is not the same thing as persistence at the server, which is the server policy's job (write-through or commit, above).

Bio-daemons (block I/O daemons) are background processes that keep the client from blocking on the network. They do read-ahead, fetching the next block after each read, and delayed write, sending a block once it fills. They help performance but are not required for correctness.

Past exam question

Check yourselft = 8 s. Client A validated a cached block at Tc = 20 s. Client B's update reaches the server at 23 s. A reads at 26 s and again at 29 s. What does each read return, and does it contact the server?

At 26 s: 26 − 20 = 6 < 8, so the local test passes. A uses its cache with no network I/O and returns the old (stale) value.

At 29 s: 29 − 20 = 9 ≥ 8, so A sends getattr. Tm_server changed at 23 s, so it no longer equals Tm_client; A invalidates the entry, fetches B's new value, and sets Tc = 29.

Check yourselfWhy does the NFS client test T − Tc < t before comparing Tm_client with Tm_server?

1. Because the Tm comparison is less accurate.
2. Because the first test is local and free, while the second needs a getattr round trip.
3. Because the server requires it.
4. Because Tc is always larger than Tm.

B. The time test needs no network I/O, so doing it first avoids a server round trip on most accesses. Only when it fails is the expensive getattr made.

Check yourselfAn administrator halves t on every client. What gets better, and what gets worse?

Better: consistency. Readers can now be stale for at most half as long, closer to one-copy semantics. Worse: load. Entries expire twice as often, so every client sends more getattr calls, and the server handles more traffic.

NFS Evaluated exam

How well does NFS actually meet the eight requirements of 8.3? This section first looks at performance, where measurements found two surprises, and then scores NFS requirement by requirement. The scorecard is a ready-made answer to "evaluate NFS".

Performance and optimisations

NFS moves data in 8 KB blocks, and UDP packets are extended to 9 KB so that one block fits in one RPC; version 3 can negotiate larger sizes. Every operation also acts as an implicit getattr, because replies carry attributes.

Measurements found two problems: frequent getattr calls (the cache validation of 8.9), and slow write-through writes. Writes are only about 5% of calls, and the slow writes were later fixed by commit. Meanwhile lookup makes up about 50% of server calls, because of step-by-step pathname translation. Even so, large NFS servers sustain very high loads.

The scorecard

A few lines deserve unpacking. Location transparency is not enforced because each client mounts remote file systems wherever it likes, so the same file can have different pathnames on different clients; a uniform name space is achievable only if every client uses the same mount tables. Mobility transparency is absent because moving a file system to another server means editing every client's mount table. Scalability is limited by hot-spot files: a single heavily used file lives on one server, and adding servers doesn't spread its load. Fault tolerance is acceptable: the server is stateless and its operations idempotent, so clients resume after a server restart, and client crashes don't affect the server at all; hard mounts, however, hinder graceful handling. NIS (Network Information Service) is a companion service that replicates simple key-value databases on a master–slave basis.

Check yourself · Self-test Q9Evaluate NFS against the DFS requirements.

NFS provides access transparency, but location transparency is not enforced and there is no mobility transparency. Scalability is good but limited by hot spots, and replication is read-only. Heterogeneity is good, and so is fault tolerance, because the server is stateless and its operations idempotent. Consistency is approximate and tunable, security is available via Kerberos, and efficiency is good.

Check yourselfTwo clients mount the same NFS export, one at /usr/staff and the other at /home. Which transparency does this show NFS lacks?

1. Access transparency
2. Location transparency (a uniform name space)
3. Performance transparency
4. Heterogeneity

B. The same file has different pathnames on the two clients, so there is no uniform name space. NFS still has access transparency (A): both clients use the same calls.

Check yourselfAbout what share of NFS server calls are lookups, and why so many?

About 50%, because pathnames are translated one component per RPC at the client (8.8). Writes, by contrast, are only about 5% of calls.

AFS, and NFS vs AFS

NFS pays for its stateless server with constant polling: every client keeps asking "has this changed?". The Andrew File System (AFS), from Carnegie Mellon University (CMU), asks the opposite question: what if the server remembered who has a copy and told them when it changed? AFS also gives transparent access to UNIX files, but its main design goal is scalability, which it achieves by minimising client–server interaction. AFS is not on the slides (CDK §12.4), so it is low priority; you need just enough to answer "compare NFS and AFS".

How AFS works

• Whole-file serving and caching. AFS transfers whole files (in 64 KB chunks in AFS-3) and caches them on the client's local disk, where the cache survives reboots.
• Venus and Vice. Venus is the client process, playing the part of the client module; Vice is the server process. The kernel intercepts open and close on shared files (those under /cmu) and passes them to Venus.
• Volumes. Files are grouped into volumes (AFS's file groups) and identified by a 96-bit fid (file identifier).
• Open, read/write, close. When open is called and there is no valid cached copy, Venus fetches the whole file. read and write then use the local copy, with no server contact. On close, a modified file is sent back to the server.

The callback promise

When Vice sends a file to a client, it gives a callback promise: a promise to notify that client if another client updates the file. When a client stores a new version, Vice sends a callback to every other holder, and their promises become cancelled. So on open, Venus uses its cached copy if the promise is still valid, and re-fetches the file if the promise was cancelled. In case a callback message was lost, promises are also revalidated after a client reboot, or after T minutes in which nothing has been heard from the server.

Semantics: last close wins

Consistency is enforced only at open and close. If there are concurrent writers on different hosts, the last close wins, and the other updates are silently lost.

NFS vs AFS

The key contrast is in how copies are checked. NFS keeps its server stateless and makes clients poll; AFS keeps callback lists at the server so that it can notify clients only when a file changes.

AFS's design rests on assumptions about typical UNIX workloads: mostly small files, reads about 6× more common than writes, mostly sequential access, little sharing, and bursty reuse (a file used now is likely to be used again soon). Under those assumptions whole-file caching is a big win. It is therefore unsuitable for databases, which are large, shared and updated in small pieces.

Check yourself · Self-test Q10Compare the NFS and AFS consistency mechanisms. Which scales better?

NFS has a stateless server and caches blocks, and its clients poll with timestamps (Tc, Tm, t). AFS caches whole files on disk and relies on callback promises from a stateful server, which gives open/close semantics in which the last close wins.

AFS scales better. It contacts the server only on change, and opening a valid cached file needs no server contact at all.

Check yourselfTwo users on different AFS workstations edit the same file at the same time. A closes first, then B. What does the server end up with?

B's version. Each close stores the whole file, so the last close wins and A's changes are silently lost. AFS only enforces consistency at open and close.

Check yourselfWhich is true of AFS but not of NFS?

1. The server keeps no per-client state.
2. Clients cache blocks in memory.
3. The server keeps callback lists and notifies clients when a file changes.
4. Clients send getattr to validate their caches.

C. A, B and D describe NFS. AFS's server is stateful precisely because it remembers who holds each file.

Later developments (brief)

The designs above have since been extended in several directions, which you only need to know by name.

Tutorial 8, Dropbox and Pitfalls

The Week 9 tutorial (Tutorial 08) asked four questions and then looked at how a real file-synchronisation service, Dropbox, is built. The answers below are the tutors' official answers from Ed, which staff confirmed are examinable. Questions 2–4 and the Dropbox case study go beyond the lecture material earlier in this Part. Question 1 (the transparencies) is answered in 8.3.

T8-2 and T8-3 · Absolute names vs mount points

An absolute name gives a file's complete address, server plus path, in the form <machine name : path name>, for example fileserver3:/data/report.txt. Mount points are what Sun NFS uses: the client creates local names that refer to remote locations, and at boot each local name is bound to its remote name. The OS keeps a mount table saying which server and path each mount point maps to.

T8-4 · Remote access vs client caching

Compare two DFS designs: (A) every read() and write() goes to the server by RPC, and clients do no caching; (B) clients cache files and perform file operations locally.

Case study: Dropbox

The model. A client daemon (a background process) watches a local "sync" folder. Changes are uploaded automatically, and new versions are downloaded automatically. Every user's computer holds a complete copy of their Dropbox.

Scale and goals. Dropbox has over 100 million users and handles 1 billion files a day. It needs a small client that uses few resources, must cope with a possibly low-capacity network link to the user, and needs a scalable back end. 99% of the code is Python.

Traffic. Traffic flows only when something changes, and each change is one upload and one download. The read/write ratio is therefore about 1:1, unlike typical read-heavy services (compare AFS's 6:1 assumption). Compression reduces the traffic.

Download takes three steps:

1. The client periodically polls the metaserver, listing the files it knows about.
2. The metaserver returns information on new files.
3. The client fetches the blocks it lacks from the blockserver and rebuilds the file using ordinary local system calls (open(), write()).

Streaming sync: other clients start downloading blocks before the upload has committed, so large files sync sooner. LAN sync: clients broadcast on the local network by UDP; a response leads to a TCP connection, and the blocks are then pulled over HTTP from the peer instead of from the cloud.

Architecture evolution

The Dropbox back end went through five versions as it grew, and each step illustrates a concept from the course.

In words: v1 shows that a single server is a bottleneck and a single point of failure. In v2 the server ran out of disk, so file data moved to Amazon S3, a key-value store, and the overloaded database moved to its own machine; files were split into 4 MB chunks with a hash per chunk, and identical chunks stored once. v3 replaced polling by a notification server (push instead of pull) and split metadata from data. v4 put more servers behind a load balancer, had blockservers send RPCs to metaservers instead of accessing the database, and put memcache in front of the database (caching, Part II). v5 introduced a two-level hierarchy of notification servers, about 1 million connections per server: hierarchy for scalability (Part I).

Check yourself · Ed practice quiz 4Files are referenced as <server_name>:<path>, hard-coded in clients, so moving files breaks references. What naming scheme is this, and what does it lack?

Absolute naming; it provides no location transparency.

Check yourself · Ed practice quiz 5Every read() and write() goes to the server by RPC with no caching: consistent, but slow under load. What kind of design is this?

A stateless RPC-based DFS with no caching (design (A) of T8-4): it gives a consistent view, but performance suffers from constant network access and server load.

Check yourselfWhy does Dropbox's upload send hashes to the metaserver before sending any blocks?

So the metaserver can reply "need blocks" only for hashes it doesn't already know. Blocks with known hashes are never re-sent: this deduplication saves traffic on the user's possibly slow link.

Common pitfalls