Adding an Uncacheable Attribute to NFSv4.2

Status of This Memo

This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.¶

Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.¶

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."¶

This Internet-Draft will expire on 22 September 2025.¶

Copyright Notice

Copyright (c) 2025 IETF Trust and the persons identified as the document authors. All rights reserved.¶

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License.¶

Table of Contents

1. Introduction

With a remote filesystem, the client typically caches both directory entries (dirents) and file contents in order to improve performance. Several assumptions are made about the rate of change in the dirents and the number of clients trying to concurrently access a file. With NFSv4.2, this could theoretically be mitigated by directory delegations for the dirents and practically by file delegations for the file contents.¶

There are prior efforts to bypass both the dirent and file caching. Access Based Enumeration (ABE) is used in Server Message Block (SMB) [SMB2] protocol to effectively limit the namespace visibility per user. In Highly Parallel Computing (HPC) workloads, file caching is bypassed in order to achieve consistent work flows.¶

This document introduces the uncacheable attribute to NFSv4.2 to implement ABE and to bypass file caching on the client. As such, it is an OPTIONAL to implement attribute for NFSv4.2. However, if both the client and the server support this attribute, then the client MUST follow the semantics of uncacheable.What about mixed modes?¶

A client can easily determine whether or not a server supports the uncacheable attribute with a simple GETATTR on any dirent. If the server does not support the uncacheable attribute, it will return an error of NFS4ERR_ATTRNOTSUPP.¶

The only way that the server can determine that the client supports the attribute is if the client sends either a GETATTR or a SETATTR with the uncacheable attribute.¶

While some argument could be made to introduce two new attributes, the functionality of the uncacheable attribute dictates which cache is to be bypassed. As ABE is concerned with walking the namespace, it is only applicable to be acted on dirents which are of type attribute value of NF4DIR. And as bypassing file caching is file based, it is only applicable for dirents which are of type attribute value of NF4REG.What about the other file types?¶

Using the process detailed in [RFC8178], the revisions in this document become an extension of NFSv4.2 [RFC7862]. They are built on top of the external data representation (XDR) [RFC4506] generated from [RFC7863].¶

2. Caching of Dirents

With a remote filesystem, the client typically caches directory entries (dirents) locally to improve performance. This cooperation succeeds because both the server and client operate under POSIX semantics ([POSIX.1]) and agree to interpretation of mode bits with respect to the uid and gid in NFSv3 [RFC1813]. For NFSv4.2, these would respectively be the mode, owner, and owner_group attributes defined in Section 5 of [RFC8881]. Note that this cooperation does not apply to Access Control List (ACLs) entries as NFSv4.2 does not implement a strict POSIX style ACL.¶

NFSv4.2 does implement NFSv4.1 ACLs, which are enforced on the server and not the client. As such, ACL enforcement requires the client to bypass the dirent cache to have checks done when a new user attempts to access the dirent.¶

Another consideration is that not all server implementations natively support the SMB [SMB2]. Instead, they layer Samba [Samba] on top of the NFSv4.2 service. The attributes of hidden, system, and offline have already been introduced in the NFSv4.2 protocol to support Samba. The Samba implementation can utilize these attributes to provide SMB semantics. While private protocols can supply these features, it is better to drive them into open standards.¶

Another concept that can be adapted from SMB is that of Access Based Enumeration (ABE). If a share or a folder has ABE enabled, then the user can only see the files and sub-folders for which they have permissions.¶

Under the POSIX model, this can be done on the client and not the server. However, that only works with uid, gid, and mode bits. If we consider identity mappings, ACLS, and server local policies, then the determination of ABE MUST be done on the server.¶

Since cached dirents are shared by all users on a client, and the client cannot determine access permissions for individual dirents, all users are presented with the same set of attributes. To address this, this document introduces the new uncacheable attribute. This attribute instructs the client not to cache the dirent for a file or directory object. Consequently, each time a client queries for these attributes, the server's response can be tailored to the specific user making the request.¶

3. Caching of File Data

In addition to caching metadata, clients can also cache file data. The uncacheable attribute also instructs the client to bypass its page cache for the file. This behavior is similar to using the O_DIRECT flag with the open call ([open_2_]). This can be beneficial for files that are not shared or for files that do not exhibit access patterns suitable for caching.¶

However, the real need for bypassing write caching is evident in HPC workloads. In general, these involve massive data transfers and require extremely low latency. Write caching can introduce unpredictable latency, as data is buffered and flushed later.¶

4. XDR for Offline Attribute

///
/// typedef bool            fattr4_uncacheable;
///
/// const FATTR4_UNCACHEABLE            = 87;
///

5. Extraction of XDR

This document contains the external data representation (XDR) [RFC4506] description of the uncacheable attribute. The XDR description is presented in a manner that facilitates easy extraction into a ready-to-compile format. To extract the machine-readable XDR description, use the following shell script:¶

#!/bin/sh
grep '^ *///' $* | sed 's?^ */// ??' | sed 's?^ *///$??'

For example, if the script is named 'extract.sh' and this document is named 'spec.txt', execute the following command:¶

sh extract.sh < spec.txt > uncacheable_prot.x

This script removes leading blank spaces and the sentinel sequence '///' from each line. XDR descriptions with the sentinel sequence are embedded throughout the document.¶

Note that the XDR code contained in this document depends on types from the NFSv4.2 nfs4_prot.x file (generated from [RFC7863]). This includes both nfs types that end with a 4, such as offset4, length4, etc., as well as more generic types such as uint32_t and uint64_t.¶

While the XDR can be appended to that from [RFC7863], the code snippets should be placed in their appropriate sections within the existing XDR.¶

6. Security Considerations

For a given user A, a client MUST NOT make access decisions for uncacheable dirents retrieved for another user B. These decisions MUST be made by the server. If the client is Labeled NFS aware ([RFC7204]), then the client MUST locally enforce the MAC security policies.¶

The uncacheable attribute allows dirents to be annotated such that attributes are presented to the user based on the server's access control decisions.¶

7. IANA Considerations

This document has no IANA actions.¶

8. References

Acknowledgments

Trond Myklebust and Thomas Haynes all worked on the prototype at Hammerspace.¶

Author's Address