Elliswilson: Created page with "TOC == Overview == Lustre supports '''client-side encryption''' of file data, built on the Linux '''fscrypt''' framework. Encryption and decryption happen only on Lustre clients. Keys are never sent to servers, servers never encrypt or decrypt, and bulk RPCs carry ciphertext that servers write to disk as-is. File contents are therefore protected both '''in transit''' between clients and servers and '''at rest''' on the OSTs. The feature is designed to give users a..."

2026-10-12T15:07:24Z

Created page with "__TOC__ == Overview == Lustre supports '''client-side encryption''' of file data, built on the Linux '''fscrypt''' framework. Encryption and decryption happen only on Lustre clients. Keys are never sent to servers, servers never encrypt or decrypt, and bulk RPCs carry ciphertext that servers write to disk as-is. File contents are therefore protected both '''in transit''' between clients and servers and '''at rest''' on the OSTs. The feature is designed to give users a..."

New page

__TOC__

== Overview ==

Lustre supports '''client-side encryption''' of file data, built on the Linux '''fscrypt''' framework. Encryption and decryption happen only on Lustre clients. Keys are never sent to servers, servers never encrypt or decrypt, and bulk RPCs carry ciphertext that servers write to disk as-is. File contents are therefore protected both '''in transit''' between clients and servers and '''at rest''' on the OSTs.

The feature is designed to give users a private directory for sensitive files. Users manage it per directory with the standard fscrypt tools; no server-side configuration or key material is needed.

{| class="wikitable"
! Release !! Change
|-
| 2.14 || Client-side encryption introduced ([https://jira.whamcloud.com/browse/LU-12275 LU-12275]). File '''contents''' only; file and directory names stay in clear text.
|-
| 2.15 || Filename encryption added ([https://jira.whamcloud.com/browse/LU-13717 LU-13717]). See [[#Filename Encryption|Filename Encryption]] for when names are actually encrypted.
|}

The authoritative references are the ''Managing Security'' chapter of the [https://doc.lustre.org/lustre_manual.xhtml Lustre Operations Manual] and the <code>Documentation/client_side_encryption/</code> directory in the Lustre source tree. For other security features, see [[Setting up Lustre Security]].

== What Is and Is Not Protected ==

{| class="wikitable"
! Protected !! Not protected
|-
|
* File contents, on the wire and on disk
* File and directory names, and symlink targets (2.15+, see [[#Filename Encryption|Filename Encryption]])
|
* File sizes, permissions, ownership, and timestamps
* Extended attributes
* The existence and location of holes in sparse files
* Plaintext on a client while the key is loaded. Other users on that client are kept out only by normal permissions, ACLs, and LSMs.
|}

Because servers never hold keys, an attacker who compromises a Lustre server, even to the point of reading arbitrary memory, cannot recover file contents. An attacker who compromises a '''client''' while a key is loaded can.

== Requirements ==

{| class="wikitable"
! Component !! Requirement
|-
| Client Lustre version || 2.14 or later
|-
| Client kernel || Linux 5.4 or later uses the in-kernel fscrypt library. Older distribution kernels with basic encryption support use an fscrypt library embedded in Lustre: RHEL/CentOS 8.1 and later, Ubuntu 18.04 and later, SLES 15 SP2 and later.
|-
| Client build || Encryption support is built by default. Clients built with <code>--enable-crypto=no</code> are ''encryption-unaware'' (see [[#Limitations|Limitations]]).
|-
| Userspace tool || <code>fscrypt</code> ([https://github.com/google/fscrypt github.com/google/fscrypt]) on every client that will manage or unlock encrypted directories
|-
| Servers || No configuration and no keys. Servers store and serve ciphertext.
|}

Encryption is '''enabled by default''' on Lustre clients. There is no server-side feature flag to turn on.

== Setting Up fscrypt ==

Two one-time steps need root.

'''On every client that will use encryption''', create the global fscrypt configuration:

# fscrypt setup
Customizing passphrase hashing difficulty for this system...
Created global config file at "/etc/fscrypt.conf".
Metadata directories created at "/.fscrypt".

Then edit <code>/etc/fscrypt.conf</code> and set <code>policy_version</code> to <code>2</code>. With version 2 policies, locking a directory also wipes its cached plaintext from memory.

'''On one client only''', create the fscrypt metadata directory on the Lustre filesystem:

# fscrypt setup /mnt/lustre
Metadata directories created at "/mnt/lustre/.fscrypt"

== Encrypting a Directory ==

Any user can then encrypt an '''empty''' directory that they own:

$ mkdir /mnt/lustre/vault
$ fscrypt encrypt /mnt/lustre/vault
The following protector sources are available:
1 - Your login passphrase (pam_passphrase)
2 - A custom passphrase (custom_passphrase)
3 - A raw 256-bit key (raw_key)
Enter the source number for the new protector [2 - custom_passphrase]: 2
Enter a name for the new protector: shield
Enter custom passphrase for protector "shield":
Confirm passphrase:
"/mnt/lustre/vault" is now encrypted, unlocked, and ready for use.

Every file, directory, and symlink created under <code>/mnt/lustre/vault</code> afterwards inherits the encryption policy. The policy cannot be changed for a subdirectory, and it cannot be removed once set. To decrypt data, copy it out to an unencrypted directory while the directory is unlocked.

=== Locking and Unlocking ===

$ fscrypt lock /mnt/lustre/vault
"/mnt/lustre/vault" is now locked.

$ fscrypt unlock /mnt/lustre/vault
Enter custom passphrase for protector "shield":
"/mnt/lustre/vault" is now unlocked and ready for use.

Locking removes the master key from kernel memory. With version 2 policies it also evicts cached inodes, so files that are not open revert to their locked, encrypted form. Non-root users can remove their own keys.

Keys are loaded '''per client'''. Unlocking a directory on one client does not unlock it anywhere else; each client that needs the data must unlock it.

=== Protectors and Sharing ===

fscrypt never exposes the master key directly. Each policy's master key is wrapped by one or more ''protectors'' (a custom passphrase, a login passphrase, or a raw key). This lets a passphrase change without re-encrypting any data, and lets several users share an encrypted directory without sharing a secret:

# Change a protector's passphrase
$ fscrypt metadata change-passphrase --protector=/mnt/lustre:deacab807bf0e788

# Create a second protector and attach it to an existing policy
$ fscrypt metadata create protector /mnt/lustre
$ fscrypt metadata add-protector-to-policy \
--protector=/mnt/lustre:f3cc1b5cf9b8f41c \
--policy=/mnt/lustre:52b2b5aff0e59d8e0d58f962e715862e

<code>fscrypt status /mnt/lustre</code> lists all protectors and policies. <code>fscrypt status /mnt/lustre/vault</code> shows a directory's policy, its options, and whether it is unlocked.

'''Keep protector secrets safe.''' If every protector for a policy is lost, the data under it cannot be recovered.

== Encryption Modes ==

fscrypt uses one mode for file contents and one for filenames. Lustre supports:

* AES-256-XTS for contents with AES-256-CTS-CBC for filenames (recommended)
* AES-128-CBC for contents with AES-128-CTS-CBC for filenames

== Filename Encryption ==

Whether names are encrypted depends on how the client was built and which Lustre version created the directory:

{| class="wikitable"
! Situation !! Names encrypted?
|-
| Directory created by a 2.14 client || '''No''', contents only. Files and directories created under it later, by any version, also keep clear-text names.
|-
| 2.15+ client using the in-kernel fscrypt library || '''Yes''', always. These clients do not have the <code>enable_filename_encryption</code> parameter.
|-
| 2.15+ client using the embedded fscrypt library || Only if <code>llite.*.enable_filename_encryption=1</code> when the directory is created. '''Defaults to 0''' (contents only).
|}

To turn on filename encryption for every client that uses the embedded library, run on the MGS:

mgs# lctl set_param -P llite.*.enable_filename_encryption=1

This applies to '''new''' encrypted directories only. To get encrypted names for an existing directory: create a new encrypted directory (it can reuse an existing protector), unlock the old one, copy everything across, then remove the old directory.

== Access Without the Key ==

With the key loaded, encrypted files behave like ordinary files. On a client where the key is '''not''' loaded:

* <code>stat()</code> works, and directories can be listed. Encrypted names appear in an encoded form that is at most <code>NAME_MAX</code> bytes and never contains <code>/</code> or NUL.
* Files and empty directories can be deleted, so <code>rm -r</code> works.
* Opening or truncating a regular file fails with <code>ENOKEY</code> ("Required key not available"). So does creating, linking, or renaming anything inside an encrypted directory.

== Striping and Layouts ==

The client encrypts file pages before sending them in bulk RPCs, so encryption is independent of the file layout. Striping, PFL, and OST pools work normally on encrypted files:

lfs setstripe -c 4 -S 4M /mnt/lustre/vault

== Administrative Controls ==

{| class="wikitable"
! Control !! Effect
|-
| <code>noencrypt</code> client mount option || Prevents that mount from using encryption.
|-
| Nodemap property <code>forbid_encryption=1</code> || Enforced from the server side: clients in the nodemap cannot use encryption. See [[Lustre Multi Tenancy]].
|-
| Nodemap <code>rbac</code> role <code>fscrypt_admin</code> || Needed to create or modify fscrypt protectors and policies. Without it, users can still lock and unlock existing encrypted directories.
|}

mgs# lctl nodemap_modify --name tenantA --property forbid_encryption --value 1

== Limitations ==

* '''No backup or restore without the key.''' No API yet exists to back up or restore ciphertext. Back up from a client where the directory is unlocked, and protect those backups, because they contain plaintext.
* '''Encryption-unaware clients''' are clients older than 2.14, clients built with <code>--enable-crypto=no</code>, and clients on kernels without the needed support. To prevent corruption, they get <code>ENOKEY</code> when they try to open, create, link, rename, or migrate encrypted files. They can still list, stat, and remove them, using the encoded names.
* '''Moving data in.''' Unencrypted files, and files under a different policy, cannot be renamed or hard-linked into an encrypted directory. <code>mv</code> falls back to copy-and-delete, and the original plaintext may stay recoverable from free space on the OSTs. Encrypt directories before you put data in them.
* '''fallocate()''' modes <code>FALLOC_FL_COLLAPSE_RANGE</code>, <code>FALLOC_FL_INSERT_RANGE</code>, and <code>FALLOC_FL_ZERO_RANGE</code> fail with <code>EOPNOTSUPP</code>.
* '''DAX''' is not supported. Direct I/O and <code>mmap</code> are supported.
* '''Symlinks.''' <code>st_size</code> reports the ciphertext length, and the maximum target length is 2 bytes shorter than for an unencrypted symlink.
* '''Special files''' (named pipes, device nodes, and UNIX sockets) are not encrypted.
* '''CPU cost.''' Encryption runs on client CPUs. Hardware AES acceleration (for example, AES-NI) greatly reduces the overhead; benchmark your own workload.

== See Also ==

* [[Setting up Lustre Security]]
* [[Lustre Multi Tenancy]] — nodemaps, including <code>forbid_encryption</code>
* [[Lustre Kerberos SSK Authentication]] — encrypting all RPC traffic on the wire

[[Category:Administration]]
[[Category:Security]]

Lustre Client Encryption - Revision history