Lustre Client Encryption

From Lustre Wiki
Jump to navigation Jump to search

Overview

Lustre supports client-side encryption of file data, built on the Linux fscrypt framework. Encryption and decryption happen only on Lustre clients. Keys are never sent to servers, servers never encrypt or decrypt, and bulk RPCs carry ciphertext that servers write to disk as-is. File contents are therefore protected both in transit between clients and servers and at rest on the OSTs.

The feature is designed to give users a private directory for sensitive files. Users manage it per directory with the standard fscrypt tools; no server-side configuration or key material is needed.

Release Change
2.14 Client-side encryption introduced (LU-12275). File contents only; file and directory names stay in clear text.
2.15 Filename encryption added (LU-13717). See Filename Encryption for when names are actually encrypted.

The authoritative references are the Managing Security chapter of the Lustre Operations Manual and the Documentation/client_side_encryption/ directory in the Lustre source tree. For other security features, see Setting up Lustre Security.

What Is and Is Not Protected

Protected Not protected
  • File contents, on the wire and on disk
  • File and directory names, and symlink targets (2.15+, see Filename Encryption)
  • File sizes, permissions, ownership, and timestamps
  • Extended attributes
  • The existence and location of holes in sparse files
  • Plaintext on a client while the key is loaded. Other users on that client are kept out only by normal permissions, ACLs, and LSMs.

Because servers never hold keys, an attacker who compromises a Lustre server, even to the point of reading arbitrary memory, cannot recover file contents. An attacker who compromises a client while a key is loaded can.

Requirements

Component Requirement
Client Lustre version 2.14 or later
Client kernel Linux 5.4 or later uses the in-kernel fscrypt library. Older distribution kernels with basic encryption support use an fscrypt library embedded in Lustre: RHEL/CentOS 8.1 and later, Ubuntu 18.04 and later, SLES 15 SP2 and later.
Client build Encryption support is built by default. Clients built with --enable-crypto=no are encryption-unaware (see Limitations).
Userspace tool fscrypt (github.com/google/fscrypt) on every client that will manage or unlock encrypted directories
Servers No configuration and no keys. Servers store and serve ciphertext.

Encryption is enabled by default on Lustre clients. There is no server-side feature flag to turn on.

Setting Up fscrypt

Two one-time steps need root.

On every client that will use encryption, create the global fscrypt configuration: 

# fscrypt setup
Customizing passphrase hashing difficulty for this system...
Created global config file at "/etc/fscrypt.conf".
Metadata directories created at "/.fscrypt".

Then edit /etc/fscrypt.conf and set policy_version to 2. With version 2 policies, locking a directory also wipes its cached plaintext from memory.

On one client only, create the fscrypt metadata directory on the Lustre filesystem: 

# fscrypt setup /mnt/lustre
Metadata directories created at "/mnt/lustre/.fscrypt"

Encrypting a Directory

Any user can then encrypt an empty directory that they own: 

$ mkdir /mnt/lustre/vault
$ fscrypt encrypt /mnt/lustre/vault
The following protector sources are available:
1 - Your login passphrase (pam_passphrase)
2 - A custom passphrase (custom_passphrase)
3 - A raw 256-bit key (raw_key)
Enter the source number for the new protector [2 - custom_passphrase]: 2
Enter a name for the new protector: shield
Enter custom passphrase for protector "shield":
Confirm passphrase:
"/mnt/lustre/vault" is now encrypted, unlocked, and ready for use.

Every file, directory, and symlink created under /mnt/lustre/vault afterwards inherits the encryption policy. The policy cannot be changed for a subdirectory, and it cannot be removed once set. To decrypt data, copy it out to an unencrypted directory while the directory is unlocked.

Locking and Unlocking

$ fscrypt lock /mnt/lustre/vault
"/mnt/lustre/vault" is now locked.

$ fscrypt unlock /mnt/lustre/vault
Enter custom passphrase for protector "shield":
"/mnt/lustre/vault" is now unlocked and ready for use.

Locking removes the master key from kernel memory. With version 2 policies it also evicts cached inodes, so files that are not open revert to their locked, encrypted form. Non-root users can remove their own keys.

Keys are loaded per client. Unlocking a directory on one client does not unlock it anywhere else; each client that needs the data must unlock it.

Protectors and Sharing

fscrypt never exposes the master key directly. Each policy's master key is wrapped by one or more protectors (a custom passphrase, a login passphrase, or a raw key). This lets a passphrase change without re-encrypting any data, and lets several users share an encrypted directory without sharing a secret: 

# Change a protector's passphrase
$ fscrypt metadata change-passphrase --protector=/mnt/lustre:deacab807bf0e788

# Create a second protector and attach it to an existing policy
$ fscrypt metadata create protector /mnt/lustre
$ fscrypt metadata add-protector-to-policy \
      --protector=/mnt/lustre:f3cc1b5cf9b8f41c \
      --policy=/mnt/lustre:52b2b5aff0e59d8e0d58f962e715862e

fscrypt status /mnt/lustre lists all protectors and policies. fscrypt status /mnt/lustre/vault shows a directory's policy, its options, and whether it is unlocked.

Keep protector secrets safe. If every protector for a policy is lost, the data under it cannot be recovered.

Encryption Modes

fscrypt uses one mode for file contents and one for filenames. Lustre supports:

  • AES-256-XTS for contents with AES-256-CTS-CBC for filenames (recommended)
  • AES-128-CBC for contents with AES-128-CTS-CBC for filenames

Filename Encryption

Whether names are encrypted depends on how the client was built and which Lustre version created the directory:

Situation Names encrypted?
Directory created by a 2.14 client No, contents only. Files and directories created under it later, by any version, also keep clear-text names.
2.15+ client using the in-kernel fscrypt library Yes, always. These clients do not have the enable_filename_encryption parameter.
2.15+ client using the embedded fscrypt library Only if llite.*.enable_filename_encryption=1 when the directory is created. Defaults to 0 (contents only).

To turn on filename encryption for every client that uses the embedded library, run on the MGS: 

mgs# lctl set_param -P llite.*.enable_filename_encryption=1

This applies to new encrypted directories only. To get encrypted names for an existing directory: create a new encrypted directory (it can reuse an existing protector), unlock the old one, copy everything across, then remove the old directory.

Access Without the Key

With the key loaded, encrypted files behave like ordinary files. On a client where the key is not loaded:

  • stat() works, and directories can be listed. Encrypted names appear in an encoded form that is at most NAME_MAX bytes and never contains / or NUL.
  • Files and empty directories can be deleted, so rm -r works.
  • Opening or truncating a regular file fails with ENOKEY ("Required key not available"). So does creating, linking, or renaming anything inside an encrypted directory.

Striping and Layouts

The client encrypts file pages before sending them in bulk RPCs, so encryption is independent of the file layout. Striping, PFL, and OST pools work normally on encrypted files: 

lfs setstripe -c 4 -S 4M /mnt/lustre/vault

Administrative Controls

Control Effect
noencrypt client mount option Prevents that mount from using encryption.
Nodemap property forbid_encryption=1 Enforced from the server side: clients in the nodemap cannot use encryption. See Lustre Multi Tenancy.
Nodemap rbac role fscrypt_admin Needed to create or modify fscrypt protectors and policies. Without it, users can still lock and unlock existing encrypted directories. 
mgs# lctl nodemap_modify --name tenantA --property forbid_encryption --value 1

Limitations

  • No backup or restore without the key. No API yet exists to back up or restore ciphertext. Back up from a client where the directory is unlocked, and protect those backups, because they contain plaintext.
  • Encryption-unaware clients are clients older than 2.14, clients built with --enable-crypto=no, and clients on kernels without the needed support. To prevent corruption, they get ENOKEY when they try to open, create, link, rename, or migrate encrypted files. They can still list, stat, and remove them, using the encoded names.
  • Moving data in. Unencrypted files, and files under a different policy, cannot be renamed or hard-linked into an encrypted directory. mv falls back to copy-and-delete, and the original plaintext may stay recoverable from free space on the OSTs. Encrypt directories before you put data in them.
  • fallocate() modes FALLOC_FL_COLLAPSE_RANGE, FALLOC_FL_INSERT_RANGE, and FALLOC_FL_ZERO_RANGE fail with EOPNOTSUPP.
  • DAX is not supported. Direct I/O and mmap are supported.
  • Symlinks. st_size reports the ciphertext length, and the maximum target length is 2 bytes shorter than for an unencrypted symlink.
  • Special files (named pipes, device nodes, and UNIX sockets) are not encrypted.
  • CPU cost. Encryption runs on client CPUs. Hardware AES acceleration (for example, AES-NI) greatly reduces the overhead; benchmark your own workload.

See Also

Retrieved from "http://wiki.lustre.org/index.php?title=Lustre_Client_Encryption&oldid=5759"