Lustre Kerberos SSK Authentication

From Lustre Wiki
Jump to navigation Jump to search

Overview

Lustre can authenticate nodes and protect PtlRPC traffic through GSSAPI, using one of two mechanisms:

Kerberos (krb5) Shared-Secret Key (SSK)
Infrastructure MIT Kerberos 5 KDC None; the administrator distributes key files
Authenticates Nodes, and individual users via kinit Nodes only (a single root context per connection)
Binds a client to its identity by A per-host principal and keytab A key file bound to a nodemap
Can encrypt traffic Yes (krb5p) Yes (skpi)

SSK was added in Lustre 2.9 (LU-3289). It suits sites that want node authentication and wire encryption without running a KDC.

Both mechanisms need Lustre built with GSS support, the lsvcgssd daemon on servers, and the lgss_keyring request-key upcall on every node. The authoritative references are the Configuring Shared-Secret Key (SSK) Security and Managing Security chapters of the Lustre Operations Manual.

This page covers transport security. It does not encrypt data at rest; for that, see Lustre Client Encryption.

Security Flavors

A security flavor sets the authentication and protection applied to a connection, covering both RPC messages and bulk data.

Flavor Mechanism RPC message Bulk data Notes
null — None None Default for all connections
plain — Plaintext with a hash on each RPC No authentication
krb5n Kerberos None Checksum Lightest Kerberos flavor
krb5a Kerberos Header integrity only Checksum
krb5i Kerberos Integrity Integrity Heavy overhead
krb5p Kerberos Privacy Privacy Heaviest overhead
skn SSK None None Authentication only: a valid key is required to mount
ska SSK Integrity None Integrity for non-bulk RPCs only
ski SSK Integrity Integrity
skpi SSK Privacy and integrity Privacy and integrity The only SSK flavor that encrypts
gssnull GSS None None Testing only

Only krb5p and skpi encrypt traffic. The other flavors authenticate nodes and/or detect tampering, but send data in the clear. Every SSK flavor requires a valid key to mount.

Setting Flavors with sptlrpc Rules

Flavors are set by persistent rules defined on the MGS with lctl conf_param: 

<target>.srpc.flavor.<network>[.<direction>]=<flavor>
  • <target> is a filesystem name (testfs) or a specific target (testfs-MDT0000, testfs-OST0001).
  • <network> is the LNet network of the RPC initiator (tcp0, o2ib0), or default for any network without a more specific rule.
  • <direction> is optional: cli2mdt, cli2ost, mdt2mdt, or mdt2ost.

The most specific matching rule wins. Examples: 

# Authenticate and encrypt everything by default
mgs# lctl conf_param testfs.srpc.flavor.default=skpi

# ...but on physically secure, high-performance networks, authenticate and check integrity only
mgs# lctl conf_param testfs.srpc.flavor.tcp1=ska
mgs# lctl conf_param testfs.srpc.flavor.o2ib0=ska

# List the rules
mgs# lctl get_param mgs.*.live.testfs | grep srpc.flavor

# Delete a rule
mgs# lctl conf_param -d testfs.srpc.flavor.o2ib0

Important behavior:

  • Rule changes reach existing as well as new connections, typically within seconds to a few minutes depending on system load.
  • Make sure every affected node is ready before you change a rule. A node that cannot negotiate the new flavor (missing key, daemon not running, Kerberos misconfigured) can be evicted.
  • lctl set_param -P does not work for sptlrpc rules. Use lctl conf_param.
  • Rules do not cover the MGS connection itself, because a node only learns the rules after it connects to the MGS. See Securing the MGS Connection.

SSK Setup

This walkthrough follows the example in the Operations Manual: filesystem testfs, a combined MGS/MDS at 172.16.0.1@tcp, one OSS, and two clients. It protects client-to-server traffic with skpi.

Step 1: Confirm the GSS Components Are Installed

When Lustre is built with GSS keyring support, its packages install:

  • /etc/request-key.d/lgssc.conf on every node, containing this single line:
create lgssc * * /usr/sbin/lgss_keyring %o %k %t %d %c %u %g %T %P %S
  • lsvcgss.service and /etc/sysconfig/lsvcgss on servers.

Check that the request-key file exists on every node, clients included. Without it, nodes cannot establish security contexts.

Step 2: Create a Key Directory on Every Node

# mkdir /secure_directory
# chmod 600 /secure_directory

Step 3: Generate the Server Key

mds# lgss_sk -t server -f testfs -w /secure_directory/testfs.server.key

lgss_sk reads /dev/random by default. On a system without a hardware random number generator, it can block until enough entropy is available.

Step 4: Distribute the Key

Copy the key to every other server, and to every client under a client filename, over a secure channel: 

mds# scp /secure_directory/testfs.server.key oss:/secure_directory/
mds# scp /secure_directory/testfs.server.key client1:/secure_directory/testfs.client.key

Never put key files on a shared filesystem.

Step 5: Convert Each Client Copy to a Client Key

On each client: 

client1# lgss_sk -t client -m /secure_directory/testfs.client.key

This sets the key type to client and generates the Diffie-Hellman prime that only client keys carry. Generating the prime can take a while.

Step 6: Start lsvcgssd on the Servers

Set LSVCGSSDARGS in /etc/sysconfig/lsvcgss. -s enables SSK; -m, -o, and -g select the MDS, OSS, and MGS services the daemon handles. 

# /etc/sysconfig/lsvcgss on the MDS
LSVCGSSDARGS="-s -m"

# /etc/sysconfig/lsvcgss on the OSS
LSVCGSSDARGS="-s -o"

mds# systemctl start lsvcgss.service
oss# systemctl start lsvcgss.service

Step 7: Mount the Targets with skpath

mds# mount -t lustre -o skpath=/secure_directory /dev/mdt_device /mnt/mdt
oss# mount -t lustre -o skpath=/secure_directory /dev/ost_device /mnt/ost0

skpath loads every key file in the directory into the kernel keyring. A world-writable key file fails to load.

Step 8: Set the Flavor

mgs# lctl conf_param testfs.srpc.flavor.tcp.cli2mdt=skpi
mgs# lctl conf_param testfs.srpc.flavor.tcp.cli2ost=skpi

Server-to-server connections keep the default null flavor. See Securing Server-to-Server Traffic.

Step 9: Mount the Clients

client1# mount -t lustre -o skpath=/secure_directory 172.16.0.1@tcp:/testfs /mnt/testfs

A client without a valid key is refused: 

mount.lustre: mount 172.16.0.1@tcp:/testfs at /mnt/testfs failed: Connection refused

Then verify the contexts.

lgss_sk Reference

Option Description
-w, --write FILE Generate a new key file. Will not overwrite an existing file.
-m, --modify FILE Change only the attributes given on the command line
-r, --read FILE Show a key file's attributes (- reads standard input)
-l, --load FILE Load one key file into the session keyring. Must be run as root.
-t, --type TYPE mgs, server, or client; combine with a comma (e.g. mgs,server). Required when writing.
-f, --fsname NAME Filesystem the key is for. Writing requires -f or -g.
-g, --mgsnids NIDS Comma-separated MGS NIDs. Needed only when mgssec is used.
-n, --nodemap NAME Nodemap the key is bound to (default: default)
-e, --expire SECONDS How long session contexts created from this key last before they are renegotiated (default 604800, 7 days). This is not an expiry date for the key file.
-i, --hmac ALG Integrity hash: SHA256 (default) or SHA512
-c, --crypt ALG Cipher for privacy: AES-256-CTR (the default and only choice)
-p, --prime-bits BITS Diffie-Hellman prime length (default 2048). On server and MGS keys, this is also the minimum prime length accepted from clients.
-k, --key-bits BITS Shared key length (default 256)
-d, --data FILE Entropy source for the shared key (default /dev/random). Use /dev/urandom for testing only.
-v, --verbose More detailed error messages

Binding Keys to Nodemaps

Every SSK key names a nodemap (-n, default default). When a client connects, it sends a hash of the nodemap name from its key. The server looks up which nodemap the client's NID belongs to, and lsvcgssd checks that the two match. A key used from a NID outside that nodemap's ranges is therefore rejected.

This is what stops one tenant from impersonating another with a copied key. Issue a separate key for each nodemap: 

mgs# lgss_sk -t server -f testfs -n biology -w /secure_directory/testfs.server.biology.key

The NID-to-nodemap lookup works whether or not nodemaps have been activated with lctl nodemap_activate.

Alternatively, the nodemap property gssonly_identification=1 identifies clients by the GSS token they authenticate with, rather than by NID; such a nodemap cannot have NID ranges. See Lustre Multi Tenancy.

Securing the MGS Connection

Each node chooses the flavor for its MGS connection with the mgssec mount option. A node has only one MGS connection, so every target and client on that node uses the flavor set when the first connection was made, and it cannot change until remount.

  1. On the MGS, add -g to LSVCGSSDARGS and restart lsvcgss.service.
  2. Add the mgs type and the MGS NIDs to the server key, and load it:
    mgs# lgss_sk -t mgs,server -g 172.16.0.1@tcp,172.16.0.2@tcp -m /secure_directory/testfs.server.key
    mgs# lgss_sk -l /secure_directory/testfs.server.key
  3. Add the MGS NIDs to each client key:
    client1# lgss_sk -g 172.16.0.1@tcp,172.16.0.2@tcp -m /secure_directory/testfs.client.key
  4. Remount each client with mgssec:
    client1# mount -t lustre -o mgssec=skpi,skpath=/secure_directory 172.16.0.1@tcp:/testfs /mnt/testfs

By default the MGS accepts any flavor. To require a particular one, add a rule on the special target _mgs: 

mgs# lctl conf_param _mgs.srpc.flavor.tcp=skpi

Securing Server-to-Server Traffic

When servers talk to each other (for example, MDT to OST), they act as clients, so they need a client-type key too. To apply ski to MDT-to-OST traffic:

  1. Put the servers in their own nodemap:
    mgs# lctl nodemap_add LustreServers
    mgs# lctl nodemap_add_range --name LustreServers --range 172.16.0.[1-3]@tcp
  2. Create an mgs,server key for that nodemap and copy it to every server:
    mds# lgss_sk -t mgs,server -f testfs -g 172.16.0.1@tcp,172.16.0.2@tcp -n LustreServers -w /secure_directory/testfs.LustreServers.key
  3. On each server, copy that key to testfs.LustreServers.client.key and convert the copy with lgss_sk -t client -m.
  4. On each server, load both keys with lgss_sk -l.
  5. Set the rule:
    mgs# lctl conf_param testfs.srpc.flavor.tcp.mdt2ost=ski

If nodemaps are active, the servers' nodemap must also have admin=1 and trusted=1. See Lustre Multi Tenancy.

Revoking and Rotating Keys

SSK keys cannot be revoked as such, and Lustre has no way to flush established contexts. To lock out a compromised key, do one of the following:

  • Unload the key from the keyring on the servers (with keyctl). New connections that use it then fail.
  • Move the affected NIDs to a nodemap with a different name. Keys that name the old nodemap can no longer establish contexts.

Contexts that already exist last until they expire (-e, 7 days by default) or the targets are unmounted. If you need revocation to take effect quickly, create keys with a shorter expiry; the manual suggests 3600 seconds as reasonable for some sites.

To rotate keys: generate new key files, distribute and load them, and remove the old files from every skpath directory so they are not reloaded at the next mount.

Kerberos Setup

Lustre supports MIT Kerberos 5, version 1.3 or later, with the aes128-cts and aes256-cts enctypes. Clocks must be synchronized across all nodes and the KDC.

Packages and Build

Install on every node:

  • krb5-workstation, krb5-libs
  • keyutils, keyutils-libs

Lustre must be built with GSS support. On the build host, install krb5-devel and keyutils-libs-devel, then configure with: 

./configure --enable-gss

Principals and Keytabs

Create a principal for each node, using that node's fully qualified hostname, and install its keytab on that node:

Node role Principal
Client lustre_root/client_host.domain@REALM
MGS lustre_mgs/mgs_host.domain@REALM
MDS lustre_mds/mds_host.domain@REALM
OSS lustre_oss/oss_host.domain@REALM 
kadmin> addprinc -randkey lustre_mds/mds_host.domain@REALM
kadmin> ktadd lustre_mds/mds_host.domain@REALM

Notes:

  • The hostname in each principal must be the one that resolves to the IP address in that node's Lustre NID. Otherwise the servers do not recognize its GSS requests.
  • To avoid a keytab per client, you can create a single lustre_root@REALM principal and install its keytab on every client. The cost is that one compromised client compromises them all.
  • On networks without hostname resolution (anything other than TCP or InfiniBand), each node needs an executable /etc/lustre/nid2hostname script that translates a NID into a hostname. The Configuring Kerberos (KRB) Security section of the manual includes an example.

Start lsvcgssd

On each server, run lsvcgssd with -k, plus the services that node runs (-g MGS, -m MDS, -o OSS), before starting Lustre: 

# /etc/sysconfig/lsvcgss on a combined MGS/MDS
LSVCGSSDARGS="-k -g -m"

# systemctl start lsvcgss.service

Clients use the same request-key upcall as SSK; check that /etc/request-key.d/lgssc.conf exists on every client.

Set the Flavor

Use sptlrpc rules with a krb5 flavor, for example: 

mgs# lctl conf_param testfs.srpc.flavor.default=krb5i

User Authentication

With Kerberos, users authenticate as well as nodes. Non-root users must run kinit before accessing Lustre, as with any Kerberized application.

  • Each user's principal (username@REALM) must exist in the KDC.
  • Clients and MDS nodes must share the same user database for name and UID/GID translation.

Users can destroy their security contexts before logging out: 

$ lfs flushctx -k -r /mnt/testfs

-k also destroys the on-disk Kerberos credential cache (like kdestroy), and -r reaps revoked keys from the keyring.

Verifying Security Contexts

On a client (or on a server, for its own outgoing connections), check the flavor each import is using: 

client1# lctl get_param *.*.srpc_info
mdc.testfs-MDT0000-mdc-ffff8800da9f0800.srpc_info=
rpc flavor:     skpi
bulk flavor:    skpi
...

srpc_contexts shows each established context and its mechanism (mech: sk or krb5). SSK supports only a single root context per import, so there should be exactly one. 

client1# lctl get_param *.*.srpc_contexts

An mgc import without a secure context means the MGS connection is not protected; see Securing the MGS Connection.

Troubleshooting

  • Mount fails with "Connection refused": the client has no valid key for this filesystem and nodemap, or the key did not load. Check the skpath directory, and use lgss_sk -r to confirm the key's type (client), filesystem name, and nodemap.
  • A key file fails to load: the file is world-writable.
  • Nodes are evicted after a rule change: a node could not negotiate the new flavor. Check that lsvcgssd is running with the right options, that the request-key file exists, and that the keys or keytabs are in place.

Client-side logging. lgss_keyring runs from the request-key upcall, which has no standard output, so it logs to syslog. Raise its level (0 error, 1 warn, 2 info, 3 debug, 4 trace): 

client# lctl set_param sptlrpc.gss.lgss_keyring.debug_level=4

Server-side logging. Run the daemon in the foreground with extra -v flags. It logs to standard output in the foreground and to syslog as a daemon: 

server# lsvcgssd -f -vvv -s

To inspect the keys in a keyring: 

# keyctl show
# keyctl pipe <key_id> | lgss_sk -r -

Performance

Overhead grows with the level of protection. Authentication-only flavors (skn, krb5n) are light, integrity flavors cost more, and the privacy flavors (skpi, krb5p) cost the most, because they encrypt bulk data. CPUs with hardware AES acceleration reduce the cost of skpi. Benchmark your workload before enabling privacy system-wide, and consider rules that apply privacy only on untrusted networks.

Limitations

  • SSK authenticates nodes, not users. For multi-tenant isolation, combine it with nodemaps (UID/GID mapping and root squash). Use Kerberos if you need per-user authentication.
  • A key file is a shared secret. Anyone who obtains it can authenticate as a member of its nodemap, from a NID in that nodemap. Use separate keys per nodemap, keep them off shared filesystems, and protect them like passwords.
  • Each connection uses one flavor, so Kerberos and SSK cannot both apply to the same connection. Rules can still give different networks, directions, or targets different mechanisms.
  • Transport security does not protect data at rest. See Lustre Client Encryption.

See Also

Retrieved from "http://wiki.lustre.org/index.php?title=Lustre_Kerberos_SSK_Authentication&oldid=5760"