C H A P T E R  11

Kerberos

This chapter describes how to use Kerberos with Lustre and includes the following sections:


11.1 What is Kerberos?

Kerberos is a mechanism for authenticating all entities (such as users and services) on an “unsafe” network. Users and services, known as "principals", share a secret password (or key) with the Kerberos server. This key enables principals to verify that messages from the Kerberos server are authentic. By trusting the Kerberos server, users and services can authenticate one another.



caution icon Caution - Kerberos is a future Lustre feature that is not available in current versions. If you want to test Kerberos with a pre-release version of Lustre, check out the Lustre source from the CVS repository and build it. For more information on checking out Lustre source code, see CVS.


 


11.2 Lustre Setup with Kerberos

Setting up Lustre with Kerberos can provide advanced security protections for the Lustre network. Broadly, Kerberos offers three types of benefit:

Kerberos uses the “kernel keyring” client upcall mechanism.

11.2.1 Configuring Kerberos for Lustre

This section describes supported Kerberos distributions and how to set up and configure Kerberos on Lustre.

11.2.1.1 Kerberos Distributions Supported on Lustre

Lustre supports the following Kerberos distributions:

On a number of operating systems, the Kerberos RPMs are installed when the operating system is first installed. To determine if Kerberos RPMs are installed on your OS, run:

# rpm -qa | grep krb

If Kerberos is installed, the command returns a list like this:

krb5-devel-1.4.3-5.1
krb5-libs-1.4.3-5.1
krb5-workstation-1.4.3-5.1
pam_krb5-2.2.6-2.2


Note - The Heimdal implementation of Kerberos is not currently supported on Lustre, although it support will be added in an upcoming release.


11.2.1.2 Preparing to Set Up Lustre with Kerberos

To set up Lustre with Kerberos:

1. Configure NTP to synchronize time across all machines.

2. Configure DNS with zones.

3. Verify that there are fully-qualified domain names (FQDNs), that are resolvable in both forward and reverse directions for all servers. This is required by Kerberos.

4. On every node, install flowing packages:

Some newer Linux distributions include libgssapi by default. If you do not have libgssapi, build and install it from source:

http://www.citi.umich.edu/projects/nfsv4/linux/libgssapi/libssapi-0.10.tar.gz

11.2.1.3 Configuring Lustre for Kerberos

To configure Lustre for Kerberos:

1. Configure the client nodes.

a. For each client node, create a lustre_root principal and generate the keytab.

kadmin> addprinc -randkey lustre_root/client_host.domain@REALM
kadmin> ktadd -e aes128-cts:normal lustre_root/client_host.domain@REALM

b. Install the keytab on the client node.



Note - For each client-OST pair, there is only one security context, shared by all users on the client. This protects data written by one user to be passed to an OST by another user due to asynchronous bulk I/O. The client-OST connection only guarantees message integrity or privacy; it does not authenticate users.


2. Configure the MDS nodes.

a. For each MDS node, create a lustre_mds principal and generate the keytab.

kadmin> addprinc -randkey lustre_mds/mdthost.domain@REALM
kadmin> ktadd -e aes128-cts:normal lustre_mds/mdthost.domain@REALM

b. Install the keytabl on the MDS node.

3. Configure the OSS nodes.

a. For each OSS node, create a lustre_oss principal and generate the keytab.

kadmin> addprinc -randkey lustre_oss/osthost.domain@REALM
kadmin> ktadd -e aes128-cts:normal lustre_oss/osshost.domain@REALM

b. Install the keytab on the OSS node.



Tip - To avoid assigning a unique keytab to each client node, create a general lustre_root principal and keytab, and install the keytab on as many client nodes as needed.

kadmin> addprinc -randkey lustre_root@REALM
kadmin> ktadd -e aes128-cts:normal lustre_root@REALM

Remember that if you use a general keytab, then one compromised client means that all client nodes are insecure.


General Installation Notes

For MIT Kerberos 1.3.x, only des-cbc-md5 works because of a known issue between libgssapi and the Kerberos library.



Note - The encryption type (or enctype) is an identifier specifying the encryption, mode and hash algorithms. Each Kerberos key has an associated enctype that identifies the cryptographic algorithm and mode used when performing cryptographic operations with the key. It is important that the enctypes requested by the client are actually supported on the system hosting the client. This is the case if the defaults that control enctypes are not overridden.


11.2.1.4 Configuring Kerberos

To configure Kerberos to work with Lustre:

1. Modify the files for Kerberos:

$ /etc/krb5.conf
[libdefaults]
default_realm = CLUSTERFS.COM
 
[realms]
CLUSTERFS.COM = {
kdc = mds16.clustrefs.com
admin_server = mds16.clustrefs.com
}
 
[domain_realm]
.clustrefs.com = CLUSTERFS.COM
clustrefs.com = CLSUTREFS.COM
 
[logging]
default = FILE:/var/log/kdc.log

2. Prepare the Kerberos database.

3. Create service principals so Lustre supports Kerberos authentication.



Note - You can create service principals when configuring your other services to support Kerberos authentication.


4. Configure the client nodes. For each client node:

a. Create a lustre_root principal and generate the keytab:

kadmin> addprinc -randkey lustre_root/client_host.domain@REALM
	kadmin> ktadd -e aes128-cts:normal
lustre_root/client_host.domain@REALM 

This process populates /etc/krb5.keytab, which is not human-readable. Use the ktutil program to read and modify it.

b. Install the keytab.



Note - There is only one security context for each client-OST pair, shared by all users on the client. This protects data written by one user to be passed to an OST by another user due to asynchronous bulk I/O. The client-OST connection only guarantees message integrity or privacy; it does not authenticate users.


5. Configure the MDS nodes. For each MDT node, create a lustre_mds principal, and generate and install the keytab.

kadmin> addprinc -randkey lustre_mds/mdthost.domain@REALM
	kadmin> ktadd -e aes128-cts:normal
lustre_mds/mdthost.domain@REALM

6. Configure the OSS nodes. For each OST node, create a lustre_oss principal, and generate and install the keytab.

kadmin> addprinc -randkey lustre_oss/oss_host.domain@REALM
	kadmin> ktadd -e aes128-cts:normal lustre_oss/oss_host.domain@REALM

To save the trouble of assigning a unique keytab for each client node, create a general lustre_root principal and its keytab, and then install the keytab on as many client nodes as needed.

kadmin> addprinc -randkey lustre_root@REALM
kadmin> ktadd -e aes128-cts:normal lustre_root@REALM


Note - If one client is compromised, all client nodes become insecure.


For more detailed information on installing and configuring Kerberos, see:

http://web.mit.edu/Kerberos/krb5-1.6/#documentation

11.2.1.5 Setting the Environment

Perform the following steps to configure the system and network to use Kerberos.

System-wide Configuration

1. On each MDT, OST, and client node, add the following line to /etc/fstab to mount them automatically.

nfsd	/proc/fs/nfsd	nfsd	defaults	0	0

2. On each MDT and client node, dd the following line to /etc/request-key.conf.

create lgssc * * /usr/sbin/lgss_keyring %o %k %t %d %c %u %g %T %P %S
Networking

If your network is not using SOCKLND or InfiniBand (and uses Quadrics, Elan or Myrinet, for example), configure a /etc/lustre/nid2hostname (simple script that translates a NID to a hostname) on each server node (MDT and OST). This is an example on an Elan cluster:

#!/bin/bash
set -x
exec 2>/tmp/$(basename $0).debug
 
# convert a NID for a LND to a hostname, for GSS for example
 
# called with three arguments: lnd netid nid
#   $lnd will be string "QSWLND", "GMLND", etc.
#   $netid will be number in hex string format, like "0x16", etc.
#   $nid has the same format as $netid
# output the corresponding hostname, or error message leaded by a '@' for error logging.
 
lnd=$1
netid=$2
nid=$3
 
# uppercase the hex
nid=$(echo $nid | tr '[abcdef]' '[ABCDEF]')
# and convert to decimal
nid=$(echo -e "ibase=16\n${nid/#0x}" | bc)
case $lnd in
 
	QSWLND)			# simply stick "mtn" on the front
 
				echo "mtn$nid"
				;;
 
				*)        echo "@unknown LND: $lnd"
 
				;;
esac

11.2.1.6 Building Lustre

If you are compiling the kernel from the source, enable GSS during configuration:

# ./configure --with-linux=path_to_linux_source --enable-gss - other-options

When you enable Lustre with GSS, the configuration script checks all dependencies, like Kerberos and libgssapi installation, and in-kernel Sun RPC-related facilities. When you install lustre-xxx.rpm on target machines, RPM again checks for dependencies like Kerberos and libgssapi.

11.2.1.7 Running GSS Daemons

If you turn on GSS between an MDT-OST or MDT-MDT, GSS treats the MDT as a client. You should run lgssd on the MDT.

There are two types of GSS daemons: lgssd and lsvcgssd. Before starting Lustre, make sure they are running on each node:



Note - Verbose logging can help you make sure Kerberos is set up correctly. To use verbose logging and run it in the foreground, run lsvcgssd -vvv -f

-v increases the verbose level of a debugging message by 1. For example, to set the verbose level to 3, run lsvcgssd -v -v -v

-f runs lsvcgssd in the foreground, instead of as daemon.


We are maintaining a patch against nfs-utils, and bringing necessary patched files into the Lustre tree. After a successful build, GSS daemons are built under lustre/utils/gss and are part of lustre-xxxx.rpm.

11.2.2 Types of Lustre-Kerberos Flavors

There are three major flavors in which you can configure Lustre with Kerberos:

Select a flavor depending on your priorities and preferences.

11.2.2.1 Basic Flavors

Currently, we support six basic flavors: null, plain, krb5n, krb5a, krb5i, and krb5p.


Basic Flavor

Authentication

RPC Message Protection

Bulk Data Protection

Remarks

null

N/A

N/A

N/A[2]

Almost no performance overhead. The on-wire RPC data is compatible with old versions of Lustre (1.4.x, 1.6.x).

plain

N/A

null

checksum
(adler32)

Carries checksum (which only protects data mutating during transfer, cannot guarantee the genuine author because there is no actual authentication).

krb5n

GSS/Kerberos5

null

checksum
(adler32)

No protection of the RPC message, adler32 checksum protection of bulk data; light performance overhead.

krb5a

GSS/Kerberos5

partial integrity

checksum (adler32)

Only the header of the RPC message is integrity protected, adler32 checksum protection of bulk data, more performance overhead compared to krb5n.

krb5i

GSS/Kerberos5

integrity

integrity [sha1]

RPC message integrity protection algorithm is determined by actual Kerberos algorithms in use; heavy performance overhead.

krb5p

GSS/Kerberos5

privacy

privacy [sha1/aes128]

 

RPC message privacy protection algorithm is determined by actual Kerberos algorithms in use; heaviest performance overhead.


11.2.2.2 Security Flavor

A security flavor is a string that describes what kind of security transform is performed on a given PTLRPC connection. It covers two parts of messages, the RPC message and BULK data. You can set either part in one of the following modes:

11.2.2.3 Customized Flavor

In most situations, you do not need a customized flavor, a basic flavor is sufficient for regular use. But to some extent, you can customize the flavor string. The flavor string format is:

base_flavor[-bulk{nip}[:hash_alg[/cipher_alg]]] 

Here are some examples of customized flavors:

plain-bulkn

Use plain on the RPC message (null protection), and no protection on the bulk transfer.

krb5i-bulkn

Use krb5i on the RPC message, but do not protect the bulk transfer.

krb5p-bulki

Use krb5p on the RPC message, and protect data integrity of the bulk transfer.

krb5p-bulkp:sha512/aes256

Use krb5p on the RPC message, and protect data privacy of the bulk transfer by algorithm SHA512 and AES256.

Currently, Lustre supports these bulk data cryptographic algorithms:

11.2.2.4 Specifying Security Flavors

If you have not specified a security flavor, the CLIENT-MDT connection defaults to plain, and all other connections use null.

Specifying Flavors by Mount Options

When mounting OST or MDT devices, add the mount option (shown below) to specify the security flavor:

# mount -t lustre -o sec=plain /dev/sda1 /mnt/mdt/

This means all connections to this device will use the plain flavor. You can split this sec=flavor as:

# mount -t lustre -o sec_mdt={flavor1},sec_cli={flavor1}/dev/sda \ /mnt/mdt/

This means connections from other MDTs to this device will use flavor1, and connections from all clients to this device will use flavor2.

Specifying Flavors by On-Disk Parameters

You can also specify the security flavors by specifying on-disk parameters on OST and MDT devices:

# tune2fs -o security.rpc.mdt=flavor1 -o security.rpc.cli=flavor2 \ device

On-disk parameters are overridden by mount options.

11.2.2.5 Mounting Clients

Root on client node mounts Lustre without any special tricks.

11.2.2.6 Rules, Syntax and Examples

The general rules and syntax for using Kerberos are:

<target>.srpc.flavor.<network>[.<direction>]=flavor

Examples:

mgs> lctl conf_param lustre.srpc.flavor.default=krb5i
mgs> lctl conf_param lustre.srpc.flavor.tcp0=krb5p
mgs> lctl conf_param lustre.srpc.flavor.default=null
mgs> lctl conf_param lustre.srpc.flavor.tcp0=krb5p
mgs> lctl conf_param lustre.srpc.flavor.elan1=plain
mgs> lctl conf_param lustre.srpc.flavor.default.cli2mdt=krb5i
mgs> lctl conf_param lustre.srpc.flavor.default.cli2ost=krb5i
mgs> lctl conf_param lustre.srpc.flavor.default.mdt2mdt=null
mgs> lctl conf_param lustre.srpc.flavor.default.mdt2ost=plain

11.2.2.7 Authenticating Normal Users

On client nodes, non-root users must use kinit to access Lustre (just like other Kerberized applications). kinit is used to obtain and cache Kerberos ticket-granting tickets. Two requirements to authenticating users:

To destroy the established security contexts before logging out, run lfs flushctx:

# lfs flushctx [-k]

Here -k also means destroy the on-disk Kerberos credential cache. It is equivalent to kdestroy. Otherwise, it only destroys established contexts in the Lustre kernel.

 


1 (Footnote) Kerberos keytab file maintenance utility.
2 (TableFootnote) In Lustre 1.6.5, bulk data checksumming is enabled (by default) to provide integrity checking using the adler32 mechanism if the OSTs support it. Adler32 checksums offer lower CPU overhead than CRC32.