| Lustre 1.8 Operations Manual |    
|
| C H A P T E R 11 |
|
Kerberos |
This chapter describes how to use Kerberos with Lustre and includes the following sections:
Kerberos is a mechanism for authenticating all entities (such as users and services) on an “unsafe” network. Users and services, known as "principals", share a secret password (or key) with the Kerberos server. This key enables principals to verify that messages from the Kerberos server are authentic. By trusting the Kerberos server, users and services can authenticate one another.
|
Caution - Kerberos is a future Lustre feature that is not available in current versions. If you want to test Kerberos with a pre-release version of Lustre, check out the Lustre source from the CVS repository and build it. For more information on checking out Lustre source code, see CVS. |
Setting up Lustre with Kerberos can provide advanced security protections for the Lustre network. Broadly, Kerberos offers three types of benefit:
Kerberos uses the “kernel keyring” client upcall mechanism.
This section describes supported Kerberos distributions and how to set up and configure Kerberos on Lustre.
Lustre supports the following Kerberos distributions:
On a number of operating systems, the Kerberos RPMs are installed when the operating system is first installed. To determine if Kerberos RPMs are installed on your OS, run:
# rpm -qa | grep krb
If Kerberos is installed, the command returns a list like this:
krb5-devel-1.4.3-5.1 krb5-libs-1.4.3-5.1 krb5-workstation-1.4.3-5.1 pam_krb5-2.2.6-2.2
| Note - The Heimdal implementation of Kerberos is not currently supported on Lustre, although it support will be added in an upcoming release. |
To set up Lustre with Kerberos:
1. Configure NTP to synchronize time across all machines.
3. Verify that there are fully-qualified domain names (FQDNs), that are resolvable in both forward and reverse directions for all servers. This is required by Kerberos.
4. On every node, install flowing packages:
Some newer Linux distributions include libgssapi by default. If you do not have libgssapi, build and install it from source:
http://www.citi.umich.edu/projects/nfsv4/linux/libgssapi/libssapi-0.10.tar.gz
To configure Lustre for Kerberos:
1. Configure the client nodes.
a. For each client node, create a lustre_root principal and generate the keytab.
kadmin> addprinc -randkey lustre_root/client_host.domain@REALM kadmin> ktadd -e aes128-cts:normal lustre_root/client_host.domain@REALM
b. Install the keytab on the client node.
a. For each MDS node, create a lustre_mds principal and generate the keytab.
kadmin> addprinc -randkey lustre_mds/mdthost.domain@REALM kadmin> ktadd -e aes128-cts:normal lustre_mds/mdthost.domain@REALM
b. Install the keytabl on the MDS node.
a. For each OSS node, create a lustre_oss principal and generate the keytab.
kadmin> addprinc -randkey lustre_oss/osthost.domain@REALM kadmin> ktadd -e aes128-cts:normal lustre_oss/osshost.domain@REALM
b. Install the keytab on the OSS node.
For MIT Kerberos 1.3.x, only des-cbc-md5 works because of a known issue between libgssapi and the Kerberos library.
To configure Kerberos to work with Lustre:
1. Modify the files for Kerberos:
$ /etc/krb5.conf
[libdefaults]
default_realm = CLUSTERFS.COM
[realms]
CLUSTERFS.COM = {
kdc = mds16.clustrefs.com
admin_server = mds16.clustrefs.com
}
[domain_realm]
.clustrefs.com = CLUSTERFS.COM
clustrefs.com = CLSUTREFS.COM
[logging]
default = FILE:/var/log/kdc.log
2. Prepare the Kerberos database.
3. Create service principals so Lustre supports Kerberos authentication.
| Note - You can create service principals when configuring your other services to support Kerberos authentication. |
4. Configure the client nodes. For each client node:
a. Create a lustre_root principal and generate the keytab:
kadmin> addprinc -randkey lustre_root/client_host.domain@REALM kadmin> ktadd -e aes128-cts:normal lustre_root/client_host.domain@REALM
This process populates /etc/krb5.keytab, which is not human-readable. Use the ktutil program to read and modify it.
5. Configure the MDS nodes. For each MDT node, create a lustre_mds principal, and generate and install the keytab.
kadmin> addprinc -randkey lustre_mds/mdthost.domain@REALM kadmin> ktadd -e aes128-cts:normal lustre_mds/mdthost.domain@REALM
6. Configure the OSS nodes. For each OST node, create a lustre_oss principal, and generate and install the keytab.
kadmin> addprinc -randkey lustre_oss/oss_host.domain@REALM kadmin> ktadd -e aes128-cts:normal lustre_oss/oss_host.domain@REALM
To save the trouble of assigning a unique keytab for each client node, create a general lustre_root principal and its keytab, and then install the keytab on as many client nodes as needed.
kadmin> addprinc -randkey lustre_root@REALM kadmin> ktadd -e aes128-cts:normal lustre_root@REALM
| Note - If one client is compromised, all client nodes become insecure. |
For more detailed information on installing and configuring Kerberos, see:
http://web.mit.edu/Kerberos/krb5-1.6/#documentation
Perform the following steps to configure the system and network to use Kerberos.
1. On each MDT, OST, and client node, add the following line to /etc/fstab to mount them automatically.
nfsd /proc/fs/nfsd nfsd defaults 0 0
2. On each MDT and client node, dd the following line to /etc/request-key.conf.
create lgssc * * /usr/sbin/lgss_keyring %o %k %t %d %c %u %g %T %P %S
If your network is not using SOCKLND or InfiniBand (and uses Quadrics, Elan or Myrinet, for example), configure a /etc/lustre/nid2hostname (simple script that translates a NID to a hostname) on each server node (MDT and OST). This is an example on an Elan cluster:
#!/bin/bash
set -x
exec 2>/tmp/$(basename $0).debug
# convert a NID for a LND to a hostname, for GSS for example
# called with three arguments: lnd netid nid
# $lnd will be string "QSWLND", "GMLND", etc.
# $netid will be number in hex string format, like "0x16", etc.
# $nid has the same format as $netid
# output the corresponding hostname, or error message leaded by a '@' for error logging.
lnd=$1
netid=$2
nid=$3
# uppercase the hex
nid=$(echo $nid | tr '[abcdef]' '[ABCDEF]')
# and convert to decimal
nid=$(echo -e "ibase=16\n${nid/#0x}" | bc)
case $lnd in
QSWLND) # simply stick "mtn" on the front
echo "mtn$nid"
;;
*) echo "@unknown LND: $lnd"
;;
esac
If you are compiling the kernel from the source, enable GSS during configuration:
# ./configure --with-linux=path_to_linux_source --enable-gss - other-options
When you enable Lustre with GSS, the configuration script checks all dependencies, like Kerberos and libgssapi installation, and in-kernel Sun RPC-related facilities. When you install lustre-xxx.rpm on target machines, RPM again checks for dependencies like Kerberos and libgssapi.
If you turn on GSS between an MDT-OST or MDT-MDT, GSS treats the MDT as a client. You should run lgssd on the MDT.
There are two types of GSS daemons: lgssd and lsvcgssd. Before starting Lustre, make sure they are running on each node:
We are maintaining a patch against nfs-utils, and bringing necessary patched files into the Lustre tree. After a successful build, GSS daemons are built under lustre/utils/gss and are part of lustre-xxxx.rpm.
There are three major flavors in which you can configure Lustre with Kerberos:
Select a flavor depending on your priorities and preferences.
Currently, we support six basic flavors: null, plain, krb5n, krb5a, krb5i, and krb5p.
|
N/A[2] |
Almost no performance overhead. The on-wire RPC data is compatible with old versions of Lustre (1.4.x, 1.6.x). |
|||
|
Carries checksum (which only protects data mutating during transfer, cannot guarantee the genuine author because there is no actual authentication). |
||||
|
No protection of the RPC message, adler32 checksum protection of bulk data; light performance overhead. |
||||
|
Only the header of the RPC message is integrity protected, adler32 checksum protection of bulk data, more performance overhead compared to krb5n. |
||||
|
RPC message integrity protection algorithm is determined by actual Kerberos algorithms in use; heavy performance overhead. |
||||
|
RPC message privacy protection algorithm is determined by actual Kerberos algorithms in use; heaviest performance overhead. |
A security flavor is a string that describes what kind of security transform is performed on a given PTLRPC connection. It covers two parts of messages, the RPC message and BULK data. You can set either part in one of the following modes:
In most situations, you do not need a customized flavor, a basic flavor is sufficient for regular use. But to some extent, you can customize the flavor string. The flavor string format is:
base_flavor[-bulk{nip}[:hash_alg[/cipher_alg]]]
Here are some examples of customized flavors:
Use plain on the RPC message (null protection), and no protection on the bulk transfer.
Use krb5i on the RPC message, but do not protect the bulk transfer.
Use krb5p on the RPC message, and protect data integrity of the bulk transfer.
Use krb5p on the RPC message, and protect data privacy of the bulk transfer by algorithm SHA512 and AES256.
Currently, Lustre supports these bulk data cryptographic algorithms:
If you have not specified a security flavor, the CLIENT-MDT connection defaults to plain, and all other connections use null.
When mounting OST or MDT devices, add the mount option (shown below) to specify the security flavor:
# mount -t lustre -o sec=plain /dev/sda1 /mnt/mdt/
This means all connections to this device will use the plain flavor. You can split this sec=flavor as:
# mount -t lustre -o sec_mdt={flavor1},sec_cli={flavor1}/dev/sda \ /mnt/mdt/
This means connections from other MDTs to this device will use flavor1, and connections from all clients to this device will use flavor2.
You can also specify the security flavors by specifying on-disk parameters on OST and MDT devices:
# tune2fs -o security.rpc.mdt=flavor1 -o security.rpc.cli=flavor2 \ device
On-disk parameters are overridden by mount options.
Root on client node mounts Lustre without any special tricks.
The general rules and syntax for using Kerberos are:
<target>.srpc.flavor.<network>[.<direction>]=flavor
mgs> lctl conf_param lustre.srpc.flavor.default=krb5i
mgs> lctl conf_param lustre.srpc.flavor.tcp0=krb5p mgs> lctl conf_param lustre.srpc.flavor.default=null
mgs> lctl conf_param lustre.srpc.flavor.tcp0=krb5p mgs> lctl conf_param lustre.srpc.flavor.elan1=plain mgs> lctl conf_param lustre.srpc.flavor.default.cli2mdt=krb5i mgs> lctl conf_param lustre.srpc.flavor.default.cli2ost=krb5i mgs> lctl conf_param lustre.srpc.flavor.default.mdt2mdt=null mgs> lctl conf_param lustre.srpc.flavor.default.mdt2ost=plain
On client nodes, non-root users must use kinit to access Lustre (just like other Kerberized applications). kinit is used to obtain and cache Kerberos ticket-granting tickets. Two requirements to authenticating users:
To destroy the established security contexts before logging out, run lfs flushctx:
# lfs flushctx [-k]
Here -k also means destroy the on-disk Kerberos credential cache. It is equivalent to kdestroy. Otherwise, it only destroys established contexts in the Lustre kernel.
| Lustre 1.8 Operations Manual | 821-0035-12 |
|
Copyright © 2010, Oracle and/or its affiliates. All rights reserved.