C H A P T E R  22

Managing Lustre Security

This chapter describes Lustre security and includes the following sections:

22.1 Using ACLs

An access control list (ACL), is a set of data that informs an operating system about permissions or access rights that each user or group has to specific system objects, such as directories or files. Each object has a unique security attribute that identifies users who have access to it. The ACL lists each object and user access privileges such as read, write or execute.

22.1.1 How ACLs Work

Implementing ACLs varies between operating systems. Systems that support the Portable Operating System Interface (POSIX) family of standards share a simple yet powerful file system permission model, which should be well-known to the Linux/Unix administrator. ACLs add finer-grained permissions to this model, allowing for more complicated permission schemes. For a detailed explanation of ACLs on Linux, refer to the SuSE Labs article, Posix Access Control Lists on Linux:


We have implemented ACLs according to this model. Lustre works with the standard Linux ACL tools, setfacl, getfacl, and the historical chacl, normally installed with the ACL package.

Note - ACL support is a system-range feature, meaning that all clients have ACL enabled or not. You cannot specify which clients should enable ACL.

22.1.2 Using ACLs with Lustre

POSIX Access Control Lists (ACLs) can be used with Lustre. An ACL consists of file entries representing permissions based on standard POSIX file system object permissions that define three classes of user (owner, group and other). Each class is associated with a set of permissions [read (r), write (w) and execute (x)].

The ls -l command displays the owner, group, and other class permissions in the first column of its output (for example, -rw-r- -- for a regular file with read and write access for the owner class, read access for the group class, and no access for others).

Minimal ACLs have three entries. Extended ACLs have more than the three entries. Extended ACLs also contain a mask entry and may contain any number of named user and named group entries.

The MDS needs to be configured to enable ACLs. Use --mountfsoptions to enable ACLs when creating your configuration:

$ mkfs.lustre --fsname spfs --mountfsoptions=acl --mdt -mgs /dev/sda

Alternately, you can enable ACLs at run time by using the --acl option with mkfs.lustre:

$ mount -t lustre -o acl /dev/sda /mnt/mdt

To check ACLs on the MDS:

$ lctl get_param -n mdc.home-MDT0000-mdc-*.connect_flags | grep acl acl

To mount the client with no ACLs:

$ mount -t lustre -o noacl ibmds2@o2ib:/home /home

ACLs are enabled in Lustre on a system-wide basis; either all clients enable ACLs or none do. Activating ACLs is controlled by MDS mount options acl / noacl (enable/disableACLs). Client-side mount options acl/noacl are ignored. You do not need to change the client configuration, and the “acl” string will not appear in the client /etc/mtab. The client acl mount option is no longer needed. If a client is mounted with that option, then this message appears in the MDS syslog:

...MDS requires ACL support but client does not

The message is harmless but indicates a configuration issue, which should be corrected.

If ACLs are not enabled on the MDS, then any attempts to reference an ACL on a client return an Operation not supported error.

22.1.3 Examples

These examples are taken directly from the POSIX paper referenced above. ACLs on a Lustre file system work exactly like ACLs on any Linux file system. They are manipulated with the standard tools in the standard manner. Below, we create a directory and allow a specific user access.

[root@client lustre]# umask 027
[root@client lustre]# mkdir rain
[root@client lustre]# ls -ld rain
drwxr-x---  2 root root 4096 Feb 20 06:50 rain
[root@client lustre]# getfacl rain
# file: rain
# owner: root
# group: root
[root@client lustre]# setfacl -m user:chirag:rwx rain
[root@client lustre]# ls -ld rain
drwxrwx---+ 2 root root 4096 Feb 20 06:50 rain
[root@client lustre]# getfacl --omit-heade rain

22.2 Using Root Squash

Lustre 1.6 introduced root squash functionality, a security feature which controls super user access rights to an Lustre file system. Before the root squash feature was added, Lustre users could run rm -rf * as root, and remove data which should not be deleted. Using the root squash feature prevents this outcome.

The root squash feature works by re-mapping the user ID (UID) and group ID (GID) of the root user to a UID and GID specified by the system administrator, via the Lustre configuration management server (MGS). The root squash feature also enables the Lustre administrator to specify a set of client for which UID/GID re-mapping does not apply.

22.2.1 Configuring Root Squash

Root squash functionality is managed by two configuration parameters, root_squash and nosquash_nids.


In this example, root squash does not apply to TCP clients on subnet that have an even number as the last component of their IP address.

22.2.2 Enabling and Tuning Root Squash

The default value for nosquash_nids is NULL, which means that root squashing applies to all clients. Setting the root squash UID and GID to 0 turns root squash off.

Root squash parameters can be set when the MDT is created (mkfs.lustre --mdt). For example:

mkfs.lustre --reformat --fsname=Lustre --mdt --mgs \
	    --param "mds.root_squash=500:501" \
	    --param "mds.nosquash_nids='0@elan1 192.168.1.[10,11]'" /dev/sda1

Root squash parameters can also be changed on an unmounted device with tunefs.lustre. For example:

tunefs.lustre --param "mds.root_squash=65534:65534"  \
--param "mds.nosquash_nids=" /dev/sda1

Root squash parameters can also be changed with the lctl conf_param command. For example:

lctl conf_param Lustre.mds.root_squash="1000:100"
lctl conf_param Lustre.mds.nosquash_nids="*@tcp"

Note - When using the lctl conf_param command, keep in mind:

* lctl conf_param must be run on a live MGS
* lctl conf_param causes the parameter to change on all MDSs
* lctl conf_param is to be used once per a parameter

The nosquash_nids list can be cleared with:

lctl conf_param Lustre.mds.nosquash_nids="NONE"

- OR -

lctl conf_param Lustre.mds.nosquash_nids="clear"

If the nosquash_nids value consists of several NID ranges (e.g. 0@elan, 1@elan1), the list of NID ranges must be quoted with single (') or double ('') quotation marks. List elements must be separated with a space. For example:

mkfs.lustre ... --param "mds.nosquash_nids='0@elan1 1@elan2'" /dev/sda1
lctl conf_param Lustre.mds.nosquash_nids="24@elan 15@elan1"

These are examples of incorrect syntax:

mkfs.lustre ... --param "mds.nosquash_nids=0@elan1 1@elan2" /dev/sda1
lctl conf_param Lustre.mds.nosquash_nids=24@elan 15@elan1

To check root squash parameters, use the lctl get_param command:

lctl get_param mds.Lustre-MDT0000.root_squash
lctl get_param mds.Lustre-MDT000*.nosquash_nids

Note - An empty nosquash_nids list is reported as NONE.

22.2.3 Tips on Using Root Squash

Lustre configuration management limits root squash in several ways.

LNET NID range syntax:

<nidlist>	:== <nidrange> [ ' ' <nidrange> ]
<nidrange>	:== <addrrange> '@' <net>
<addrrange>	:== '*' |
		<ipaddr_range> |
<ipaddr_range>	:==
<numaddr_range>	:== <number> |
<expr_list>	:== '[' <range_expr> [ ',' <range_expr>] ']'
<range_expr>	:== <number> |
		<number> '-' <number> |
		<number> '-' <number> '/' <number>
<net>	:== <netname> | <netname><number>
<netname>	:== "lo" | "tcp" | "o2ib" | "cib" | "openib" | "iib" | 
		"vib" | "ra" | "elan" | "gm" | "mx" | "ptl"
<number>	:== <nonnegative decimal> | <hexadecimal>

Note - For networks using numeric addresses (e.g. elan), the address range must be specified in the <numaddr_range> syntax. For networks using IP addresses, the address range must be in the <ipaddr_range>. For example, if elan is using numeric addresses, is incorrect.