Lustre Multi Tenancy

From Lustre Wiki
Jump to navigation Jump to search

Overview

Multi-tenancy lets separate groups of users share one Lustre filesystem without being able to see or reach each other's data, even when they have root on their own clients. It is built from three pieces:

  • Nodemaps group clients by NID. Each group gets its own identity mapping (UID/GID/PROJID), root squash, and policy (read-only mounts, encryption, which admin operations are allowed, and more).
  • Filesets confine a nodemap's clients to one or more subdirectories, which those clients see as the filesystem root.
  • Client identification ensures a client really is the node it claims to be. Nodemaps trust NIDs, so isolation is only as strong as your guarantee that one tenant cannot use another tenant's NID. Use a trusted network design, or strong authentication with Kerberos or Shared-Secret Key. SSK keys can be bound to a specific nodemap.

For the full nodemap reference, see UID/GID Mapping and the Mapping UIDs and GIDs with Nodemap chapter of the Lustre Operations Manual. For the broader security picture, see Setting up Lustre Security.

Run all lctl nodemap_* commands on the MGS. Since Lustre 2.9 the configuration is stored on the MGS and propagated to all servers automatically, usually within about ten seconds. During that window, clients may be served under either the old or the new settings, so make changes in a maintenance window, or while the affected clients are not writing.

Before You Start: A Nodemap for the Servers

Do this first. Once nodemaps are active, any node that matches no nodemap falls into the default nodemap, which squashes root and maps every user to nobody. That includes Lustre servers, which act as clients when they talk to each other. The manual requires a nodemap that covers all Lustre server nodes, with both admin and trusted set: 

mgs# lctl nodemap_add TrustedSystems
mgs# lctl nodemap_add_range --name TrustedSystems --range 10.0.0.[1-20]@tcp
mgs# lctl nodemap_modify --name TrustedSystems --property admin --value 1
mgs# lctl nodemap_modify --name TrustedSystems --property trusted --value 1

Put administrative clients (backup nodes, data movers, and the nodes you run chown from) in this nodemap or a similar one. You need at least one group with full access for maintenance.

Nodemaps

Creating a Nodemap and Assigning Clients

mgs# lctl nodemap_add tenantA
mgs# lctl nodemap_add_range --name tenantA --range 10.0.1.[0-255]@tcp
mgs# lctl nodemap_add_range --name tenantA --range 10.0.2.[0-255]@tcp

Clients whose NIDs fall in these ranges belong to tenantA. To remove a range with nodemap_del_range, give exactly the range you added. You cannot remove a sub-range or a partly overlapping range.

The default Nodemap

Clients that match no nodemap belong to the built-in default nodemap, which cannot be deleted. Only some properties can be set on it (admin, trusted, squash_uid, squash_gid, fileset, audit_mode, and rbac), and no ID mappings can be defined on it. With its default values (admin=0, trusted=0), clients that match no other nodemap have root squashed and every other user squashed to nobody.

Be careful changing admin or trusted on default. Every unlisted node inherits the change, possibly including your servers.

Properties

Set properties with lctl nodemap_modify --name NAME --property PROPERTY --value VALUE. The ones that matter most for tenancy:

Property Default Meaning
admin 0 1 = do not squash root. 0 = map root to squash_uid/squash_gid.
trusted 0 1 = use client IDs as-is (no mapping) for non-root users. 0 = map users through the nodemap's idmaps or offset, and squash unmapped users.
squash_uid, squash_gid, squash_projid 65534 (nobody) The ID that root (when admin=0) and unmapped IDs (when trusted=0) are mapped to. Cannot be 0. squash_projid exists since 2.15.
deny_unknown 0 1 = deny unmapped users entirely instead of squashing them. This includes root, when admin=0 and root is not mapped.
map_mode all Which ID types are mapped: one or more of uid, gid, projid, both (UIDs and GIDs), or all.
readonly_mount 0 1 = clients are forced to a read-only mount unless they explicitly specify otherwise.
deny_mount 0 1 = refuse new client mounts from this nodemap. Existing mounts keep working until a remount is attempted.
forbid_encryption 0 1 = prevent clients from using client-side encryption.
audit_mode 1 0 = do not record this nodemap's access events in Changelogs. Useful for backup or HSM agent nodes that would otherwise flood them.
rbac all Admin operations the nodemap's clients may perform, such as quota_ops, chlg_ops, dne_ops, file_perms, fscrypt_admin, hsm_ops, and byfid_ops. Roles not listed are forbidden. Accepts +role/-role to add or remove a single role.
gssonly_identification 0 1 = identify clients by the GSS token they authenticate with instead of by NID. Such a nodemap cannot have NID ranges.

Not every property exists in every release. Check lctl-nodemap-modify(8) on your version, or run lctl nodemap_info --list (2.16.52 and later).

How admin and trusted combine:

  • Both 0 (the defaults): root is squashed, mapped users get their mapped IDs, and everyone else is squashed. This is the normal setting for a tenant.
  • admin=1, trusted=0: root stays root, explicit idmaps are honored, and everyone else is squashed.
  • admin=0, trusted=1: full access to canonical IDs, but root is squashed.
  • Both 1: unrestricted, as if nodemaps were off. Use this only for servers and administrative nodes.

Mapping IDs

With trusted=0, only mapped IDs keep a distinct identity. Everything else is squashed, or denied if deny_unknown=1. There are two mapping mechanisms.

Explicit idmaps map a client ID to a filesystem ID: 

mgs# lctl nodemap_add_idmap --name tenantA --idtype uid --idmap 500:10500
mgs# lctl nodemap_add_idmap --name tenantA --idtype gid --idmap 500:10500
mgs# lctl nodemap_add_idmap --name tenantA --idtype projid --idmap 33:1

UID and GID maps are independent of each other. Lustre 2.16 and later also accept a range: 

mgs# lctl nodemap_add_idmap --name tenantA --idtype uid --idmap 1000-1999:11000-11999

On earlier releases, each ID must be mapped one at a time. To remove a mapping, run lctl nodemap_del_idmap with the same arguments.

Offsets (Lustre 2.16 and later) shift a tenant's whole ID space for UIDs, GIDs, and PROJIDs at once: fs_id = client_id + offset, for client IDs 0 through limit-1. This is the simplest way to give each tenant a non-overlapping ID range: 

mgs# lctl nodemap_add_offset --name tenantA --offset 100000 --limit 100000
mgs# lctl nodemap_add_offset --name tenantB --offset 200000 --limit 100000

Offset rules:

  • Offset ranges of different nodemaps cannot overlap, and a nodemap can have only one offset.
  • The offset is applied last, after any explicit idmap. Keep idmap targets and squash_* values within 0 to limit-1, or the resulting IDs fall outside the tenant's range.
  • Choose a limit large enough to include nobody (65534).
  • Choose the offset once. Changing or deleting it does not remap existing files; they must be fixed with chown and lfs project from a trusted client.

Make sure the resulting filesystem IDs resolve on the MDS (through LDAP or another identity source), or set the MDS identity_upcall to NONE.

Activating Nodemaps

Nodemaps have no effect until the feature is activated, globally, on the MGS: 

mgs# lctl nodemap_activate 1

lctl nodemap_activate 0 turns it off again. Check the current state with lctl get_param nodemap.active. After activating, check that the mappings are correct before letting tenants mount.

Viewing the Configuration

# Lustre 2.16.52 and later
mgs# lctl nodemap_info --list             # global state, nodemap names, property names
mgs# lctl nodemap_info --name tenantA     # every property of one nodemap
mgs# lctl nodemap_info --property ranges  # one property across all nodemaps

# Any version
mgs# lctl get_param nodemap.tenantA.*

Filesets

A fileset confines a nodemap's clients to a subdirectory. They mount it as their filesystem root and cannot see or reach anything outside it, including through ...

Create the directory first, from a client in a trusted nodemap. If a nodemap's primary fileset does not exist, no client in that nodemap can mount at all. 

trusted-client# mkdir /mnt/lustre/tenantA

Then set its owner and permissions to match the tenant's mapped filesystem IDs.

Lustre 2.17 and Later: nodemap_fileset_add

mgs# lctl nodemap_fileset_add --name tenantA --fileset /tenantA

Each nodemap can have:

  • One primary fileset, used as the root when a client mounts without naming a subdirectory.
  • Up to 255 alternate filesets (--alt). Clients mount these by naming them explicitly; an alternate never becomes the default.
  • A read-only flag on any fileset (--ro). Mounts that resolve to that fileset are forced read-only.
mgs# lctl nodemap_fileset_add --name tenantA --fileset /tenantA_scratch --alt
mgs# lctl nodemap_fileset_add --name tenantA --fileset /reference --alt --ro

Use lctl nodemap_fileset_modify to change a fileset (rename it, switch it between primary and alternate, or toggle --ro/--rw), and lctl nodemap_fileset_del to remove one (--fileset PATH) or all of them (--all). The nodemap's readonly_mount property overrides a read-write fileset.

Before Lustre 2.17: nodemap_set_fileset

Earlier releases support only one fileset per nodemap: 

mgs# lctl nodemap_set_fileset --name tenantA --fileset /tenantA
mgs# lctl nodemap_set_fileset --name tenantA --fileset ""     # remove it

Running nodemap_set_fileset again replaces the fileset; it does not add a second one. Before 2.17, also store the fileset persistently, so that servers still enforce it after a restart: 

mgs# lctl set_param -P nodemap.tenantA.fileset=/tenantA

nodemap_set_fileset is deprecated in newer releases and affects only the primary fileset. Use nodemap_fileset_add, nodemap_fileset_del, and nodemap_fileset_modify where they are available.

How Clients Mount

With a primary fileset of /tenantA, a tenantA client's ordinary mount is silently redirected: 

tenantA-client# mount -t lustre mgs@tcp:/testfs /mnt/lustre
# /mnt/lustre now shows /tenantA

Clients can also mount an explicit subdirectory that matches one of their filesets. The full precedence rules, including how a requested path combines with the primary fileset, are in lctl-nodemap-fileset-add(8). Test the exact mounts your tenants will use before going live.

Complete Example: Two Tenants

This sets up two isolated tenants on Lustre 2.17 or later. On older releases, use nodemap_set_fileset plus set_param -P instead of nodemap_fileset_add; before 2.16, use explicit idmaps instead of offsets. 

# 0. Servers and admin nodes get full access (see "Before You Start")
mgs# lctl nodemap_add TrustedSystems
mgs# lctl nodemap_add_range --name TrustedSystems --range 10.0.0.[1-20]@tcp
mgs# lctl nodemap_modify --name TrustedSystems --property admin --value 1
mgs# lctl nodemap_modify --name TrustedSystems --property trusted --value 1

# 1. Tenant nodemaps and their client NIDs
mgs# lctl nodemap_add tenantA
mgs# lctl nodemap_add tenantB
mgs# lctl nodemap_add_range --name tenantA --range 10.0.1.[0-255]@tcp
mgs# lctl nodemap_add_range --name tenantB --range 10.0.2.[0-255]@tcp

# 2. Non-overlapping ID spaces
mgs# lctl nodemap_add_offset --name tenantA --offset 100000 --limit 100000
mgs# lctl nodemap_add_offset --name tenantB --offset 200000 --limit 100000

# 3. Nothing to do: admin=0 and trusted=0 are the defaults, so root is squashed and IDs are mapped

# 4. Tenant directories, owned by each tenant's mapped ID (client UID/GID 1000 here)
trusted-client# mkdir /mnt/lustre/tenantA /mnt/lustre/tenantB
trusted-client# chown 101000:101000 /mnt/lustre/tenantA
trusted-client# chown 201000:201000 /mnt/lustre/tenantB

# 5. Filesets
mgs# lctl nodemap_fileset_add --name tenantA --fileset /tenantA
mgs# lctl nodemap_fileset_add --name tenantB --fileset /tenantB

# 6. Activate, then check before tenants mount
mgs# lctl nodemap_activate 1
mgs# lctl nodemap_info

Result:

  • tenantA clients see only /tenantA. Their UID 1000 is stored on disk as 101000, and their root is squashed.
  • tenantB clients see only /tenantB. Their UID 1000 is stored as 201000.
  • The two tenants can use the same client UIDs without colliding on disk.
  • Nodes in none of the three nodemaps fall into default: root squashed, every user mapped to nobody.

To stop a tenant from impersonating another by spoofing a NID, also issue each tenant an SSK key bound to its nodemap. See Binding Keys to Nodemaps.

Removing a Tenant

mgs# lctl nodemap_del tenantA

Deleting a nodemap removes its ranges, mappings, and properties, and its clients fall back to the default nodemap. Files keep the filesystem IDs they were written with.

See Also

Retrieved from "http://wiki.lustre.org/index.php?title=Lustre_Multi_Tenancy&oldid=5761"