Lustre File System Expansion

Lustre File System Expansion: Adding MDTs and OSTs

This page provides step-by-step procedures for adding new OSTs and MDTs to an existing Lustre filesystem. These operations can be performed online — the filesystem remains available to clients during the expansion.

For removing OSTs or MDTs, see Lustre OST MDT Removal.

Adding a New OST

Use this procedure to increase storage capacity.

Prerequisites

A block device (or ZFS pool) for the new OST on the target OSS.
The OSS has Lustre server packages installed.
The OSS can reach the MGS on port 988.
The next available OST index. Check with: lctl dl | grep osc lfs osts /mnt/lustre

Procedure

Step 1: Format the new OST.

On the OSS node:

mkfs.lustre --fsname=testfs --mgsnode=mgsnode@tcp \
    --ost --index=<next_index> /dev/sdb

For ZFS:

mkfs.lustre --backfstype=zfs --fsname=testfs --mgsnode=mgsnode@tcp \
    --ost --index=<next_index> ost_pool/ost_dataset /dev/sdb

Step 2: Create mount point and mount the OST.

mkdir -p /mnt/testfs/ost<N>
mount -t lustre /dev/sdb /mnt/testfs/ost<N>

Step 3: Verify the new OST is visible.

On any client:

lfs df /mnt/lustre

The new OST should appear in the list with its full capacity.

Step 4: (Optional) Rebalance existing data.

New files will automatically prefer OSTs with more free space (via the weighted allocator). To rebalance existing files:

# Migrate files from a specific full OST to the new one:
lfs find --ost testfs-OST0004 -size +4G /mnt/lustre | lfs_migrate -y

# Or migrate an entire directory:
lfs_migrate /mnt/lustre/old_project/

Note: For best load balancing, configure the same number of OSTs per OSS across all OSSs.

Adding a New MDT (DNE)

Use this procedure to increase metadata capacity or performance by distributing the namespace across multiple MDS servers.

When to Add an MDT

The existing MDT(s) are approaching the 4 billion inode limit (ldiskfs).
Metadata operation rates exceed what a single MDS can handle.
You want to isolate different projects onto different MDTs.

Prerequisites

A block device for the new MDT on the MDS server. SSD/NVMe strongly recommended.
Lustre server packages installed on the MDS.
The next available MDT index.

Procedure

Step 1: Find the next available MDT index.

client$ lctl dl | grep mdc

Each MDT has a unique index starting from 0.

Step 2: Format the new MDT.

On the MDS node:

mkfs.lustre --reformat --fsname=testfs --mdt \
    --mgsnode=mgsnode@tcp --index=<next_index> /dev/sdc

Step 3: Mount the new MDT.

mkdir -p /mnt/mdt<N>
mount -t lustre /dev/sdc /mnt/mdt<N>

Step 4: Create directories on the new MDT.

New MDTs do not automatically receive new files. You must explicitly create directories on them:

# Create a directory managed by the new MDT:
lfs mkdir -i <mdt_index> /mnt/lustre/new_project

# Create a striped directory across multiple MDTs (Lustre 2.8+):
lfs mkdir -c 4 /mnt/lustre/large_project

# Set auto-distribution for new subdirectories (Lustre 2.13+):
lfs setdirstripe -D -c 1 -i -1 /mnt/lustre/auto_balanced/

Note: In Lustre 2.15+, if no default directory layout is set on the root directory, the MDS auto-configures round-robin distribution across all MDTs for new top-level subdirectories.

Step 5: Verify.

# Check which MDT manages a directory:
lfs getstripe --mdt-index /mnt/lustre/new_project

# Check MDT usage:
lfs df -i /mnt/lustre

Important Notes

The root directory always remains on MDT0000.
Files and subdirectories below a directory inherit its MDT unless explicitly overridden.
Clients older than Lustre 2.4 cannot access directories on MDTs other than MDT0000.

Updating fstab

After adding new targets, update /etc/fstab on the relevant servers for persistence across reboots:

LABEL=testfs-OST0012 /mnt/testfs/ost12 lustre defaults,_netdev,noauto 0 0
LABEL=testfs-MDT0004 /mnt/testfs/mdt4  lustre defaults,_netdev,noauto 0 0

If using Pacemaker for HA, update the resource configuration instead of fstab.

Lustre File System Expansion

Contents