Elliswilson: Created page with "== Lustre File System Expansion: Adding MDTs and OSTs == This page provides step-by-step procedures for adding new OSTs and MDTs to an existing Lustre filesystem. These operations can be performed '''online''' — the filesystem remains available to clients during the expansion. For removing OSTs or MDTs, see Lustre OST MDT Removal. === Adding a New OST === Use this procedure to increase storage capacity. ==== Prerequisites ==== * A block device (or ZFS pool) f..."

2026-07-06T14:27:05Z

Created page with "== Lustre File System Expansion: Adding MDTs and OSTs == This page provides step-by-step procedures for adding new OSTs and MDTs to an existing Lustre filesystem. These operations can be performed '''online''' — the filesystem remains available to clients during the expansion. For removing OSTs or MDTs, see Lustre OST MDT Removal. === Adding a New OST === Use this procedure to increase storage capacity. ==== Prerequisites ==== * A block device (or ZFS pool) f..."

New page

== Lustre File System Expansion: Adding MDTs and OSTs ==

This page provides step-by-step procedures for adding new OSTs and
MDTs to an existing Lustre filesystem. These operations can be
performed '''online''' — the filesystem remains available to clients
during the expansion.

For removing OSTs or MDTs, see [[Lustre OST MDT Removal]].

=== Adding a New OST ===

Use this procedure to increase storage capacity.

==== Prerequisites ====

* A block device (or ZFS pool) for the new OST on the target OSS.
* The OSS has Lustre server packages installed.
* The OSS can reach the MGS on port 988.
* The next available OST index. Check with: lctl dl | grep osc lfs osts /mnt/lustre

==== Procedure ====

'''Step 1: Format the new OST.'''

On the OSS node:

mkfs.lustre --fsname=testfs --mgsnode=mgsnode@tcp \
--ost --index=<next_index> /dev/sdb

For ZFS:

mkfs.lustre --backfstype=zfs --fsname=testfs --mgsnode=mgsnode@tcp \
--ost --index=<next_index> ost_pool/ost_dataset /dev/sdb

'''Step 2: Create mount point and mount the OST.'''

mkdir -p /mnt/testfs/ost<N>
mount -t lustre /dev/sdb /mnt/testfs/ost<N>

'''Step 3: Verify the new OST is visible.'''

On any client:

lfs df /mnt/lustre

The new OST should appear in the list with its full capacity.

'''Step 4: (Optional) Rebalance existing data.'''

New files will automatically prefer OSTs with more free space (via
the weighted allocator). To rebalance existing files:

# Migrate files from a specific full OST to the new one:
lfs find --ost testfs-OST0004 -size +4G /mnt/lustre | lfs_migrate -y

# Or migrate an entire directory:
lfs_migrate /mnt/lustre/old_project/

'''Note:''' For best load balancing, configure the same number of
OSTs per OSS across all OSSs.

=== Adding a New MDT (DNE) ===

Use this procedure to increase metadata capacity or performance by
distributing the namespace across multiple MDS servers.

==== When to Add an MDT ====

* The existing MDT(s) are approaching the 4 billion inode limit (ldiskfs).
* Metadata operation rates exceed what a single MDS can handle.
* You want to isolate different projects onto different MDTs.

==== Prerequisites ====

* A block device for the new MDT on the MDS server. SSD/NVMe strongly recommended.
* Lustre server packages installed on the MDS.
* The next available MDT index.

==== Procedure ====

'''Step 1: Find the next available MDT index.'''

client$ lctl dl | grep mdc

Each MDT has a unique index starting from 0.

'''Step 2: Format the new MDT.'''

On the MDS node:

mkfs.lustre --reformat --fsname=testfs --mdt \
--mgsnode=mgsnode@tcp --index=<next_index> /dev/sdc

'''Step 3: Mount the new MDT.'''

mkdir -p /mnt/mdt<N>
mount -t lustre /dev/sdc /mnt/mdt<N>

'''Step 4: Create directories on the new MDT.'''

New MDTs do '''not''' automatically receive new files. You must
explicitly create directories on them:

# Create a directory managed by the new MDT:
lfs mkdir -i <mdt_index> /mnt/lustre/new_project

# Create a striped directory across multiple MDTs (Lustre 2.8+):
lfs mkdir -c 4 /mnt/lustre/large_project

# Set auto-distribution for new subdirectories (Lustre 2.13+):
lfs setdirstripe -D -c 1 -i -1 /mnt/lustre/auto_balanced/

'''Note:''' In Lustre 2.15+, if no default directory layout is set
on the root directory, the MDS auto-configures round-robin
distribution across all MDTs for new top-level subdirectories.

'''Step 5: Verify.'''

# Check which MDT manages a directory:
lfs getstripe --mdt-index /mnt/lustre/new_project

# Check MDT usage:
lfs df -i /mnt/lustre

==== Important Notes ====

* The root directory always remains on MDT0000.
* Files and subdirectories below a directory inherit its MDT unless explicitly overridden.
* Clients older than Lustre 2.4 cannot access directories on MDTs other than MDT0000.

=== Updating fstab ===

After adding new targets, update <code>/etc/fstab</code> on the
relevant servers for persistence across reboots:

LABEL=testfs-OST0012 /mnt/testfs/ost12 lustre defaults,_netdev,noauto 0 0
LABEL=testfs-MDT0004 /mnt/testfs/mdt4 lustre defaults,_netdev,noauto 0 0

If using Pacemaker for HA, update the resource configuration instead
of fstab.

=== See Also ===

* [[Lustre OST MDT Removal]] — Deactivating and removing targets
* [[Lustre Configuration Log Management]] — writeconf, replace_nids
* [[Handling Full OSTs]] — Managing full OSTs
* [[Lustre Deployment Patterns]] — Reference architectures
* [[Lustre Hardware Sizing Guide]] — Planning target sizes

Lustre File System Expansion - Revision history