Lustre OST MDT Removal

Lustre OST and MDT Removal

This page covers deactivating and removing OSTs and MDTs from a Lustre filesystem. These are high-stakes operations — read the warnings carefully before proceeding.

For adding targets, see Lustre File System Expansion.

Removing an OST

There are three scenarios: migrating data off a working OST, handling a permanently failed OST, and temporary deactivation.

Scenario 1: Migrating Data Off a Working OST

Use this when you want to decommission an OST gracefully.

Step 1: Disable new file creation on the OST.

Run on all MDS nodes (important — if you have DNE, do this on every MDS):

lctl set_param osp.testfs-OST0004-osc-MDT*.max_create_count=0

To make it persistent:

lctl set_param -P osp.testfs-OST0004-osc-MDT*.max_create_count=0

Step 2: Migrate files off the OST.

# Find and migrate all files on the OST:
lfs find --ost testfs-OST0004 /mnt/lustre | lfs_migrate -y

# For large filesystems, migrate in batches:
lfs find --ost testfs-OST0004 -size +1G /mnt/lustre | lfs_migrate -y
lfs find --ost testfs-OST0004 /mnt/lustre | lfs_migrate -y

Warning: Do NOT deactivate the OST on clients during migration. Client-side deactivation causes I/O errors and migration will fail.

Step 3: Verify all files are migrated.

lfs find --ost testfs-OST0004 /mnt/lustre | wc -l

This should return 0. If not, repeat the migration.

Step 4: Permanently deactivate the OST.

On the MGS:

# Lustre 2.16+:
lctl del_ost --target testfs-OST0004

# Older versions:
lctl conf_param testfs-OST0004.osc.active=0

Step 5: Unmount and (optionally) remove the OST device.

Scenario 2: Permanently Failed OST

If an OST has failed and cannot be recovered:

Step 1: Identify files with data on the failed OST.

lfs find --ost testfs-OST0004 -print0 /mnt/lustre > /tmp/affected_files

Step 2: Delete or restore affected files.

Files with data on the failed OST will return I/O errors. You must delete them or restore from backup:

# Delete affected files:
cat /tmp/affected_files | xargs -0 -n 1 unlink

Step 3: Permanently deactivate the OST.

On the MGS:

lctl conf_param testfs-OST0004.osc.active=0

Scenario 3: Temporary OST Deactivation

To temporarily take an OST offline for maintenance:

# On each client (temporary, resets on remount):
lctl set_param osc.testfs-OST0004-*.active=0

# On MDS (prevents new allocations):
lctl set_param osp.testfs-OST0004-osc-MDT*.max_create_count=0

To bring it back:

# On MDS:
lctl set_param osp.testfs-OST0004-osc-MDT*.max_create_count=20000

# On each client:
lctl set_param osc.testfs-OST0004-*.active=1

Returning a Permanently Deactivated OST to Service

# On MGS:
lctl conf_param testfs-OST0004.osc.active=1

Removing an MDT

Removing an MDT is more complex than removing an OST because it contains namespace data (directories, filenames, permissions).

If the MDT Is Available

You have two options, depending on whether you want to keep or discard the data currently stored under the MDT you are removing.

Option A (recommended): Migrate directories off the MDT, then remove it.

This preserves your data. The directory inodes — and the files within them — are moved to another MDT, leaving the target MDT empty so it can be safely deactivated. This is the approach documented in the Lustre Operations Manual for removing an MDT, and it avoids the data loss that rm -r would cause.

Step 1: Find the directories whose inodes live on the MDT to be removed (for example, MDT index 3):

# Show which MDT currently holds a given directory:
lfs getdirstripe --mdt-index /mnt/lustre/some_directory

Step 2: Migrate each such directory to a surviving MDT (for example, MDT0000). Migration is recursive by default, moving the directory together with all subdirectories and inodes beneath it:

lfs migrate -m 0 /mnt/lustre/some_directory

Step 3: Verify the directory and its files now report the new MDT index.

lfs getdirstripe --mdt-index /mnt/lustre/some_directory
lfs getstripe --mdt-index /mnt/lustre/some_directory/*

Repeat until no directories or inodes remain on the MDT being removed.

Migration caveats:

• Only the root user can migrate directories.
• Migration operates on whole directories. Migrating or restriping striped directories requires Lustre 2.12 or later.
• Migration cannot be interrupted once started, although the directory and its files remain accessible during the operation.
• Each migrated file receives a new FID and therefore reports a new inode number to userspace. Tools that identify files by inode number (backup/archive tools, NFS, Samba) may treat migrated files as new. If the filesystem is re-exported over NFS, cached stale file handles can make files temporarily inaccessible; restart the NFS service (and possibly the clients) to clear them.
• Data-on-MDT (DoM) files cannot be migrated between MDTs directly. First migrate them to a non-DoM layout, then migrate the inodes.
• Files that are currently open or have been archived by HSM will fail to migrate; rerun the same command once they are available.
• The root directory always resides on MDT0000, so MDT0000 itself cannot be emptied and removed by this procedure.

Step 4: Once the MDT is empty, deactivate it on the MGS.

lctl conf_param testfs-MDT0003.mdc.active=0

Option B: Delete the data on the MDT.

Use this only if you genuinely want to destroy everything stored under that MDT. Unlike migration, rm -r permanently deletes the directories and every file beneath them — it empties the MDT by throwing the data away, not by relocating it:

rm -r /mnt/lustre/directory_on_mdt

Then deactivate the MDT on the MGS:

lctl conf_param testfs-MDT0003.mdc.active=0

If the MDT Is Permanently Inaccessible

Standard rmdir will fail with I/O error for directories on an inaccessible MDT. Use lfs rm_entry instead:

lfs rm_entry /mnt/lustre/directory_on_failed_mdt

Then mark the MDT as permanently inactive:

lctl conf_param testfs-MDT0003.mdc.active=0

Check Which MDT Manages a Directory

lfs getstripe --mdt-index /mnt/lustre/some_directory

Warning

All files and directories below an inactive MDT become inaccessible until the MDT is reactivated. Clients will receive EIO errors when accessing them.

Important Warnings

• Do not use lctl conf_param to deactivate a working OST unless you intend it to be permanent. This immediately and permanently deactivates it on both the MDS and all clients.
• Migrate data before deactivating. Once an OST is deactivated on clients, files on it return I/O errors and cannot be migrated.
• Free space/object counts on a deactivated OST won't decrease when files are deleted. Object destruction is deferred until the OST reconnects to the MDS.
• A permanently deactivated OST still appears in filesystem configuration until writeconf is run or it is removed with del_ost.

See Also

• Lustre File System Expansion — Adding MDTs and OSTs
• Handling Full OSTs — Managing full OSTs
• Lustre Configuration Log Management — writeconf, replace_nids