Elliswilson: Created page with "== Lustre OST and MDT Removal == This page covers deactivating and removing OSTs and MDTs from a Lustre filesystem. These are '''high-stakes operations''' — read the warnings carefully before proceeding. For adding targets, see Lustre File System Expansion. === Removing an OST === There are three scenarios: migrating data off a working OST, handling a permanently failed OST, and temporary deactivation. ==== Scenario 1: Migrating Data Off a Working OST ==== Us..."

2026-07-06T15:33:41Z

Created page with "== Lustre OST and MDT Removal == This page covers deactivating and removing OSTs and MDTs from a Lustre filesystem. These are '''high-stakes operations''' — read the warnings carefully before proceeding. For adding targets, see Lustre File System Expansion. === Removing an OST === There are three scenarios: migrating data off a working OST, handling a permanently failed OST, and temporary deactivation. ==== Scenario 1: Migrating Data Off a Working OST ==== Us..."

New page

== Lustre OST and MDT Removal ==

This page covers deactivating and removing OSTs and MDTs from a
Lustre filesystem. These are '''high-stakes operations''' — read
the warnings carefully before proceeding.

For adding targets, see [[Lustre File System Expansion]].

=== Removing an OST ===

There are three scenarios: migrating data off a working OST,
handling a permanently failed OST, and temporary deactivation.

==== Scenario 1: Migrating Data Off a Working OST ====

Use this when you want to decommission an OST gracefully.

'''Step 1: Disable new file creation on the OST.'''

Run on '''all MDS nodes''' (important — if you have DNE, do this
on every MDS):

lctl set_param osp.testfs-OST0004-osc-MDT*.max_create_count=0

To make it persistent:

lctl set_param -P osp.testfs-OST0004-osc-MDT*.max_create_count=0

'''Step 2: Migrate files off the OST.'''

# Find and migrate all files on the OST:
lfs find --ost testfs-OST0004 /mnt/lustre | lfs_migrate -y

# For large filesystems, migrate in batches:
lfs find --ost testfs-OST0004 -size +1G /mnt/lustre | lfs_migrate -y
lfs find --ost testfs-OST0004 /mnt/lustre | lfs_migrate -y

'''Warning:''' Do NOT deactivate the OST on clients during migration.
Client-side deactivation causes I/O errors and migration will fail.

'''Step 3: Verify all files are migrated.'''

lfs find --ost testfs-OST0004 /mnt/lustre | wc -l

This should return 0. If not, repeat the migration.

'''Step 4: Permanently deactivate the OST.'''

On the MGS:

# Lustre 2.16+:
lctl del_ost --target testfs-OST0004

# Older versions:
lctl conf_param testfs-OST0004.osc.active=0

'''Step 5: Unmount and (optionally) remove the OST device.'''

==== Scenario 2: Permanently Failed OST ====

If an OST has failed and cannot be recovered:

'''Step 1: Identify files with data on the failed OST.'''

lfs find --ost testfs-OST0004 -print0 /mnt/lustre > /tmp/affected_files

'''Step 2: Delete or restore affected files.'''

Files with data on the failed OST will return I/O errors. You must
delete them or restore from backup:

# Delete affected files:
cat /tmp/affected_files | xargs -0 -n 1 unlink

'''Step 3: Permanently deactivate the OST.'''

On the MGS:

lctl conf_param testfs-OST0004.osc.active=0

==== Scenario 3: Temporary OST Deactivation ====

To temporarily take an OST offline for maintenance:

# On each client (temporary, resets on remount):
lctl set_param osc.testfs-OST0004-*.active=0

# On MDS (prevents new allocations):
lctl set_param osp.testfs-OST0004-osc-MDT*.max_create_count=0

To bring it back:

# On MDS:
lctl set_param osp.testfs-OST0004-osc-MDT*.max_create_count=20000

# On each client:
lctl set_param osc.testfs-OST0004-*.active=1

==== Returning a Permanently Deactivated OST to Service ====

# On MGS:
lctl conf_param testfs-OST0004.osc.active=1

=== Removing an MDT ===

Removing an MDT is more complex than removing an OST because it
contains namespace data (directories, filenames, permissions).

==== If the MDT Is Available ====

You have two options, depending on whether you want to '''keep''' or
'''discard''' the data currently stored under the MDT you are removing.

'''Option A (recommended): Migrate directories off the MDT, then remove it.'''

This preserves your data. The directory inodes — and the files within
them — are moved to another MDT, leaving the target MDT empty so it can
be safely deactivated. This is the approach documented in the Lustre
Operations Manual for removing an MDT, and it avoids the data loss that
<code>rm -r</code> would cause.

'''Step 1: Find the directories whose inodes live on the MDT to be removed'''
(for example, MDT index 3):

# Show which MDT currently holds a given directory:
lfs getdirstripe --mdt-index /mnt/lustre/some_directory

'''Step 2: Migrate each such directory to a surviving MDT''' (for
example, MDT0000). Migration is recursive by default, moving the
directory together with all subdirectories and inodes beneath it:

lfs migrate -m 0 /mnt/lustre/some_directory

'''Step 3: Verify the directory and its files now report the new MDT index.'''

lfs getdirstripe --mdt-index /mnt/lustre/some_directory
lfs getstripe --mdt-index /mnt/lustre/some_directory/*

Repeat until no directories or inodes remain on the MDT being removed.

'''Migration caveats''':

* Only the '''root user''' can migrate directories.
* Migration operates on '''whole directories'''. Migrating or restriping ''striped'' directories requires Lustre 2.12 or later.
* Migration '''cannot be interrupted''' once started, although the directory and its files remain accessible during the operation.
* Each migrated file receives a '''new FID''' and therefore reports a new inode number to userspace. Tools that identify files by inode number (backup/archive tools, NFS, Samba) may treat migrated files as new. If the filesystem is re-exported over NFS, cached stale file handles can make files temporarily inaccessible; restart the NFS service (and possibly the clients) to clear them.
* '''Data-on-MDT (DoM)''' files cannot be migrated between MDTs directly. First migrate them to a non-DoM layout, then migrate the inodes.
* Files that are currently open or have been archived by HSM will fail to migrate; rerun the same command once they are available.
* The root directory always resides on '''MDT0000''', so MDT0000 itself cannot be emptied and removed by this procedure.

'''Step 4: Once the MDT is empty, deactivate it on the MGS.'''

lctl conf_param testfs-MDT0003.mdc.active=0

'''Option B: Delete the data on the MDT.'''

Use this '''only''' if you genuinely want to '''destroy''' everything
stored under that MDT. Unlike migration, <code>rm -r</code> permanently
deletes the directories and every file beneath them — it empties the
MDT by throwing the data away, not by relocating it:

rm -r /mnt/lustre/directory_on_mdt

Then deactivate the MDT on the MGS:

lctl conf_param testfs-MDT0003.mdc.active=0

==== If the MDT Is Permanently Inaccessible ====

Standard <code>rmdir</code> will fail with I/O error for directories
on an inaccessible MDT. Use <code>lfs rm_entry</code> instead:

lfs rm_entry /mnt/lustre/directory_on_failed_mdt

Then mark the MDT as permanently inactive:

lctl conf_param testfs-MDT0003.mdc.active=0

==== Check Which MDT Manages a Directory ====

lfs getstripe --mdt-index /mnt/lustre/some_directory

==== Warning ====

All files and directories below an inactive MDT become inaccessible
until the MDT is reactivated. Clients will receive <code>EIO</code>
errors when accessing them.

=== Important Warnings ===

* '''Do not use <code>lctl conf_param</code> to deactivate a working OST''' unless you intend it to be permanent. This immediately and permanently deactivates it on both the MDS and all clients.
* '''Migrate data before deactivating.''' Once an OST is deactivated on clients, files on it return I/O errors and cannot be migrated.
* '''Free space/object counts''' on a deactivated OST won't decrease when files are deleted. Object destruction is deferred until the OST reconnects to the MDS.
* A permanently deactivated OST still appears in filesystem configuration until <code>writeconf</code> is run or it is removed with <code>del_ost</code>.

=== See Also ===

* [[Lustre File System Expansion]] — Adding MDTs and OSTs
* [[Handling Full OSTs]] — Managing full OSTs
* [[Lustre Configuration Log Management]] — writeconf, replace_nids

Lustre OST MDT Removal - Revision history