Elliswilson: Created page with "== Purpose == This page is a '''condensed, actionable checklist''' for Lustre upgrades. It does '''not''' replace the full upgrade procedure — see Upgrading to a New Version of Lustre for the authoritative step-by-step guide. Use this page as a quick reference before, during, and after an upgrade. == Pre-Upgrade Preparation == Complete '''all''' of the following before beginning any upgrade: # '''Back up every MDT''' — `e2image -ra /dev/mdtdev /backup/md..."`


2026-07-20T15:25:37Z
Created page with "== Purpose ==  This page is a '''condensed, actionable checklist''' for Lustre upgrades. It does '''not''' replace the full upgrade procedure — see Upgrading to a New Version of Lustre for the authoritative step-by-step guide. Use this page as a quick reference before, during, and after an upgrade.  == Pre-Upgrade Preparation ==  Complete '''all''' of the following before beginning any upgrade:  # '''Back up every MDT''' — <code>e2image -ra /dev/mdtdev /backup/md..."
New page
== Purpose ==



This page is a '''condensed, actionable checklist''' for Lustre upgrades. It does '''not''' replace the full upgrade procedure — see [[Upgrading to a New Version of Lustre]] for the authoritative step-by-step guide. Use this page as a quick reference before, during, and after an upgrade.



== Pre-Upgrade Preparation ==



Complete '''all''' of the following before beginning any upgrade:



# '''Back up every MDT''' — <code>e2image -ra /dev/mdtdev /backup/mdt0.img</code> (ldiskfs) or ZFS snapshot. MDT data loss is unrecoverable.

# '''Document current settings''' — save output of <code>lctl get_param -R '*'</code> and <code>lctl get_param osc.*.active</code> on every server.

# '''Record pool definitions''' — <code>lctl pool_list fsname</code> for every pool. Pools are destroyed by <code>--writeconf</code>.

# '''Read the Release Notes''' for the target version. Note any required e2fsprogs or kernel version changes.

# '''Check interoperability''' — verify that client and server versions are within the supported interop matrix (see Release Notes).

# '''Test the upgrade''' on a non-production system or VM first.

# '''Notify users''' — schedule a maintenance window and unmount all clients.

# '''Verify backups''' — confirm MDT backups are readable: <code>e2image -ra -I /backup/mdt0.img /dev/null</code>.



== Major Upgrade Checklist ==



A ''major'' upgrade changes the Lustre feature release (e.g., 2.14 → 2.16). This typically requires downtime.



# ☐ Unmount all clients.

# ☐ Stop all copytools and HSM coordinators.

# ☐ Unmount all OSTs, then MDTs, then the MGS (reverse mount order).

# ☐ Install new Lustre packages on '''all''' servers (MGS, MDS, OSS).

# ☐ If Release Notes require it, run <code>tunefs.lustre --writeconf</code> on the MGS device. '''Warning:''' this erases all <code>conf_param</code> settings and pool definitions. Re-apply them afterward.

# ☐ Mount the MGS first.

# ☐ Re-apply any <code>conf_param</code> settings that were cleared by <code>--writeconf</code> (use the backup from pre-upgrade step 2).

# ☐ Re-create OST pools if they were destroyed (use the backup from pre-upgrade step 3).

# ☐ Mount MDTs in index order (MDT0000, MDT0001, …).

# ☐ Mount all OSTs.

# ☐ Install new Lustre client packages on all clients.

# ☐ Mount clients.

# ☐ Run post-upgrade verification (see below).



== Minor / Rolling Upgrade Checklist ==



A ''minor'' upgrade stays within the same feature release (e.g., 2.14.0 → 2.14.2). Rolling upgrades are often possible — check the Release Notes.



# ☐ Upgrade and restart one OSS at a time. Clients will reconnect automatically.

# ☐ After all OSSes are upgraded, upgrade the MDS (one at a time for DNE).

# ☐ Upgrade the MGS last (if separate from MDS).

# ☐ Upgrade clients in batches. Unmount, install new packages, remount.

# ☐ <code>--writeconf</code> is typically '''not''' needed for minor upgrades. Confirm in Release Notes.

# ☐ Run post-upgrade verification.



== Post-Upgrade Verification ==



Run these checks on every server and a sample of clients:



 # On any client — file system health

 lfs df

 lctl get_param version

 lctl get_param health_check        # must return "healthy"



 # On each server — target status

 lctl get_param obdfilter.*.recovery_status   # OSTs

 lctl get_param mdt.*.recovery_status         # MDTs



 # Functional smoke test

 dd if=/dev/zero of=/mnt/lustre/upgrade_test bs=1M count=10

 md5sum /mnt/lustre/upgrade_test

 rm /mnt/lustre/upgrade_test



== Optional Feature Enablement ==



Some features must be explicitly enabled after upgrading. They may be '''irreversible''' — read the notes carefully.



{| class="wikitable"

! Feature !! Enable Command !! Reversible? !! Notes

|-

| Project quotas || <code>tune2fs -O project /dev/mdtdev</code> then <code>tunefs.lustre --enable-project /dev/mdtdev</code> || '''No''' || Cannot be disabled once enabled. Requires ldiskfs reformat to remove.

|-

| Large xattr / ea_inode || <code>tune2fs -O ea_inode /dev/ostdev</code> || '''No''' || Needed for >255 stripes or large ACLs. Older clients cannot read ea_inode data.

|-

| index_in_idif || <code>tunefs.lustre --param ost.index_in_idif=1 /dev/ostdev</code> || '''No''' || Embeds OST index in FIDs. Required to support >65535 OSTs.

|-

| DNE (remote dirs) || <code>lfs mkdir -i 1 /mnt/lustre/remote_dir</code> || Per-directory || Striped/remote directories require all MDTs to be at the new version.

|-

| Overstriping || <code>lfs setstripe -C ''count'' /mnt/lustre/dir</code> || Per-file || Allows stripe count > OST count. Available since 2.14.

|}



== Common Gotchas ==



* '''<code>--writeconf</code> destroys OST pools and <code>conf_param</code> settings.''' Always back up pool definitions and parameter settings before running it. Re-apply them immediately after mounting the MGS.

* '''Project quota enablement is irreversible.''' Once enabled on an MDT or OST, it cannot be disabled without reformatting the target.

* '''Mount order matters after upgrade.''' Always mount MGS → MDTs (in index order) → OSTs. Mounting out of order can cause configuration replay errors.

* '''Client/server version skew.''' Lustre supports N−1 interop (e.g., 2.14 clients with 2.16 servers). Wider gaps are unsupported and may cause silent data corruption.

* '''SELinux contexts.''' New Lustre packages may reset SELinux file contexts. Run <code>restorecon -Rv /usr/lib64/lustre/</code> if services fail to start with permission errors.

* '''LNET configuration.''' New LNET module parameters or DLC settings may not carry forward. Verify <code>/etc/lnet.conf</code> is correct before starting LNET after upgrade.



== See Also ==



* [[Upgrading to a New Version of Lustre]] — Full step-by-step upgrade procedure

* [[Lustre Common Mistakes]] — Frequently encountered pitfalls



[[Category:Administration]]

[[Category:Upgrades]]

[[Category:Checklists]]
Lustre Upgrade Checklist - Revision history
