Lustre Upgrade Checklist

From Lustre Wiki
Revision as of 08:25, 20 July 2026 by Elliswilson (talk | contribs) (Created page with "== Purpose == This page is a '''condensed, actionable checklist''' for Lustre upgrades. It does '''not''' replace the full upgrade procedure — see Upgrading to a New Version of Lustre for the authoritative step-by-step guide. Use this page as a quick reference before, during, and after an upgrade. == Pre-Upgrade Preparation == Complete '''all''' of the following before beginning any upgrade: # '''Back up every MDT''' — <code>e2image -ra /dev/mdtdev /backup/md...")
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

Purpose

This page is a condensed, actionable checklist for Lustre upgrades. It does not replace the full upgrade procedure — see Upgrading to a New Version of Lustre for the authoritative step-by-step guide. Use this page as a quick reference before, during, and after an upgrade.

Pre-Upgrade Preparation

Complete all of the following before beginning any upgrade:

  1. Back up every MDT — e2image -ra /dev/mdtdev /backup/mdt0.img (ldiskfs) or ZFS snapshot. MDT data loss is unrecoverable.
  2. Document current settings — save output of lctl get_param -R '*' and lctl get_param osc.*.active on every server.
  3. Record pool definitions — lctl pool_list fsname for every pool. Pools are destroyed by --writeconf.
  4. Read the Release Notes for the target version. Note any required e2fsprogs or kernel version changes.
  5. Check interoperability — verify that client and server versions are within the supported interop matrix (see Release Notes).
  6. Test the upgrade on a non-production system or VM first.
  7. Notify users — schedule a maintenance window and unmount all clients.
  8. Verify backups — confirm MDT backups are readable: e2image -ra -I /backup/mdt0.img /dev/null.

Major Upgrade Checklist

A major upgrade changes the Lustre feature release (e.g., 2.14 → 2.16). This typically requires downtime.

  1. ☐ Unmount all clients.
  2. ☐ Stop all copytools and HSM coordinators.
  3. ☐ Unmount all OSTs, then MDTs, then the MGS (reverse mount order).
  4. ☐ Install new Lustre packages on all servers (MGS, MDS, OSS).
  5. ☐ If Release Notes require it, run tunefs.lustre --writeconf on the MGS device. Warning: this erases all conf_param settings and pool definitions. Re-apply them afterward.
  6. ☐ Mount the MGS first.
  7. ☐ Re-apply any conf_param settings that were cleared by --writeconf (use the backup from pre-upgrade step 2).
  8. ☐ Re-create OST pools if they were destroyed (use the backup from pre-upgrade step 3).
  9. ☐ Mount MDTs in index order (MDT0000, MDT0001, …).
  10. ☐ Mount all OSTs.
  11. ☐ Install new Lustre client packages on all clients.
  12. ☐ Mount clients.
  13. ☐ Run post-upgrade verification (see below).

Minor / Rolling Upgrade Checklist

A minor upgrade stays within the same feature release (e.g., 2.14.0 → 2.14.2). Rolling upgrades are often possible — check the Release Notes.

  1. ☐ Upgrade and restart one OSS at a time. Clients will reconnect automatically.
  2. ☐ After all OSSes are upgraded, upgrade the MDS (one at a time for DNE).
  3. ☐ Upgrade the MGS last (if separate from MDS).
  4. ☐ Upgrade clients in batches. Unmount, install new packages, remount.
  5. ☐ --writeconf is typically not needed for minor upgrades. Confirm in Release Notes.
  6. ☐ Run post-upgrade verification.

Post-Upgrade Verification

Run these checks on every server and a sample of clients: 

# On any client — file system health
lfs df
lctl get_param version
lctl get_param health_check        # must return "healthy"

# On each server — target status
lctl get_param obdfilter.*.recovery_status   # OSTs
lctl get_param mdt.*.recovery_status         # MDTs

# Functional smoke test
dd if=/dev/zero of=/mnt/lustre/upgrade_test bs=1M count=10
md5sum /mnt/lustre/upgrade_test
rm /mnt/lustre/upgrade_test

Optional Feature Enablement

Some features must be explicitly enabled after upgrading. They may be irreversible — read the notes carefully.

Feature Enable Command Reversible? Notes
Project quotas tune2fs -O project /dev/mdtdev then tunefs.lustre --enable-project /dev/mdtdev No Cannot be disabled once enabled. Requires ldiskfs reformat to remove.
Large xattr / ea_inode tune2fs -O ea_inode /dev/ostdev No Needed for >255 stripes or large ACLs. Older clients cannot read ea_inode data.
index_in_idif tunefs.lustre --param ost.index_in_idif=1 /dev/ostdev No Embeds OST index in FIDs. Required to support >65535 OSTs.
DNE (remote dirs) lfs mkdir -i 1 /mnt/lustre/remote_dir Per-directory Striped/remote directories require all MDTs to be at the new version.
Overstriping lfs setstripe -C count /mnt/lustre/dir Per-file Allows stripe count > OST count. Available since 2.14.

Common Gotchas

  • --writeconf destroys OST pools and conf_param settings. Always back up pool definitions and parameter settings before running it. Re-apply them immediately after mounting the MGS.
  • Project quota enablement is irreversible. Once enabled on an MDT or OST, it cannot be disabled without reformatting the target.
  • Mount order matters after upgrade. Always mount MGS → MDTs (in index order) → OSTs. Mounting out of order can cause configuration replay errors.
  • Client/server version skew. Lustre supports N−1 interop (e.g., 2.14 clients with 2.16 servers). Wider gaps are unsupported and may cause silent data corruption.
  • SELinux contexts. New Lustre packages may reset SELinux file contexts. Run restorecon -Rv /usr/lib64/lustre/ if services fail to start with permission errors.
  • LNET configuration. New LNET module parameters or DLC settings may not carry forward. Verify /etc/lnet.conf is correct before starting LNET after upgrade.

See Also

Retrieved from "http://wiki.lustre.org/index.php?title=Lustre_Upgrade_Checklist&oldid=5579"