Lustre Common Mistakes
Common Lustre Mistakes and How to Avoid Them
This page collects common pitfalls encountered by Lustre administrators and users. These are real issues drawn from community experience, the Lustre Operations Manual, and the FAQ. Reading this page before putting a Lustre filesystem into production can save significant trouble.
Installation & Startup
Mounting Servers in the Wrong Order
Mistake: Mounting OSTs or clients before the MGS/MDT is up.
What happens: The mount hangs or fails because the server cannot reach the MGS to register itself and retrieve configuration.
Fix: Always start in this order:
- MGT (Management Target)
- MDT(s) (Metadata Targets)
- OST(s) (Object Storage Targets)
- Clients
Shut down in reverse order: clients first, then OSTs, then MDT, then MGT. Failure to unmount clients first causes client hangs and evictions.
Port 988 Blocked by Firewall
Mistake: Forgetting to open TCP port 988 between Lustre nodes.
What happens: Mounts hang or time out with "Connection timed out." LNet cannot establish connections.
Fix: Open port 988 on all Lustre nodes:
firewall-cmd --permanent --add-port=988/tcp firewall-cmd --reload
Port 988 Blocked by Firewall
Mistake: Forgetting to open TCP port 988 between Lustre nodes.
What happens: Mounts hang or time out with "Connection timed out." LNet cannot establish connections.
Fix: Open port 988 on all Lustre nodes:
firewall-cmd --permanent --add-port=988/tcp firewall-cmd --reload
Not Using _netdev or noauto in fstab
Mistake: Adding Lustre mounts to /etc/fstab without
_netdev or noauto.
What happens: The system tries to mount Lustre before the network is up during boot, causing a hang or boot failure.
Fix: Always include _netdev (and noauto
if using an HA manager like Pacemaker):
LABEL=testfs-MDT0000 /mnt/mdt lustre defaults,_netdev,noauto 0 0
For systemd-based systems, also add
x-systemd.requires=lnet.service.
Running a Lustre Client on a Server Node
Mistake: Mounting the Lustre filesystem on the same node that runs an MDS or OSS.
What happens: Under memory pressure, the client and server compete for memory, which can cause deadlocks. The server needs to allocate memory to process the client's I/O, but the client is waiting for the server, and the system is out of memory.
Fix: Use dedicated client nodes. Do not mount the filesystem on MDS or OSS nodes.
Not Specifying --index at Format Time
Mistake: Running mkfs.lustre without
--index=N.
What happens: The target gets an unassigned index (label ends
in FFFF) until its first mount, making it harder to
identify and manage.
Fix: Always specify --index=0,
--index=1, etc. when formatting MDTs and OSTs.
Networking
Inconsistent UID/GID Across Nodes
Mistake: Different /etc/passwd or
/etc/group contents on client and server nodes.
What happens: Files appear owned by the wrong user. Operations
fail with EIDRM (error -43) when a UID/GID on the
client doesn't match any known identity on the MDS.
Fix: Use a centralized identity service (LDAP, FreeIPA, NIS) or
ensure /etc/passwd and /etc/group are
consistent across all nodes.
Assuming Multi-Rail Provides HA
Mistake: Configuring multiple LNet interfaces (Multi-Rail) and assuming it provides network failover.
What happens: Multi-Rail aggregates bandwidth across multiple interfaces, but if the underlying switch fails, all rails connected to that switch go down simultaneously.
Fix: Multi-Rail is for bandwidth, not high availability. For true network redundancy, ensure rails go through independent network paths (different switches, different fabrics).
Storage & OST Management
Striping Across All OSTs With stripe_count=-1
Mistake: Setting lfs setstripe -c -1 (stripe
across every OST) as the default for a directory or filesystem.
What happens: Every file must allocate space on every OST.
When any single OST fills up, all new writes fail with
ENOSPC — even if the filesystem as a whole has plenty
of free space. The MDS cannot skip full OSTs when stripe_count equals
the total OST count.
Fix: Use a stripe count less than the total number of OSTs
(e.g., -c 4 or -c 8). This lets the MDS
skip full OSTs and distribute new files to OSTs with free space. Use
-c -1 only for specific large files that genuinely need
aggregate bandwidth across all OSTs.
Explicitly Setting the Starting OST Index
Mistake: Using lfs setstripe -i 0 for every file
or directory.
What happens: All files start on OST 0, causing severe imbalance. OST 0 fills up while other OSTs are nearly empty.
Fix: Let the MDS auto-select the starting OST (omit the
-i option). Lustre uses a weighted round-robin algorithm
that automatically balances across OSTs based on free space.
Not Planning for OST Growth
Mistake: Waiting until the filesystem is 95%+ full before adding OSTs.
What happens: Rebalancing data across OSTs is slow and disruptive. The MDS has no room to maneuver for load balancing. Users experience write failures.
Fix: Plan to add OSTs before hitting 80% usage. Add at least 25% more capacity each time. You cannot shrink OSTs online — only growth is supported.
Using writeconf Without Understanding the Consequences
Mistake: Running tunefs.lustre --writeconf without
realizing what it destroys.
What happens: writeconf regenerates the
configuration log from scratch. This erases all persistent
parameter settings, including pool definitions, striping defaults,
quota settings, and NRS policies. After writeconf, the filesystem
reverts to factory defaults.
Fix: Only use writeconf when you genuinely need
to regenerate the configuration log (e.g., after changing server
NIDs). Document all persistent settings beforehand so you can
restore them. Consider using replace_nids instead if
you are only changing network addresses.
Not Backing Up the MDT
Mistake: Backing up file data (OSTs) but not the MDT.
What happens: If the MDT is lost, the entire filesystem is lost — even if every OST is perfectly intact. The MDT contains the namespace: all directory structures, filenames, permissions, and file-to-object mappings. Without it, the data on OSTs is an unrecoverable pile of objects.
Fix: The MDT is the single most critical component. Back it up regularly. Device-level backups (LVM snapshot, ZFS send) are fastest. See Backing Up a Lustre File System.
Quota Management
Assuming Quotas Are Enforced by Default
Mistake: Setting quota limits with lfs setquota
and assuming they are enforced.
What happens: Quota accounting is enabled by default on ldiskfs, but quota enforcement is not. Users can exceed their limits freely.
Fix: Explicitly enable enforcement on all MDTs and OSTs:
lctl set_param osd-*.*.quota_slave_dt.enabled=ugp lctl set_param osd-*.*.quota_slave_md.enabled=ugp
Use set_param -P to make it persistent.
Quota Granted Cache Overshoot
Mistake: Setting a hard quota limit (e.g., 400 GB) and expecting it to be exact.
What happens: OSTs grant write cache to clients for performance. Clients can write data that has been locally acknowledged but not yet accounted against quota on the server. Actual usage can significantly exceed the configured limit.
Fix: This is a known design trade-off, not a bug. To minimize overshoot, reduce client dirty cache:
lctl set_param osc.*.max_dirty_mb=8
The overshoot cannot be eliminated entirely without destroying write performance.
Client Behavior
Not Checking Write Return Codes
Mistake: Writing data to a Lustre file and not checking
the return value of write() or calling
fsync().
What happens: If a client is evicted (due to network timeout, server failover, or lock callback expiration), all dirty pages in the client cache are silently dropped. This is standard POSIX semantics — the write was buffered, not committed. Without checking return codes, the application has no way to know the data was lost.
Fix: Applications must check write() return
values AND call fsync() before considering data
committed. This is not Lustre-specific — it applies to any
filesystem with write-back caching.
Confusing -EROFS With Read-Only Mount
Mistake: Seeing error -30 (EROFS) and assuming
the filesystem was mounted read-only intentionally.
What happens: EROFS in Lustre means the backend
filesystem (ldiskfs or ZFS) detected a storage error and
remounted itself read-only to prevent further corruption. This is
an emergency condition.
Fix: Investigate storage hardware immediately. Check
dmesg for disk errors. A Lustre service restart is
required to return to read-write mode — there is no other way
to ensure cache consistency.
Expecting Fully Coherent atime
Mistake: Relying on atime (access time) being
updated in real time across all clients.
What happens: Lustre updates atime lazily for
performance. It is piggy-backed on other inode changes or updated
at file close. A read from cache may not update atime at all.
Fix: Don't rely on atime for cross-client synchronization.
If atime is not needed, mount with noatime for a
small performance improvement.
Configuration Management
Using lctl set_param Without -P
Mistake: Changing a tunable with lctl set_param
and forgetting to make it permanent.
What happens: The change takes effect immediately but is lost on the next server restart. The filesystem reverts to its previous behavior.
Fix: Use lctl set_param -P to persist the change
in the MGS configuration log:
lctl set_param -P osc.*.max_dirty_mb=64
New MDTs Don't Automatically Get Used
Mistake: Adding a new MDT and expecting existing directories to automatically spread across it.
What happens: Nothing. Existing directories stay on their original MDT. The new MDT sits empty unless explicitly used.
Fix: After adding a new MDT, create new top-level directories on it:
lfs mkdir -i <mdt_index> /mnt/lustre/new_project
New files and subdirectories under /mnt/lustre/new_project
will be managed by the new MDT.