Elliswilson: Created page with "== Common Lustre Mistakes and How to Avoid Them == This page collects common pitfalls encountered by Lustre administrators and users. These are real issues drawn from community experience, the Lustre Operations Manual, and the FAQ. Reading this page before putting a Lustre filesystem into production can save significant trouble. === Installation & Startup === ==== Mounting Servers in the Wrong Order ==== '''Mistake:''' Mounting OSTs or clients before the MGS/MDT is u..."

2026-05-11T14:53:06Z

Created page with "== Common Lustre Mistakes and How to Avoid Them == This page collects common pitfalls encountered by Lustre administrators and users. These are real issues drawn from community experience, the Lustre Operations Manual, and the FAQ. Reading this page before putting a Lustre filesystem into production can save significant trouble. === Installation & Startup === ==== Mounting Servers in the Wrong Order ==== '''Mistake:''' Mounting OSTs or clients before the MGS/MDT is u..."

New page

== Common Lustre Mistakes and How to Avoid Them ==

This page collects common pitfalls encountered by Lustre administrators
and users. These are real issues drawn from community experience, the
Lustre Operations Manual, and the FAQ. Reading this page before putting
a Lustre filesystem into production can save significant trouble.

=== Installation & Startup ===

==== Mounting Servers in the Wrong Order ====

'''Mistake:''' Mounting OSTs or clients before the MGS/MDT is up.

'''What happens:''' The mount hangs or fails because the server cannot
reach the MGS to register itself and retrieve configuration.

'''Fix:''' Always start in this order:
# MGT (Management Target)
# MDT(s) (Metadata Targets)
# OST(s) (Object Storage Targets)
# Clients

Shut down in '''reverse''' order: clients first, then OSTs, then
MDT, then MGT. Failure to unmount clients first causes client hangs
and evictions.

==== Port 988 Blocked by Firewall ====

'''Mistake:''' Forgetting to open TCP port 988 between Lustre nodes.

'''What happens:''' Mounts hang or time out with "Connection timed
out." LNet cannot establish connections.

'''Fix:''' Open port 988 on all Lustre nodes:

firewall-cmd --permanent --add-port=988/tcp
firewall-cmd --reload

==== Port 988 Blocked by Firewall ====

'''Mistake:''' Forgetting to open TCP port 988 between Lustre nodes.

'''What happens:''' Mounts hang or time out with "Connection timed
out." LNet cannot establish connections.

'''Fix:''' Open port 988 on all Lustre nodes:

firewall-cmd --permanent --add-port=988/tcp
firewall-cmd --reload

==== Not Using _netdev or noauto in fstab ====

'''Mistake:''' Adding Lustre mounts to <code>/etc/fstab</code> without
<code>_netdev</code> or <code>noauto</code>.

'''What happens:''' The system tries to mount Lustre before the
network is up during boot, causing a hang or boot failure.

'''Fix:''' Always include <code>_netdev</code> (and <code>noauto</code>
if using an HA manager like Pacemaker):

LABEL=testfs-MDT0000 /mnt/mdt lustre defaults,_netdev,noauto 0 0

For systemd-based systems, also add
<code>x-systemd.requires=lnet.service</code>.

==== Running a Lustre Client on a Server Node ====

'''Mistake:''' Mounting the Lustre filesystem on the same node that
runs an MDS or OSS.

'''What happens:''' Under memory pressure, the client and server
compete for memory, which can cause '''deadlocks'''. The server needs
to allocate memory to process the client's I/O, but the client is
waiting for the server, and the system is out of memory.

'''Fix:''' Use dedicated client nodes. Do not mount the filesystem
on MDS or OSS nodes.

==== Not Specifying --index at Format Time ====

'''Mistake:''' Running <code>mkfs.lustre</code> without
<code>--index=N</code>.

'''What happens:''' The target gets an unassigned index (label ends
in <code>FFFF</code>) until its first mount, making it harder to
identify and manage.

'''Fix:''' Always specify <code>--index=0</code>,
<code>--index=1</code>, etc. when formatting MDTs and OSTs.

=== Networking ===

==== Inconsistent UID/GID Across Nodes ====

'''Mistake:''' Different <code>/etc/passwd</code> or
<code>/etc/group</code> contents on client and server nodes.

'''What happens:''' Files appear owned by the wrong user. Operations
fail with <code>EIDRM</code> (error -43) when a UID/GID on the
client doesn't match any known identity on the MDS.

'''Fix:''' Use a centralized identity service (LDAP, FreeIPA, NIS) or
ensure <code>/etc/passwd</code> and <code>/etc/group</code> are
consistent across all nodes.

==== Assuming Multi-Rail Provides HA ====

'''Mistake:''' Configuring multiple LNet interfaces (Multi-Rail)
and assuming it provides network failover.

'''What happens:''' Multi-Rail aggregates bandwidth across multiple
interfaces, but if the underlying switch fails, all rails connected
to that switch go down simultaneously.

'''Fix:''' Multi-Rail is for bandwidth, not high availability. For
true network redundancy, ensure rails go through '''independent
network paths''' (different switches, different fabrics).

=== Storage & OST Management ===

==== Striping Across All OSTs With stripe_count=-1 ====

'''Mistake:''' Setting <code>lfs setstripe -c -1</code> (stripe
across every OST) as the default for a directory or filesystem.

'''What happens:''' Every file must allocate space on '''every''' OST.
When any single OST fills up, '''all''' new writes fail with
<code>ENOSPC</code> — even if the filesystem as a whole has plenty
of free space. The MDS cannot skip full OSTs when stripe_count equals
the total OST count.

'''Fix:''' Use a stripe count less than the total number of OSTs
(e.g., <code>-c 4</code> or <code>-c 8</code>). This lets the MDS
skip full OSTs and distribute new files to OSTs with free space. Use
<code>-c -1</code> only for specific large files that genuinely need
aggregate bandwidth across all OSTs.

==== Explicitly Setting the Starting OST Index ====

'''Mistake:''' Using <code>lfs setstripe -i 0</code> for every file
or directory.

'''What happens:''' All files start on OST 0, causing severe
imbalance. OST 0 fills up while other OSTs are nearly empty.

'''Fix:''' Let the MDS auto-select the starting OST (omit the
<code>-i</code> option). Lustre uses a weighted round-robin algorithm
that automatically balances across OSTs based on free space.

==== Not Planning for OST Growth ====

'''Mistake:''' Waiting until the filesystem is 95%+ full before adding
OSTs.

'''What happens:''' Rebalancing data across OSTs is slow and
disruptive. The MDS has no room to maneuver for load balancing. Users
experience write failures.

'''Fix:''' Plan to add OSTs before hitting 80% usage. Add at least
25% more capacity each time. You cannot shrink OSTs online — only
growth is supported.

==== Using writeconf Without Understanding the Consequences ====

'''Mistake:''' Running <code>tunefs.lustre --writeconf</code> without
realizing what it destroys.

'''What happens:''' <code>writeconf</code> regenerates the
configuration log from scratch. This '''erases all persistent
parameter settings''', including pool definitions, striping defaults,
quota settings, and NRS policies. After writeconf, the filesystem
reverts to factory defaults.

'''Fix:''' Only use <code>writeconf</code> when you genuinely need
to regenerate the configuration log (e.g., after changing server
NIDs). Document all persistent settings beforehand so you can
restore them. Consider using <code>replace_nids</code> instead if
you are only changing network addresses.

==== Not Backing Up the MDT ====

'''Mistake:''' Backing up file data (OSTs) but not the MDT.

'''What happens:''' If the MDT is lost, '''the entire filesystem is
lost''' — even if every OST is perfectly intact. The MDT contains
the namespace: all directory structures, filenames, permissions, and
file-to-object mappings. Without it, the data on OSTs is an
unrecoverable pile of objects.

'''Fix:''' The MDT is the single most critical component. Back it up
regularly. Device-level backups (LVM snapshot, ZFS send) are fastest.
See [[Backing Up a Lustre File System]].

=== Quota Management ===

==== Assuming Quotas Are Enforced by Default ====

'''Mistake:''' Setting quota limits with <code>lfs setquota</code>
and assuming they are enforced.

'''What happens:''' Quota '''accounting''' is enabled by default
on ldiskfs, but quota '''enforcement''' is '''not'''. Users can
exceed their limits freely.

'''Fix:''' Explicitly enable enforcement on all MDTs and OSTs:

lctl set_param osd-*.*.quota_slave_dt.enabled=ugp
lctl set_param osd-*.*.quota_slave_md.enabled=ugp

Use <code>set_param -P</code> to make it persistent.

==== Quota Granted Cache Overshoot ====

'''Mistake:''' Setting a hard quota limit (e.g., 400 GB) and
expecting it to be exact.

'''What happens:''' OSTs grant write cache to clients for
performance. Clients can write data that has been locally
acknowledged but not yet accounted against quota on the server.
Actual usage can '''significantly exceed''' the configured limit.

'''Fix:''' This is a known design trade-off, not a bug. To minimize
overshoot, reduce client dirty cache:

lctl set_param osc.*.max_dirty_mb=8

The overshoot cannot be eliminated entirely without destroying
write performance.

=== Client Behavior ===

==== Not Checking Write Return Codes ====

'''Mistake:''' Writing data to a Lustre file and not checking
the return value of <code>write()</code> or calling
<code>fsync()</code>.

'''What happens:''' If a client is '''evicted''' (due to network
timeout, server failover, or lock callback expiration), all dirty
pages in the client cache are '''silently dropped'''. This is
standard POSIX semantics — the write was buffered, not committed.
Without checking return codes, the application has no way to know
the data was lost.

'''Fix:''' Applications must check <code>write()</code> return
values AND call <code>fsync()</code> before considering data
committed. This is not Lustre-specific — it applies to any
filesystem with write-back caching.

==== Confusing -EROFS With Read-Only Mount ====

'''Mistake:''' Seeing error -30 (<code>EROFS</code>) and assuming
the filesystem was mounted read-only intentionally.

'''What happens:''' <code>EROFS</code> in Lustre means the backend
filesystem (ldiskfs or ZFS) detected a '''storage error''' and
remounted itself read-only to prevent further corruption. This is
an '''emergency''' condition.

'''Fix:''' Investigate storage hardware immediately. Check
<code>dmesg</code> for disk errors. A Lustre service restart is
required to return to read-write mode — there is no other way
to ensure cache consistency.

==== Expecting Fully Coherent atime ====

'''Mistake:''' Relying on <code>atime</code> (access time) being
updated in real time across all clients.

'''What happens:''' Lustre updates <code>atime</code> lazily for
performance. It is piggy-backed on other inode changes or updated
at file close. A read from cache may not update atime at all.

'''Fix:''' Don't rely on atime for cross-client synchronization.
If atime is not needed, mount with <code>noatime</code> for a
small performance improvement.

=== Configuration Management ===

==== Using lctl set_param Without -P ====

'''Mistake:''' Changing a tunable with <code>lctl set_param</code>
and forgetting to make it permanent.

'''What happens:''' The change takes effect immediately but is lost
on the next server restart. The filesystem reverts to its previous
behavior.

'''Fix:''' Use <code>lctl set_param -P</code> to persist the change
in the MGS configuration log:

lctl set_param -P osc.*.max_dirty_mb=64

==== New MDTs Don't Automatically Get Used ====

'''Mistake:''' Adding a new MDT and expecting existing directories
to automatically spread across it.

'''What happens:''' Nothing. Existing directories stay on their
original MDT. The new MDT sits empty unless explicitly used.

'''Fix:''' After adding a new MDT, create new top-level directories
on it:

lfs mkdir -i <mdt_index> /mnt/lustre/new_project

New files and subdirectories under <code>/mnt/lustre/new_project</code>
will be managed by the new MDT.

=== See Also ===

* [[Frequently Asked Questions]]
* [[Lustre Quota Troubleshooting]]
* [[Handling Full OSTs]]
* [[Lustre Debugging Procedures]]
* [[Lustre Getting Started]]

Lustre Common Mistakes - Revision history