Lustre Client Eviction Guide

What Is Client Eviction?

Client eviction occurs when a Lustre server (MDS or OSS) forcibly revokes all locks and state held by a client. The server discards any knowledge of the client's cached data, and the client's dirty (unwritten) page cache for that target is discarded. Any in-flight or buffered writes that have not been acknowledged by the server are lost.

Eviction is a last-resort mechanism. It ensures that server resources (locks, open file handles, recovery slots) are not held indefinitely by a client that may be dead or unresponsive.

Causes of Eviction

Lock Callback Timeout

When the server needs to revoke a lock (e.g., because another client needs conflicting access), it sends a blocking callback (AST) to the lock holder. If the client does not respond within the lock callback timeout, the server evicts the client.

The callback timeout is derived from the adaptive timeout mechanism or the static obd_timeout parameter.

Ping Timeout

Clients and servers exchange periodic ping messages. If the server does not receive a ping from a client within 1.5 × obd_timeout seconds, it evicts the client. With the default obd_timeout=100, this gives a ping deadline of 150 seconds.

Server Shutdown

When a server target is unmounted or the server node is shut down, all clients connected to that target are evicted. Clients detect this when they fail to reconnect and begin recovery.

Network Failure

Sustained network partitions between a client and a server lead to both ping timeouts and lock callback timeouts, resulting in eviction.

Administrative Eviction

Administrators can manually evict a specific client:

lctl --device $OSDNAME evict_client $CLIENT_NID

This is sometimes used to unstick a server waiting on a misbehaving client during recovery.

Detecting Eviction

Client-Side Detection

Check the kernel log (dmesg) on the client:

dmesg | grep -i evict

Typical messages:

Lustre: testfs-MDT0000-mdc-ffff8800deadbeef: server evicted this client; invalidating state
Lustre: testfs-OST0001-osc-ffff8800deadbeef: server evicted this client; invalidating state

Server-Side Detection

Check the recovery status on the server:

lctl get_param *.*.recovery_status

Look for evicted_clients counts and the list of connected vs. recovered clients.

Server-side kernel log also records evictions:

dmesg | grep -i evict

Impact of Eviction

Data Loss Risk

Buffered writes that have not been flushed to the server are lost. This is the most critical consequence. If an application writes data, receives a successful write() return code, but the data is still in the client page cache when eviction occurs, that data is silently discarded.

After eviction the client returns -EIO (Input/output error) on subsequent operations to the affected target.

The write() Return Code Problem

The POSIX write() system call succeeds as soon as data is copied into the kernel page cache — it does not wait for data to reach the server. This means:

• write() returns success even though data may be lost on eviction.
• Applications must call fsync() or fdatasync() and check its return code to confirm data durability.
• Applications must check close() return codes, as close() flushes remaining dirty pages.

This is not specific to Lustre — it is POSIX behavior — but Lustre's distributed nature makes eviction-induced data loss more likely than on local file systems.

Application Impact

• Applications receive -EIO on the next operation after eviction.
• Open file descriptors become invalid for the evicted target.
• NFS re-exports over Lustre may return stale file handle errors.
• MPI jobs typically abort when any rank receives an I/O error.

Prevention

Tune obd_timeout

Increase obd_timeout to give clients more time to respond. The default is 100 seconds. Large-scale systems or sites with known network variability often increase this:

On servers and clients:

lctl set_param timeout=300

Note: This was historically called obd_timeout. Larger values increase recovery time after genuine failures, so choose a value that balances tolerance and recovery speed.

Enable Adaptive Timeouts

Adaptive timeouts (enabled by default since Lustre 2.x) dynamically adjust timeouts based on observed RPC latency. Verify they are enabled:

lctl get_param at_max
lctl get_param at_min

If at_max=0, adaptive timeouts are disabled and the static obd_timeout is used. Ensure at_max is set to a reasonable value (default is 600).

Network Reliability

• Monitor network health between clients and servers.
• Ensure LNet routes are redundant where possible (see Multi-Rail LNet).
• Avoid oversubscribing network switches serving Lustre traffic.
• Use dedicated Lustre networks to avoid contention with other traffic.

Client Health

A client under heavy memory pressure or CPU starvation may fail to respond to lock callbacks in time. Monitor client nodes for:

• High memory usage and swapping (prevents timely RPC processing).
• CPU saturation (kernel threads cannot run).
• Long interrupt storms or driver issues on the network interface.

Remediation After Eviction

1. Check application status: Determine whether the application detected the I/O error. Check application logs and exit codes.
2. Check data integrity: If the application did not fsync() before eviction, recently written data may be lost. Verify output files.
3. Remount the client: In many cases the client automatically reconnects. If errors persist, unmount and remount:

   umount /mnt/lustre && mount -t lustre $MGSNID:/$FSNAME /mnt/lustre

4. Investigate root cause: Determine why the eviction happened:
   • Network outage? Check switch logs, LNet status (lctl get_param nis).
   • Overloaded client? Check memory and CPU metrics at the time of eviction.
   • Server issue? Check server logs and recovery status.
   • Slow copytool or HSM activity blocking locks?
5. Review timeout tuning: If evictions are recurring, consider adjusting obd_timeout and adaptive timeout parameters.

See Also

• Lustre Common Mistakes — common pitfalls including write handling
• Lustre Recovery Overview — how Lustre recovers from failures
• Lustre Resiliency Understanding Lustre Message Loss and Tuning for Resiliency
• Lustre Timeout Hierarchy