Lustre Health Checks
--- Content (MediaWiki markup) ---
Overview
This page provides a quick-reference guide for Lustre administrators to verify that a filesystem is healthy. The checks below cover node health, storage capacity, recovery state, client connectivity, and LNet status.
For detailed statistics and monitoring, see the Lustre Monitoring and Statistics Guide.
Quick Health Check Summary
| Check | Command | Healthy Output | Unhealthy Output |
|---|---|---|---|
| Node health | lctl get_param health_check
|
healthy
|
NOT HEALTHY (with reason)
|
| Space usage | lfs df
|
All targets show reasonable usage | Any target at 100% or missing |
| Inode usage | lfs df -i
|
Inode usage well below limits | Inodes exhausted on any target |
| Recovery state | lctl get_param *.*.recovery_status
|
status: COMPLETE or INACTIVE
|
status: RECOVERING, clients not yet connected
|
| Evictions | dmesg | grep -i evict
|
No output | Messages like Evicted client ... (CONN_EXPIRED)
|
| OST exports | lctl get_param obdfilter.*.num_exports
|
Stable, expected client count | Drops suddenly or is zero |
| MDT exports | lctl get_param mdt.*.num_exports
|
Stable, expected client count | Drops suddenly or is zero |
| LNet networks | lnetctl net show
|
All NIDs listed, status up | Missing NIDs or errors |
| LNet peers | lnetctl peer show
|
All peers reachable, health value > 0 | Health value 0, state down |
Detailed Checks
Node Health
Every Lustre node (client, OSS, MDS) exposes a top-level health indicator:
lctl get_param health_check
A healthy node returns healthy. If the node is unhealthy, the output includes a reason such as NOT HEALTHY - LBUG or NOT HEALTHY - dead import. An unhealthy status typically means at least one OBD device has failed or LNet is down.
Space and Inode Usage
Check aggregate and per-target space usage:
lfs df /mnt/lustre
Check inode usage:
lfs df -i /mnt/lustre
Things to watch for:
- Any OST above 95% triggers space rebalancing concerns.
- An MDT running low on inodes can block new file creation.
- Uneven OST utilization may indicate striping or QoS weight issues.
Recovery Status
After a server restart or network interruption, Lustre enters a recovery window during which clients reconnect:
lctl get_param *.*.recovery_status
Key fields in the output:
- status —
COMPLETEmeans recovery finished.RECOVERINGmeans the server is still waiting for clients. - connected_clients / max_clients — how many clients have reconnected.
- time_remaining — seconds left in the recovery window.
If recovery does not complete, check whether evicted clients are still trying to reconnect or whether the recovery timeout (recovery_time_hard) needs to be extended.
Checking for Evictions
Evictions occur when a client fails to communicate with a server within the timeout window. Check server-side kernel logs:
dmesg | grep -i evict
Common eviction reasons:
- CONN_EXPIRED — client did not send a ping within the timeout.
- RECONNECT_RACE — conflicting reconnection attempts.
- ADMIN_FORCED — administrator manually evicted the client.
To see the server-side eviction count:
lctl get_param *.*.num_exports
A sudden drop indicates clients were evicted.
Target Export Count
The number of active client connections to each target:
# OSTs lctl get_param obdfilter.*.num_exports
# MDTs lctl get_param mdt.*.num_exports
Compare these values against the expected number of mounted clients. If a target shows zero exports and the filesystem is mounted, the target may be offline or in recovery.
LNet Health
Check that LNet networks are properly configured and operational:
lnetctl net show
Verify peer connectivity and health scores:
lnetctl peer show
Each peer has a health value (default max 1000). Values below 1000 indicate recent communication failures. A value of 0 means the peer is considered unreachable and LNet will avoid routing through it.
To check overall LNet status quickly:
lctl get_param nis
This shows local NIDs, their status, and traffic counters.
Key Proc Parameters
| Parameter | Description |
|---|---|
health_check
|
Top-level node health (healthy / NOT HEALTHY) |
*.*.recovery_status
|
Recovery state, connected clients, time remaining |
*.*.num_exports
|
Number of active client connections per target |
obdfilter.*.kbytestotal
|
Total capacity per OST (in KB) |
obdfilter.*.kbytesfree
|
Free space per OST (in KB) |
mdt.*.kbytestotal
|
Total capacity per MDT (in KB) |
mdt.*.filesfree
|
Free inodes per MDT |