Elliswilson: Created page with "--- Content (MediaWiki markup) --- TOC == Overview == This page provides a quick-reference guide for Lustre administrators to verify that a filesystem is healthy. The checks below cover node health, storage capacity, recovery state, client connectivity, and LNet status. For detailed statistics and monitoring, see the Lustre Monitoring and Statistics Guide. == Quick Health Check Summary == {| class="wikitable" ! Check ! Command ! Healthy Output ! Unhealthy O..."

2026-04-30T16:24:10Z

Created page with "--- Content (MediaWiki markup) --- __TOC__ == Overview == This page provides a quick-reference guide for Lustre administrators to verify that a filesystem is healthy. The checks below cover node health, storage capacity, recovery state, client connectivity, and LNet status. For detailed statistics and monitoring, see the Lustre Monitoring and Statistics Guide. == Quick Health Check Summary == {| class="wikitable" ! Check ! Command ! Healthy Output ! Unhealthy O..."

New page

--- Content (MediaWiki markup) ---

__TOC__

== Overview ==

This page provides a quick-reference guide for Lustre administrators to verify that a filesystem is healthy. The checks below cover node health, storage capacity, recovery state, client connectivity, and LNet status.

For detailed statistics and monitoring, see the [[Lustre Monitoring and Statistics Guide]].

== Quick Health Check Summary ==

{| class="wikitable"
! Check
! Command
! Healthy Output
! Unhealthy Output
|-
| Node health
| <code>lctl get_param health_check</code>
| <code>healthy</code>
| <code>NOT HEALTHY</code> (with reason)
|-
| Space usage
| <code>lfs df</code>
| All targets show reasonable usage
| Any target at 100% or missing
|-
| Inode usage
| <code>lfs df -i</code>
| Inode usage well below limits
| Inodes exhausted on any target
|-
| Recovery state
| <code>lctl get_param *.*.recovery_status</code>
| <code>status: COMPLETE</code> or <code>INACTIVE</code>
| <code>status: RECOVERING</code>, clients not yet connected
|-
| Evictions
| <code>dmesg | grep -i evict</code>
| No output
| Messages like <code>Evicted client ... (CONN_EXPIRED)</code>
|-
| OST exports
| <code>lctl get_param obdfilter.*.num_exports</code>
| Stable, expected client count
| Drops suddenly or is zero
|-
| MDT exports
| <code>lctl get_param mdt.*.num_exports</code>
| Stable, expected client count
| Drops suddenly or is zero
|-
| LNet networks
| <code>lnetctl net show</code>
| All NIDs listed, status up
| Missing NIDs or errors
|-
| LNet peers
| <code>lnetctl peer show</code>
| All peers reachable, health value > 0
| Health value 0, state down
|}

== Detailed Checks ==

=== Node Health ===

Every Lustre node (client, OSS, MDS) exposes a top-level health indicator:

lctl get_param health_check

A healthy node returns <code>healthy</code>. If the node is unhealthy, the output includes a reason such as <code>NOT HEALTHY - LBUG</code> or <code>NOT HEALTHY - dead import</code>. An unhealthy status typically means at least one OBD device has failed or LNet is down.

=== Space and Inode Usage ===

Check aggregate and per-target space usage:

lfs df /mnt/lustre

Check inode usage:

lfs df -i /mnt/lustre

Things to watch for:
* Any OST above 95% triggers space rebalancing concerns.
* An MDT running low on inodes can block new file creation.
* Uneven OST utilization may indicate striping or QoS weight issues.

=== Recovery Status ===

After a server restart or network interruption, Lustre enters a recovery window during which clients reconnect:

lctl get_param *.*.recovery_status

Key fields in the output:
* '''status''' — <code>COMPLETE</code> means recovery finished. <code>RECOVERING</code> means the server is still waiting for clients.
* '''connected_clients''' / '''max_clients''' — how many clients have reconnected.
* '''time_remaining''' — seconds left in the recovery window.

If recovery does not complete, check whether evicted clients are still trying to reconnect or whether the recovery timeout (''recovery_time_hard'') needs to be extended.

=== Checking for Evictions ===

Evictions occur when a client fails to communicate with a server within the timeout window. Check server-side kernel logs:

dmesg | grep -i evict

Common eviction reasons:
* '''CONN_EXPIRED''' — client did not send a ping within the timeout.
* '''RECONNECT_RACE''' — conflicting reconnection attempts.
* '''ADMIN_FORCED''' — administrator manually evicted the client.

To see the server-side eviction count:

lctl get_param *.*.num_exports

A sudden drop indicates clients were evicted.

=== Target Export Count ===

The number of active client connections to each target:

# OSTs
lctl get_param obdfilter.*.num_exports

# MDTs
lctl get_param mdt.*.num_exports

Compare these values against the expected number of mounted clients. If a target shows zero exports and the filesystem is mounted, the target may be offline or in recovery.

=== LNet Health ===

Check that LNet networks are properly configured and operational:

lnetctl net show

Verify peer connectivity and health scores:

lnetctl peer show

Each peer has a '''health value''' (default max 1000). Values below 1000 indicate recent communication failures. A value of 0 means the peer is considered unreachable and LNet will avoid routing through it.

To check overall LNet status quickly:

lctl get_param nis

This shows local NIDs, their status, and traffic counters.

== Key Proc Parameters ==

{| class="wikitable"
! Parameter
! Description
|-
| <code>health_check</code>
| Top-level node health (healthy / NOT HEALTHY)
|-
| <code>*.*.recovery_status</code>
| Recovery state, connected clients, time remaining
|-
| <code>*.*.num_exports</code>
| Number of active client connections per target
|-
| <code>obdfilter.*.kbytestotal</code>
| Total capacity per OST (in KB)
|-
| <code>obdfilter.*.kbytesfree</code>
| Free space per OST (in KB)
|-
| <code>mdt.*.kbytestotal</code>
| Total capacity per MDT (in KB)
|-
| <code>mdt.*.filesfree</code>
| Free inodes per MDT
|}

== See Also ==

* [[Lustre Monitoring and Statistics Guide]]
* [[Lustre Debugging Procedures]]
* [[Understanding Lustre Evictions]]
* [[Lustre Timeout Hierarchy]]

[[Category:Administration]]
[[Category:Monitoring]]

Lustre Health Checks - Revision history