Lustre Day to Day Operations

From Lustre Wiki
Jump to navigation Jump to search

Quick Reference

Task Command
Check filesystem health lctl get_param health_check
List filesystem space lfs df -h
List MDT/OST space lfs df -i (inodes), lfs df -h (blocks)
Check server status lctl dl
Mount a client mount -t lustre mgsnode@tcp:/fsname /mnt/fsname
Unmount a client umount /mnt/fsname
Start a target mount -t lustre /dev/sdX /lustre/fsname/mdt0
Stop a target umount /lustre/fsname/mdt0

Startup Sequence

Always start services in this order: 

1. MGT  (Management Target)
2. MDT  (Metadata Targets — MDT0 first)
3. OSTs (Object Storage Targets)
4. Clients

# 1. Start MGS
mount -t lustre /dev/sda1 /lustre/mgt

# 2. Start MDT(s) — MDT0 must be first
mount -t lustre /dev/sdb1 /lustre/testfs/mdt0

# 3. Start OST(s)
mount -t lustre /dev/sdc1 /lustre/testfs/ost0
mount -t lustre /dev/sdd1 /lustre/testfs/ost1

# 4. Mount clients
mount -t lustre mgsnode@tcp:/testfs /mnt/testfs

For ZFS backends, import the zpool before mounting: 

zpool import <poolname>
mount -t lustre <poolname>/mdt0 /lustre/testfs/mdt0

Note: Clients connect to the MGS first, not the MDS. If the MDTs and OSTs are running but the MGS is down, clients cannot mount.

Shutdown Sequence

Reverse of startup: 

1. Clients   (umount all clients first)
2. MDT       (stops new metadata operations)
3. OSTs
4. MGT       (optional — can remain running for other filesystems)

# 1. Unmount clients
umount /mnt/testfs

# 2. Stop MDT(s)
umount /lustre/testfs/mdt0

# 3. Stop OST(s)
umount /lustre/testfs/ost0
umount /lustre/testfs/ost1

# 4. Stop MGS (optional)
umount /lustre/mgt

For ZFS, it is not necessary to export zpools after unmounting. Only export if migrating the pool to another host.

Mounting by Label

Lustre targets can be mounted by label instead of device path. Labels are set during mkfs.lustre: 

# Mount by label
mount -t lustre -L testfs-MDT0000 /lustre/testfs/mdt0
mount -t lustre -L testfs-OST0000 /lustre/testfs/ost0

fstab Entries

Server fstab entries should use _netdev to ensure networking is up before mount, and noauto to prevent mounting before LNet is configured: 

# Server targets
/dev/sdb1  /lustre/testfs/mdt0  lustre  _netdev,noauto  0 0
/dev/sdc1  /lustre/testfs/ost0  lustre  _netdev,noauto  0 0

# Client
mgsnode@tcp:/testfs  /mnt/testfs  lustre  _netdev,noauto  0 0

Alternatively, use labels: 

LABEL=testfs-MDT0000  /lustre/testfs/mdt0  lustre  _netdev,noauto  0 0

Checking Filesystem Status

Health Check

# On any server or client
lctl get_param health_check

Returns healthy or lists active problems.

Filesystem Space

# Block usage (human-readable)
lfs df -h /mnt/testfs

# Inode usage
lfs df -i /mnt/testfs

Server Connectivity

# List loaded Lustre devices
lctl dl

# Check LNET status
lctl get_param nis
lnetctl net show

OST Status

# Check if any OSTs are degraded or unavailable
lctl get_param obdfilter.*.degraded
lctl get_param osc.*.state

Common Mount Options

Option Applies To Description
_netdev Servers, Clients Wait for network before mounting.
noauto Servers, Clients Do not mount at boot (use with init scripts or systemd).
flock Clients Enable filesystem-wide POSIX flock (default: disabled).
localflock Clients Enable node-local flock only.
user_xattr Clients Enable user extended attributes.
nouser_xattr Clients Disable user extended attributes.
lazystatfs Clients statfs does not block if an OST is unavailable.

See Also

Retrieved from "http://wiki.lustre.org/index.php?title=Lustre_Day_to_Day_Operations&oldid=5582"