Lustre Quick Start Guide

From Lustre Wiki
Jump to navigation Jump to search

Lustre Quick Start Guide

This guide walks you through setting up a minimal, working Lustre filesystem from scratch. By the end, you will have a functioning three-node Lustre cluster:

  • node1 — Combined MGS/MDS (metadata + management)
  • node2 — OSS (object storage, one OST)
  • node3 — Client

This guide uses ldiskfs as the backend filesystem and TCP networking. For ZFS, see Lustre with ZFS Install. For InfiniBand, see Infiniband Configuration Howto.

Prerequisites

Hardware (per node):

  • 64-bit x86 CPU
  • At least 2 GB RAM (more for MDS — see Lustre Hardware Sizing Guide)
  • Network connectivity between all three nodes (Ethernet is fine for testing)

Storage:

  • node1 (MDS): One dedicated block device for the combined MGT/MDT (e.g., /dev/sdb). SSD recommended.
  • node2 (OSS): One dedicated block device for the OST (e.g., /dev/sdb). Do not partition — use the entire device.

Software:

  • A supported Linux distribution: RHEL/Rocky/Alma 8.x or 9.x, SLES 15 SPx, or Ubuntu 22.04+/Debian 12+
  • Lustre packages installed on all nodes. See Compiling Lustre (from source) or Installing the Lustre Software (from RPMs)
  • e2fsprogs (Lustre-patched version) on servers

Networking:

  • All nodes must be able to reach each other on TCP port 988. Open this port in any firewall:
firewall-cmd --permanent --add-port=988/tcp
firewall-cmd --reload
  • All nodes must have consistent UID/GID mappings (same /etc/passwd and /etc/group, or LDAP/NIS).
  • Client clocks should be synchronized (NTP or chrony).

Step 1: Configure LNet

By default, LNet uses the first available TCP interface. If this is correct for your environment, no LNet configuration is needed.

If you need to specify which network interface or IP address to use, configure LNet before formatting targets.

Option A: lnetctl (recommended)

On each node, create /etc/lnet.conf: 

net:
    - net type: tcp
      local NI:
        - nid: 10.0.0.1@tcp
          interfaces:
              0: eth1

Then load and configure LNet: 

modprobe lnet
lnetctl lnet configure
lnetctl import /etc/lnet.conf
lnetctl net show

Option B: Module parameter (simpler, legacy)

Create /etc/modprobe.d/lnet.conf on each node: 

options lnet networks=tcp(eth1)

Then load LNet: 

modprobe lnet
lnetctl lnet configure
lnetctl net show

Verify: On each node, confirm LNet is up and shows the expected NID: 

$ lnetctl net show
net:
    - net type: lo
      local NI:
        - nid: 0@lo
    - net type: tcp
      local NI:
        - nid: 10.0.0.1@tcp

Note the NID of your MDS node (e.g., 10.0.0.1@tcp). You will need it when formatting OSTs and mounting clients.

Step 2: Format the Combined MGS/MDT (node1)

On node1, format the block device as a combined MGS + MDT: 

mkfs.lustre --fsname=testfs --mgs --mdt --index=0 /dev/sdb
  • --fsname=testfs — The filesystem name (up to 8 characters, alphanumeric).
  • --mgs — This target will also serve as the MGS.
  • --mdt --index=0 — This is the first (and only) MDT.

This creates an ldiskfs filesystem on /dev/sdb with Lustre metadata structures.

Step 3: Format the OST (node2)

On node2, format the block device as an OST: 

mkfs.lustre --fsname=testfs --mgsnode=10.0.0.1@tcp \
    --ost --index=0 /dev/sdb
  • --mgsnode=10.0.0.1@tcp — The NID of the MGS (node1). Replace with your actual MDS NID from Step 1.
  • --ost --index=0 — This is the first OST.

To add more OSTs: Format additional block devices with --index=1, --index=2, etc. Each OST must be a separate block device.

Step 4: Mount the Servers

Order matters. Mount the MGS/MDT first, then the OSTs.

On node1 (MGS/MDS): 

mkdir -p /mnt/mdt
mount -t lustre /dev/sdb /mnt/mdt

On node2 (OSS): 

mkdir -p /mnt/ost0
mount -t lustre /dev/sdb /mnt/ost0

Verify that both servers started successfully: 

# On node1:
dmesg | tail -20
# Look for: "Lustre: testfs-MDT0000: now serving testfs ..."

# On node2:
dmesg | tail -20
# Look for: "Lustre: testfs-OST0000: now serving testfs ..."

Step 5: Mount the Client (node3)

On node3, mount the filesystem: 

mkdir -p /mnt/lustre
mount -t lustre 10.0.0.1@tcp:/testfs /mnt/lustre
  • 10.0.0.1@tcp — The NID of the MGS.
  • /testfs — The filesystem name, preceded by a slash.

Verify the mount: 

$ df -h /mnt/lustre
Filesystem           Size  Used Avail Use% Mounted on
10.0.0.1@tcp:/testfs  50G   38M   47G   1% /mnt/lustre

$ lfs df /mnt/lustre
UUID                 1K-blocks  Used   Available Use% Mounted on
testfs-MDT0000_UUID   5135808  23840    4849856   0% /mnt/lustre[MDT:0]
testfs-OST0000_UUID  51502080  38000   48899760   0% /mnt/lustre[OST:0]

Step 6: Test the Filesystem

On the client, create a file and verify it works: 

# Create a test file
echo "Hello, Lustre!" > /mnt/lustre/hello.txt
cat /mnt/lustre/hello.txt

# Check the file's layout (which OST stores it)
lfs getstripe /mnt/lustre/hello.txt

# Create a larger file and check performance
dd if=/dev/zero of=/mnt/lustre/testfile bs=1M count=100
lfs getstripe /mnt/lustre/testfile

# Check filesystem usage
lfs df -h /mnt/lustre

Step 7: Shutdown

When you need to stop the filesystem, unmount in reverse order: 

# 1. Unmount clients first
# On node3:
umount /mnt/lustre

# 2. Unmount OSTs
# On node2:
umount /mnt/ost0

# 3. Unmount MDT/MGT last
# On node1:
umount /mnt/mdt

Troubleshooting

Problem Likely Cause Fix
Client mount hangs Port 988 blocked, or wrong MGS NID Check firewall rules. Verify MGS NID with lnetctl net show on node1.
mount: /mnt/lustre: mount(2) system call failed: Connection timed out LNet not configured, or network unreachable Run lnetctl net show on all nodes. Verify nodes can ping each other. Check that lnet module is loaded.
mkfs.lustre: command not found Lustre server packages not installed Install lustre and lustre-osd-ldiskfs-mount packages.
mount.lustre: mount /dev/sdb ... failed: No such device Lustre kernel modules not loaded Run modprobe lustre (servers) or modprobe lustre (clients).
OST mount fails with "no MGS" MGS/MDT not mounted yet Mount the MGS/MDT on node1 first. Servers must start in order: MGT → MDT → OST.
e2fsck: Bad magic number in super-block Wrong device, or device was not formatted with mkfs.lustre Double-check the device path. Format with mkfs.lustre first.

What Next?

You now have a working Lustre filesystem. Here are suggested next steps:

Retrieved from "http://wiki.lustre.org/index.php?title=Lustre_Quick_Start_Guide&oldid=5477"