Elliswilson: Created page with "== Lustre Quick Start Guide == This guide walks you through setting up a minimal, working Lustre filesystem from scratch. By the end, you will have a functioning three-node Lustre cluster: * '''node1''' — Combined MGS/MDS (metadata + management) * '''node2''' — OSS (object storage, one OST) * '''node3''' — Client This guide uses ldiskfs as the backend filesystem and TCP networking. For ZFS, see Lustre with ZFS Install. For InfiniBand, see Infiniband Config..."

2026-04-30T18:17:43Z

Created page with "== Lustre Quick Start Guide == This guide walks you through setting up a minimal, working Lustre filesystem from scratch. By the end, you will have a functioning three-node Lustre cluster: * '''node1''' — Combined MGS/MDS (metadata + management) * '''node2''' — OSS (object storage, one OST) * '''node3''' — Client This guide uses ldiskfs as the backend filesystem and TCP networking. For ZFS, see Lustre with ZFS Install. For InfiniBand, see Infiniband Config..."

New page

== Lustre Quick Start Guide ==

This guide walks you through setting up a minimal, working Lustre
filesystem from scratch. By the end, you will have a functioning
three-node Lustre cluster:

* '''node1''' — Combined MGS/MDS (metadata + management)
* '''node2''' — OSS (object storage, one OST)
* '''node3''' — Client

This guide uses ldiskfs as the backend filesystem and TCP networking.
For ZFS, see [[Lustre with ZFS Install]]. For InfiniBand, see
[[Infiniband Configuration Howto]].

=== Prerequisites ===

'''Hardware (per node):'''
* 64-bit x86 CPU
* At least 2 GB RAM (more for MDS — see [[Lustre Hardware Sizing Guide]])
* Network connectivity between all three nodes (Ethernet is fine for testing)

'''Storage:'''
* '''node1 (MDS):''' One dedicated block device for the combined MGT/MDT (e.g., <code>/dev/sdb</code>). SSD recommended.
* '''node2 (OSS):''' One dedicated block device for the OST (e.g., <code>/dev/sdb</code>). Do not partition — use the entire device.

'''Software:'''
* A supported Linux distribution: RHEL/Rocky/Alma 8.x or 9.x, SLES 15 SPx, or Ubuntu 22.04+/Debian 12+
* Lustre packages installed on all nodes. See [[Compiling Lustre]] (from source) or [[Installing the Lustre Software]] (from RPMs)
* <code>e2fsprogs</code> (Lustre-patched version) on servers

'''Networking:'''
* All nodes must be able to reach each other on '''TCP port 988'''. Open this port in any firewall:
firewall-cmd --permanent --add-port=988/tcp
firewall-cmd --reload
* All nodes must have consistent UID/GID mappings (same <code>/etc/passwd</code> and <code>/etc/group</code>, or LDAP/NIS).
* Client clocks should be synchronized (NTP or chrony).

=== Step 1: Configure LNet ===

By default, LNet uses the first available TCP interface. If this is
correct for your environment, no LNet configuration is needed.

If you need to specify which network interface or IP address to use,
configure LNet before formatting targets.

'''Option A: lnetctl (recommended)'''

On each node, create <code>/etc/lnet.conf</code>:

net:
- net type: tcp
local NI:
- nid: 10.0.0.1@tcp
interfaces:
0: eth1

Then load and configure LNet:

modprobe lnet
lnetctl lnet configure
lnetctl import /etc/lnet.conf
lnetctl net show

'''Option B: Module parameter (simpler, legacy)'''

Create <code>/etc/modprobe.d/lnet.conf</code> on each node:

options lnet networks=tcp(eth1)

Then load LNet:

modprobe lnet
lnetctl lnet configure
lnetctl net show

'''Verify:''' On each node, confirm LNet is up and shows the expected
NID:

$ lnetctl net show
net:
- net type: lo
local NI:
- nid: 0@lo
- net type: tcp
local NI:
- nid: 10.0.0.1@tcp

Note the NID of your MDS node (e.g., <code>10.0.0.1@tcp</code>).
You will need it when formatting OSTs and mounting clients.

=== Step 2: Format the Combined MGS/MDT (node1) ===

On '''node1''', format the block device as a combined MGS + MDT:

mkfs.lustre --fsname=testfs --mgs --mdt --index=0 /dev/sdb

* <code>--fsname=testfs</code> — The filesystem name (up to 8 characters, alphanumeric).
* <code>--mgs</code> — This target will also serve as the MGS.
* <code>--mdt --index=0</code> — This is the first (and only) MDT.

This creates an ldiskfs filesystem on <code>/dev/sdb</code> with
Lustre metadata structures.

=== Step 3: Format the OST (node2) ===

On '''node2''', format the block device as an OST:

mkfs.lustre --fsname=testfs --mgsnode=10.0.0.1@tcp \
--ost --index=0 /dev/sdb

* <code>--mgsnode=10.0.0.1@tcp</code> — The NID of the MGS (node1). Replace with your actual MDS NID from Step 1.
* <code>--ost --index=0</code> — This is the first OST.

'''To add more OSTs:''' Format additional block devices with
<code>--index=1</code>, <code>--index=2</code>, etc. Each OST must
be a separate block device.

=== Step 4: Mount the Servers ===

'''Order matters.''' Mount the MGS/MDT first, then the OSTs.

On '''node1''' (MGS/MDS):

mkdir -p /mnt/mdt
mount -t lustre /dev/sdb /mnt/mdt

On '''node2''' (OSS):

mkdir -p /mnt/ost0
mount -t lustre /dev/sdb /mnt/ost0

Verify that both servers started successfully:

# On node1:
dmesg | tail -20
# Look for: "Lustre: testfs-MDT0000: now serving testfs ..."

# On node2:
dmesg | tail -20
# Look for: "Lustre: testfs-OST0000: now serving testfs ..."

=== Step 5: Mount the Client (node3) ===

On '''node3''', mount the filesystem:

mkdir -p /mnt/lustre
mount -t lustre 10.0.0.1@tcp:/testfs /mnt/lustre

* <code>10.0.0.1@tcp</code> — The NID of the MGS.
* <code>/testfs</code> — The filesystem name, preceded by a slash.

Verify the mount:

$ df -h /mnt/lustre
Filesystem Size Used Avail Use% Mounted on
10.0.0.1@tcp:/testfs 50G 38M 47G 1% /mnt/lustre

$ lfs df /mnt/lustre
UUID 1K-blocks Used Available Use% Mounted on
testfs-MDT0000_UUID 5135808 23840 4849856 0% /mnt/lustre[MDT:0]
testfs-OST0000_UUID 51502080 38000 48899760 0% /mnt/lustre[OST:0]

=== Step 6: Test the Filesystem ===

On the client, create a file and verify it works:

# Create a test file
echo "Hello, Lustre!" > /mnt/lustre/hello.txt
cat /mnt/lustre/hello.txt

# Check the file's layout (which OST stores it)
lfs getstripe /mnt/lustre/hello.txt

# Create a larger file and check performance
dd if=/dev/zero of=/mnt/lustre/testfile bs=1M count=100
lfs getstripe /mnt/lustre/testfile

# Check filesystem usage
lfs df -h /mnt/lustre

=== Step 7: Shutdown ===

When you need to stop the filesystem, unmount in reverse order:

# 1. Unmount clients first
# On node3:
umount /mnt/lustre

# 2. Unmount OSTs
# On node2:
umount /mnt/ost0

# 3. Unmount MDT/MGT last
# On node1:
umount /mnt/mdt

=== Troubleshooting ===

{| class="wikitable"
|-
! Problem !! Likely Cause !! Fix
|-
| Client mount hangs
| Port 988 blocked, or wrong MGS NID
| Check firewall rules. Verify MGS NID with <code>lnetctl net show</code> on node1.
|-
| <code>mount: /mnt/lustre: mount(2) system call failed: Connection timed out</code>
| LNet not configured, or network unreachable
| Run <code>lnetctl net show</code> on all nodes. Verify nodes can ping each other. Check that <code>lnet</code> module is loaded.
|-
| <code>mkfs.lustre: command not found</code>
| Lustre server packages not installed
| Install <code>lustre</code> and <code>lustre-osd-ldiskfs-mount</code> packages.
|-
| <code>mount.lustre: mount /dev/sdb ... failed: No such device</code>
| Lustre kernel modules not loaded
| Run <code>modprobe lustre</code> (servers) or <code>modprobe lustre</code> (clients).
|-
| OST mount fails with "no MGS"
| MGS/MDT not mounted yet
| Mount the MGS/MDT on node1 first. Servers must start in order: MGT → MDT → OST.
|-
| <code>e2fsck: Bad magic number in super-block</code>
| Wrong device, or device was not formatted with mkfs.lustre
| Double-check the device path. Format with <code>mkfs.lustre</code> first.
|}

=== What Next? ===

You now have a working Lustre filesystem. Here are suggested next steps:

* '''Add more OSTs''' for additional storage capacity. See [[Lustre File System Expansion]].
* '''Configure file striping''' for your workload. See [[Lustre Striping Best Practices]].
* '''Set up high availability''' so the filesystem survives server failures. See [[Creating a Framework for High Availability with Pacemaker]].
* '''Read the common mistakes page''' before going to production. See [[Lustre Common Mistakes]].
* '''Explore monitoring''' to understand your filesystem's health. See [[Lustre Health Checks]].

Lustre Quick Start Guide - Revision history