Lustre Deployment Patterns

Lustre Deployment Patterns

This page describes common Lustre deployment architectures at different scales. Use these patterns as starting points for your own deployment. For detailed hardware sizing, see Lustre Hardware Sizing Guide.

Pattern 1: Minimal (3 Nodes)

Use case: Development, testing, learning Lustre, small research groups.

Pattern 1 Topology

Node	Role	Storage	Notes
node1	MGS + MDS (combined)	MDT: 100 GB SSD	Serves both management and metadata
node2	OSS	OST0: 24 TB HDD	Single object storage server
node3	Client	—	Mounts the filesystem
Network: All nodes connected via 10/25 GbE

Characteristics:

• Combined MGS/MDT on a single device, single MDS node.
• One OSS with one or two OSTs.
• No high availability — if any server fails, the filesystem is down.
• Total capacity: 24–48 TB.

When to use this: Learning Lustre, CI/CD testing environments, small datasets with few concurrent users. Not suitable for production workloads.

Setup guide: Lustre Quick Start Guide

Pattern 2: Small Production (5–10 Nodes)

Use case: Small research groups, departmental storage, moderate datasets.

Pattern 2 Topology

Node	Role	Storage	Notes
mds1	MGS + MDS (combined)	MDT: 500 GB SSD	Single metadata server
oss1	OSS	OST0, OST1 (2× 24 TB each)
oss2	OSS	OST2, OST3 (2× 24 TB each)	Scale to 4 OSSs as needed
3–8 nodes	Clients	—
Network: Dedicated Lustre network (InfiniBand or 25+ GbE)

Characteristics:

• Combined MGS/MDT. Single MDS is sufficient for < 100M files.
• 2–4 OSSs, each with 2–4 OSTs.
• No HA — acceptable when downtime for repair is tolerable.
• Total capacity: 100 TB – 1 PB.

When to add HA: If the filesystem supports production workloads where downtime costs more than the HA hardware.

Pattern 3: Medium Production with HA (20–100 Nodes)

Use case: Department clusters, medium HPC systems, production environments requiring uptime.

Pattern 3 Topology — HA Pairs

HA Pair	Nodes	Role	Storage	Failover Mode
MDS pair	mds1, mds2	MGS + MDS	MGS+MDT on shared SSD	Active/passive
OSS pair A	oss1, oss2	OSS	OST0–3 on shared storage	Active/active (each serves 2 OSTs normally; one takes all 4 on failover)
OSS pair B	oss3, oss4	OSS	OST4–7 on shared storage	Active/active
Clients: 20–100 nodes. Network: Dedicated Lustre fabric. Fencing: STONITH required for all HA pairs.

Characteristics:

• MDS in an HA pair (active/passive). Both MDS nodes can access the MDT via shared storage (SAN, multipath, or ZFS with shared disk).
• OSSs in HA pairs (active/active). Each pair shares access to their OSTs. In normal operation, each OSS serves half the OSTs. On failover, one OSS takes over all OSTs in the pair.
• Pacemaker/Corosync manages failover.
• Total capacity: 500 TB – 5 PB.
• Typical: 4–8 OSSs (2–4 HA pairs), each with 4–8 OSTs.

Key design decisions:

• Shared storage for HA: The MDS and OSS pairs need shared access to their targets. This is typically achieved with a SAN (Fibre Channel or iSCSI) or shared JBODs with multipath.
• Network separation: Lustre traffic should be on a dedicated network or VLAN, separate from management traffic.
• Fencing: HA requires STONITH/fencing to prevent split-brain. See Lustre Server Fault Isolation with Pacemaker Node Fencing.

Setup guide: Creating a Framework for High Availability with Pacemaker

Pattern 4: Large HPC with DNE (100+ Nodes)

Use case: Large HPC centers, national labs, cloud-scale storage.

Pattern 4 Topology — Large HPC with DNE

Component	Nodes	Storage	Notes
MGS (HA pair)	Standalone or co-located with mds-pair-1	MGT (< 100 MB)	Independent HA for manageability
MDS pair 1	2 nodes (active/passive)	MDT0	Serves namespace subtree A
MDS pair 2	2 nodes (active/passive)	MDT1	Serves namespace subtree B (add more pairs as needed)
OSS HA pairs	25–100+ pairs	OST0 – OST399+	Each pair: 4–8 OSTs on shared storage
LNet routers	2+ nodes (optional)	—	Required only if clients and servers are on different network fabrics
Clients	100–10,000+ nodes	—	Connected via high-speed fabric; Multi-Rail recommended

Characteristics:

• Separate MGS — independent of any MDS, for manageability. The MGS is small (< 100 MB) and can be its own HA pair.
• Multiple MDTs (DNE) — the filesystem namespace is distributed across 2–4+ MDTs, each on its own MDS server. Different top-level directories (or even subdirectories within a directory) can be assigned to different MDTs.
• Many OSS HA pairs — 25 to 100+ pairs, each with 4–8 OSTs.
• LNet routing — if clients are on a different network fabric than servers (e.g., Ethernet clients accessing InfiniBand servers), LNet routers bridge the networks.
• Multi-Rail — servers and clients use multiple network interfaces for bandwidth aggregation and fault tolerance.
• Total capacity: 10 PB – 1 EB.
• Total OSTs: hundreds to thousands.

DNE considerations:

• Each top-level directory is created on a specific MDT. Use lfs mkdir -i <mdt_index> to control placement.
• Subdirectories default to the parent's MDT unless explicitly set otherwise.
• DNE is transparent to applications — the unified namespace looks like a single directory tree.
• Striped directories (a single directory spread across multiple MDTs) help when a single directory has very high metadata load.

When DNE helps:

• More than ~4 billion files (ldiskfs MDT limit)
• Metadata operation rate exceeding what a single MDS can handle
• Need to isolate different projects/groups onto different MDTs

Co-location Decisions

Question	Recommendation
Combine MGS and MDT?	Yes for small/medium deployments. Separate only when managing multiple independent filesystems or needing independent MGS HA.
Put MDS and OSS on the same node?	No for production. The MDS and OSS have very different I/O patterns and compete for CPU, memory, and network. Acceptable only for testing or very small deployments.
Run Lustre clients on server nodes?	No. Mounting the filesystem on a server node adds lock contention and can cause deadlocks under memory pressure. Use dedicated client nodes.
Multiple OSTs per OSS?	Yes, 2–8 is typical. More than 8 per OSS may bottleneck on network or CPU. Fewer than 2 underutilizes the OSS hardware.
Multiple MDTs per MDS?	1–4 is typical. More MDTs per MDS reduces the benefit of DNE (the MDS is the bottleneck, not the MDT storage).

Next Steps

• Size your hardware: Lustre Hardware Sizing Guide
• Set up your first filesystem: Lustre Quick Start Guide
• Configure high availability: Creating a Framework for High Availability with Pacemaker
• Understand the architecture: Lustre Architecture for Admins