Elliswilson: Created page with "== Lustre Deployment Patterns == This page describes common Lustre deployment architectures at different scales. Use these patterns as starting points for your own deployment. For detailed hardware sizing, see Lustre Hardware Sizing Guide. === Pattern 1: Minimal (3 Nodes) === '''Use case:''' Development, testing, learning Lustre, small research groups. {| class="wikitable" |+ '''Pattern 1 Topology''' |- ! Node !! Role !! Storage !! Notes |- | node1 || MGS + MDS (..."

2026-05-11T14:50:12Z

Created page with "== Lustre Deployment Patterns == This page describes common Lustre deployment architectures at different scales. Use these patterns as starting points for your own deployment. For detailed hardware sizing, see Lustre Hardware Sizing Guide. === Pattern 1: Minimal (3 Nodes) === '''Use case:''' Development, testing, learning Lustre, small research groups. {| class="wikitable" |+ '''Pattern 1 Topology''' |- ! Node !! Role !! Storage !! Notes |- | node1 || MGS + MDS (..."

New page

== Lustre Deployment Patterns ==

This page describes common Lustre deployment architectures at different
scales. Use these patterns as starting points for your own deployment.
For detailed hardware sizing, see [[Lustre Hardware Sizing Guide]].

=== Pattern 1: Minimal (3 Nodes) ===

'''Use case:''' Development, testing, learning Lustre, small research
groups.

{| class="wikitable"
|+ '''Pattern 1 Topology'''
|-
! Node !! Role !! Storage !! Notes
|-
| node1 || MGS + MDS (combined) || MDT: 100 GB SSD || Serves both management and metadata
|-
| node2 || OSS || OST0: 24 TB HDD || Single object storage server
|-
| node3 || Client || — || Mounts the filesystem
|-
| colspan="4" | '''Network:''' All nodes connected via 10/25 GbE
|}

'''Characteristics:'''
* Combined MGS/MDT on a single device, single MDS node.
* One OSS with one or two OSTs.
* No high availability — if any server fails, the filesystem is down.
* Total capacity: 24–48 TB.

'''When to use this:''' Learning Lustre, CI/CD testing environments,
small datasets with few concurrent users. Not suitable for production
workloads.

'''Setup guide:''' [[Lustre Quick Start Guide]]

=== Pattern 2: Small Production (5–10 Nodes) ===

'''Use case:''' Small research groups, departmental storage, moderate
datasets.

{| class="wikitable"
|+ '''Pattern 2 Topology'''
|-
! Node !! Role !! Storage !! Notes
|-
| mds1 || MGS + MDS (combined) || MDT: 500 GB SSD || Single metadata server
|-
| oss1 || OSS || OST0, OST1 (2× 24 TB each) ||
|-
| oss2 || OSS || OST2, OST3 (2× 24 TB each) || Scale to 4 OSSs as needed
|-
| 3–8 nodes || Clients || — ||
|-
| colspan="4" | '''Network:''' Dedicated Lustre network (InfiniBand or 25+ GbE)
|}

'''Characteristics:'''
* Combined MGS/MDT. Single MDS is sufficient for < 100M files.
* 2–4 OSSs, each with 2–4 OSTs.
* No HA — acceptable when downtime for repair is tolerable.
* Total capacity: 100 TB – 1 PB.

'''When to add HA:''' If the filesystem supports production workloads
where downtime costs more than the HA hardware.

=== Pattern 3: Medium Production with HA (20–100 Nodes) ===

'''Use case:''' Department clusters, medium HPC systems, production
environments requiring uptime.

{| class="wikitable"
|+ '''Pattern 3 Topology — HA Pairs'''
|-
! HA Pair !! Nodes !! Role !! Storage !! Failover Mode
|-
| MDS pair || mds1, mds2 || MGS + MDS || MGS+MDT on shared SSD || Active/passive
|-
| OSS pair A || oss1, oss2 || OSS || OST0–3 on shared storage || Active/active (each serves 2 OSTs normally; one takes all 4 on failover)
|-
| OSS pair B || oss3, oss4 || OSS || OST4–7 on shared storage || Active/active
|-
| colspan="5" | '''Clients:''' 20–100 nodes. '''Network:''' Dedicated Lustre fabric. '''Fencing:''' STONITH required for all HA pairs.
|}

'''Characteristics:'''
* MDS in an HA pair (active/passive). Both MDS nodes can access the MDT via shared storage (SAN, multipath, or ZFS with shared disk).
* OSSs in HA pairs (active/active). Each pair shares access to their OSTs. In normal operation, each OSS serves half the OSTs. On failover, one OSS takes over all OSTs in the pair.
* Pacemaker/Corosync manages failover.
* Total capacity: 500 TB – 5 PB.
* Typical: 4–8 OSSs (2–4 HA pairs), each with 4–8 OSTs.

'''Key design decisions:'''
* '''Shared storage for HA:''' The MDS and OSS pairs need shared access to their targets. This is typically achieved with a SAN (Fibre Channel or iSCSI) or shared JBODs with multipath.
* '''Network separation:''' Lustre traffic should be on a dedicated network or VLAN, separate from management traffic.
* '''Fencing:''' HA requires STONITH/fencing to prevent split-brain. See [[Lustre Server Fault Isolation with Pacemaker Node Fencing]].

'''Setup guide:''' [[Creating a Framework for High Availability with Pacemaker]]

=== Pattern 4: Large HPC with DNE (100+ Nodes) ===

'''Use case:''' Large HPC centers, national labs, cloud-scale storage.

{| class="wikitable"
|+ '''Pattern 4 Topology — Large HPC with DNE'''
|-
! Component !! Nodes !! Storage !! Notes
|-
| MGS (HA pair) || Standalone or co-located with mds-pair-1 || MGT (< 100 MB) || Independent HA for manageability
|-
| MDS pair 1 || 2 nodes (active/passive) || MDT0 || Serves namespace subtree A
|-
| MDS pair 2 || 2 nodes (active/passive) || MDT1 || Serves namespace subtree B (add more pairs as needed)
|-
| OSS HA pairs || 25–100+ pairs || OST0 – OST399+ || Each pair: 4–8 OSTs on shared storage
|-
| LNet routers || 2+ nodes (optional) || — || Required only if clients and servers are on different network fabrics
|-
| Clients || 100–10,000+ nodes || — || Connected via high-speed fabric; Multi-Rail recommended
|}

'''Characteristics:'''
* '''Separate MGS''' — independent of any MDS, for manageability. The MGS is small (< 100 MB) and can be its own HA pair.
* '''Multiple MDTs (DNE)''' — the filesystem namespace is distributed across 2–4+ MDTs, each on its own MDS server. Different top-level directories (or even subdirectories within a directory) can be assigned to different MDTs.
* '''Many OSS HA pairs''' — 25 to 100+ pairs, each with 4–8 OSTs.
* '''LNet routing''' — if clients are on a different network fabric than servers (e.g., Ethernet clients accessing InfiniBand servers), LNet routers bridge the networks.
* '''Multi-Rail''' — servers and clients use multiple network interfaces for bandwidth aggregation and fault tolerance.
* Total capacity: 10 PB – 1 EB.
* Total OSTs: hundreds to thousands.

'''DNE considerations:'''
* Each top-level directory is created on a specific MDT. Use <code>lfs mkdir -i <mdt_index></code> to control placement.
* Subdirectories default to the parent's MDT unless explicitly set otherwise.
* DNE is transparent to applications — the unified namespace looks like a single directory tree.
* Striped directories (a single directory spread across multiple MDTs) help when a single directory has very high metadata load.

'''When DNE helps:'''
* More than ~4 billion files (ldiskfs MDT limit)
* Metadata operation rate exceeding what a single MDS can handle
* Need to isolate different projects/groups onto different MDTs

=== Co-location Decisions ===

{| class="wikitable"
|-
! Question !! Recommendation
|-
| Combine MGS and MDT?
| '''Yes''' for small/medium deployments. Separate only when managing
multiple independent filesystems or needing independent MGS HA.
|-
| Put MDS and OSS on the same node?
| '''No''' for production. The MDS and OSS have very different I/O
patterns and compete for CPU, memory, and network. Acceptable only
for testing or very small deployments.
|-
| Run Lustre clients on server nodes?
| '''No.''' Mounting the filesystem on a server node adds lock
contention and can cause deadlocks under memory pressure. Use
dedicated client nodes.
|-
| Multiple OSTs per OSS?
| '''Yes, 2–8 is typical.''' More than 8 per OSS may bottleneck on
network or CPU. Fewer than 2 underutilizes the OSS hardware.
|-
| Multiple MDTs per MDS?
| '''1–4 is typical.''' More MDTs per MDS reduces the benefit of DNE
(the MDS is the bottleneck, not the MDT storage).
|}

=== Next Steps ===

* '''Size your hardware:''' [[Lustre Hardware Sizing Guide]]
* '''Set up your first filesystem:''' [[Lustre Quick Start Guide]]
* '''Configure high availability:''' [[Creating a Framework for High Availability with Pacemaker]]
* '''Understand the architecture:''' [[Lustre Architecture for Admins]]

Lustre Deployment Patterns - Revision history