Introduction to Lustre - Revision history

Elliswilson at 15:12, 11 May 2026

2026-05-11T15:12:32Z

Adilger: /* Lustre Architecture */ minor updates

2024-05-03T02:14:38Z

Lustre Architecture: minor updates

← Older revision		Revision as of 19:14, 2 May 2024
Line 20:		Line 20:
	The building blocks of a Lustre file system are the Metadata Servers and Object Storage Servers, which provide namespace operations and bulk IO services respectively. There is also the Management Server, which is a global registry for configuration information that is functionally independent of any single Lustre instance. Lustre clients provide the interface between applications and the Lustre services. It is the client software that presents a coherent, POSIX interface to end-user applications. Lustre’s network protocol, LNet, provides the communications framework that binds the services together.		The building blocks of a Lustre file system are the Metadata Servers and Object Storage Servers, which provide namespace operations and bulk IO services respectively. There is also the Management Server, which is a global registry for configuration information that is functionally independent of any single Lustre instance. Lustre clients provide the interface between applications and the Lustre services. It is the client software that presents a coherent, POSIX interface to end-user applications. Lustre’s network protocol, LNet, provides the communications framework that binds the services together.

	'''Metadata Server (MDS):''' The MDS manages all ~~name space~~ operations for a Lustre file system. A file system’s directory hierarchy and file information are contained on storage devices referred to as Metadata Targets (MDT), and the MDS provides the logical interface to this storage. A Lustre file system will always have at least one MDS and corresponding MDT, and more can be added to meet the scaling requirements of a particular environment. The MDS controls the allocation of storage objects on the Object Storage Servers for the file content when a file is created~~, and manages the opening and closing of files, file deletions and renames, and other namespace operations~~.		'''Metadata Server (MDS):''' The MDS node manages all namespace operations for a Lustre file system, such as directories (folders) and filenames. A file system’s directory hierarchy and file information are contained on storage devices referred to as Metadata Targets (MDT), and the MDS provides the logical interface to this storage. A Lustre file system will always have at least one MDS and corresponding MDT, and more can be added to meet the scaling requirements of a particular environment. The MDS controls file access permission, the opening and closing of files, file deletions and renames, and other namespace operations, as well as the allocation of storage ''objects'' on the Object Storage Servers for the file content when a file is created.

	An MDT stores namespace metadata, such as filenames, directories, access permissions, and file layout, effectively providing the index for the data held on the file system. The ability to have multiple MDTs in a single file system allows ~~directory subtrees~~ to reside on the secondary MDTs, which is useful for isolating workloads that are especially metadata-intensive onto dedicated hardware (one could allocate an MDT for a specific set of projects, for example). Large, single directories can be distributed across multiple MDTs as well, providing scalability for applications that generate ~~large numbers~~ of files in a flat directory ~~hierarchy~~.		An MDT stores namespace metadata, such as filenames, directories, access permissions, and file layout, effectively providing the index for the data held on the file system. The ability to have multiple MDTs in a single file system allows subdirectory trees to reside on the secondary MDTs, which is useful for isolating workloads that are especially metadata-intensive onto dedicated hardware (one could allocate an MDT for a specific set of projects, for example), or for balancing the workload and metadata capacity across multiple servers. Large, single directories can be distributed across multiple MDTs as well, providing scalability for applications that generate millions of files in a single flat directory.

	'''Object Storage Server (OSS):''' ~~OSSs provide~~ bulk storage for the contents of files in a Lustre file system. One or more object storage servers (OSS) store file data on one or more object storage targets (OST)~~, and a~~ single Lustre file system can scale to hundreds of ~~OSSs~~. A single OSS typically serves between two and eight OSTs (although more are possible), with the OSTs stored on direct-attached storage. The capacity of a Lustre file system is the sum of the capacities provided by the OSTs across all of the OSS hosts.		'''Object Storage Server (OSS):''' OSS nodes manage the bulk storage for the contents of files in a Lustre file system. One or more object storage servers (OSS) store the file data on one or more object storage targets (OST). A single Lustre file system can scale to hundreds of OSS nodes and thousands of OSTs. A single OSS typically serves between two and eight OSTs (although more are possible), with the OSTs stored on direct-attached storage. The capacity of a Lustre file system is the sum of the capacities provided by the OSTs across all of the OSS hosts.

	~~OSSs~~ are usually configured in pairs, with each ~~pair~~ connected to a shared ~~external~~ storage enclosure that stores the OSTs. The OSTs in the enclosure are accessible ~~via~~ the ~~two~~ servers in an active-passive ~~high availability~~ failover configuration, to provide service continuity in the event of a server or component failure. The OSTs are mounted on only one server at a time, and are usually evenly distributed across the OSS ~~hosts~~ to balance performance and maximize throughput.		OSS nodes are usually configured in pairs or quads, with each node connected to a shared storage enclosure that stores the HDD or NVMe devices for the OSTs. The OSTs in the enclosure are accessible to the servers in an active-passive High Availability failover configuration, to provide service continuity in the event of a server or component failure. The OSTs are mounted on only one server at a time, and are usually evenly distributed across the OSS nodes to balance performance and maximize throughput.

	'''Management Server (MGS):''' The MGS stores configuration information for all the Lustre file systems in a cluster and provides this information to other Lustre hosts. Servers and clients connect to the MGS on startup in order to retrieve the configuration log for the file system. Notification of changes to a file system’s configuration, including server restarts, are distributed by the MGS.		'''Management Server (MGS):''' The MGS node stores configuration information for all the Lustre file systems in a cluster and provides this information to other Lustre hosts. Servers and clients connect to the MGS on startup in order to retrieve the configuration log for the file system. Notification of changes to a file system’s configuration, including server restarts, are distributed by the MGS.

	Persistent configuration information for all Lustre nodes is recorded by the MGS onto a storage device called a Management Target (MGT). The MGS can be paired with an MDS in a high-availability configuration, with each server connected to shared storage. Multiple Lustre file systems can be managed by a single MGS.		Persistent configuration information for all Lustre nodes is recorded by the MGS onto a storage device called a Management Target (MGT). The MGS can be paired with an MDS in a high-availability configuration, with each server connected to shared storage. Multiple Lustre file systems can be managed by a single MGS, but the low resource requirements for the MGS/MGT usually mean each filesystem will have its own MGS/MGT.

	'''Clients:''' Applications access and use file system data by interfacing with Lustre clients. A Lustre client is represented as a file system mount point on a host and presents applications with a unified namespace for all of the files and data in the file system, using standard POSIX semantics. A Lustre file system mounted on the client operating system looks much like any other POSIX file system; each Lustre instance is presented as a separate mount point on the client’s operating system, and each client can mount several different Lustre file system instances concurrently.		'''Clients:''' Applications access and use file system data by interfacing with Lustre clients. A Lustre client is represented as a file system mount point on a host and presents applications with a unified namespace for all of the files and data in the file system, using standard POSIX semantics. A Lustre file system mounted on the client operating system looks much like any other POSIX file system; each Lustre instance is presented as a separate mount point on the client’s operating system, and each client can mount several different Lustre file system instances concurrently.
Line 38:		Line 38:
	LNet abstracts network details from the file system itself. LNet allows for full RDMA throughput and zero copy communications when available.		LNet abstracts network details from the file system itself. LNet allows for full RDMA throughput and zero copy communications when available.

	LNet supports routing, which provides maximum flexibility for connecting different network topologies. LNet routing provides an efficient protocol for bridging different networks, or employing different fabric technologies, such as ~~Intel®~~ OPA and ~~InfiniBand. In larger file systems, with very large client counts~~, LNet can also ~~support multihoming to improve performance~~ and ~~reliability~~. For more details and guidance, see the document Configuring LNet Routers for File Systems ~~based on Intel* EE for Lustre* Software~~.		LNet supports routing, which provides maximum flexibility for connecting different network topologies. LNet routing provides an efficient protocol for bridging different networks, or employing different fabric technologies, such as Ethernet, InfiniBand, and OPA. To improve performance and reliability, LNet can also balance traffic across multiple network interfaces in either/both the client and server nodes, in a Multi-Rail configuration. For more details and guidance, see the document Configuring LNet Routers for File Systems.

	== Lustre Scalability ==		== Lustre Scalability ==

Adilger: /* Overview */ minor updates

2024-05-03T02:04:20Z

Overview: minor updates

← Older revision		Revision as of 19:04, 2 May 2024
Line 1:		Line 1:
	== Overview ==		== Overview ==
	Lustre* is an open-source, global single-namespace, POSIX-compliant, distributed parallel file system designed for scalability, high-performance, and high-availability. Lustre runs on Linux-based operating systems and employs a client-server network architecture. Storage is provided by a set of servers that can scale to populations measuring up to several hundred hosts. Lustre servers for a single file system instance can, in aggregate, present up to ~~tens~~ of petabytes of storage to thousands of compute clients, with ~~more than a terabyte-~~per-second of combined throughput.		Lustre* is an open-source, global single-namespace, POSIX-compliant, distributed parallel file system designed for scalability, high-performance, and high-availability. Lustre runs on Linux-based operating systems and employs a client-server network architecture. Storage is provided by a set of servers that can scale to populations measuring up to several hundred hosts. Lustre servers for a single file system instance can, in aggregate, present up to hundreds of petabytes of storage to thousands of compute clients, with multiple terabytes per second of combined throughput.

	Lustre is a file system that scales to meet the requirements of applications running on a range of systems from small-scale HPC environments up to the very largest supercomputers and has been created using object-based storage building blocks to maximize scalability.		Lustre is a file system that scales to meet the requirements of applications running on a range of systems from small-scale HPC environments up to the very largest supercomputers and has been created using object-based storage building blocks to maximize scalability.

	Redundant servers support storage fail-over, while metadata and data are stored on separate servers, allowing each file system to be optimized for different workloads. Lustre can deliver fast IO to applications across high-speed network fabrics, such as ~~Intel®~~ Omni-Path ~~Architecture~~ (OPA), InfiniBand* and ~~Ethernet~~.		Redundant servers support storage fail-over, while metadata and data are stored on separate servers, allowing each file system to be optimized for different workloads. Lustre can deliver fast IO to applications across high-speed network fabrics, such as Ethernet, InfiniBand (IB), Omni-Path (OPA), and others.

	== Lustre Architecture ==		== Lustre Architecture ==

Malcolm: /* Overview */

2017-09-08T05:10:03Z

Overview

← Older revision		Revision as of 22:10, 7 September 2017
Line 1:		Line 1:
	== Overview ==		== Overview ==
	Lustre* is a global single-namespace, POSIX-compliant, distributed parallel file system designed for scalability, high-performance, and high-availability. Lustre runs on Linux-based operating systems and employs a client-server network architecture. Storage is provided by a set of servers that can scale to populations measuring up to several hundred hosts. Lustre servers for a single file system instance can, in aggregate, present up to tens of petabytes of storage to thousands of compute clients, with more than a terabyte-per-second of combined throughput.		Lustre* is an open-source, global single-namespace, POSIX-compliant, distributed parallel file system designed for scalability, high-performance, and high-availability. Lustre runs on Linux-based operating systems and employs a client-server network architecture. Storage is provided by a set of servers that can scale to populations measuring up to several hundred hosts. Lustre servers for a single file system instance can, in aggregate, present up to tens of petabytes of storage to thousands of compute clients, with more than a terabyte-per-second of combined throughput.

	Lustre is a file system that scales to meet the requirements of applications running on a range of systems from small-scale HPC environments up to the very largest supercomputers and has been created using object-based storage building blocks to maximize scalability.		Lustre is a file system that scales to meet the requirements of applications running on a range of systems from small-scale HPC environments up to the very largest supercomputers and has been created using object-based storage building blocks to maximize scalability.

Malcolm at 05:05, 8 September 2017

2017-09-08T05:05:55Z

← Older revision		Revision as of 22:05, 7 September 2017
Line 1:		Line 1:
	== ~~Introduction~~ ==		== Overview ==
	Lustre* is a global single-namespace, POSIX-compliant, distributed parallel file system designed for scalability, high-performance, and high-availability. Lustre runs on Linux-based operating systems and employs a client-server network architecture. Storage is provided by a set of servers that can scale to populations measuring up to several hundred hosts. Lustre servers for a single file system instance can, in aggregate, present up to tens of petabytes of storage to thousands of compute clients, with more than a terabyte-per-second of combined throughput.		Lustre* is a global single-namespace, POSIX-compliant, distributed parallel file system designed for scalability, high-performance, and high-availability. Lustre runs on Linux-based operating systems and employs a client-server network architecture. Storage is provided by a set of servers that can scale to populations measuring up to several hundred hosts. Lustre servers for a single file system instance can, in aggregate, present up to tens of petabytes of storage to thousands of compute clients, with more than a terabyte-per-second of combined throughput.

Line 6:		Line 6:
	Redundant servers support storage fail-over, while metadata and data are stored on separate servers, allowing each file system to be optimized for different workloads. Lustre can deliver fast IO to applications across high-speed network fabrics, such as Intel® Omni-Path Architecture (OPA), InfiniBand* and Ethernet.		Redundant servers support storage fail-over, while metadata and data are stored on separate servers, allowing each file system to be optimized for different workloads. Lustre can deliver fast IO to applications across high-speed network fabrics, such as Intel® Omni-Path Architecture (OPA), InfiniBand* and Ethernet.

	== ~~Overview of~~ Lustre Architecture ==		== Lustre Architecture ==

Malcolm at 05:02, 8 September 2017

2017-09-08T05:02:51Z

← Older revision		Revision as of 22:02, 7 September 2017
Line 9:		Line 9:


	<div id="Figure 1-1">		<div id="Figure 1">
	[[File: Lustre File System Overview (DNE) lowres v1.png\|thumb\|center\|512px\|Figure 1-1. Lustre File System Architecture]]		[[File: Lustre File System Overview (DNE) lowres v1.png\|thumb\|center\|512px\|Figure 1. Lustre File System Architecture]]
	</div>		</div>

Line 135:		Line 135:
	A single Lustre file system can scale linearly based on the number of building blocks. The minimum HA configuration for Lustre is a metadata and management building block that provides the MDS and MGS services, plus a single object storage building block for the OSS services. Using these basic units, one can create file systems with hundreds of OSSs as well as several MDSs, using HA building blocks to provide a reliable, high-performance platform.		A single Lustre file system can scale linearly based on the number of building blocks. The minimum HA configuration for Lustre is a metadata and management building block that provides the MDS and MGS services, plus a single object storage building block for the OSS services. Using these basic units, one can create file systems with hundreds of OSSs as well as several MDSs, using HA building blocks to provide a reliable, high-performance platform.

	[[#Figure 1-2\|Figure 1-2]] shows a blue-print for typical high-availability Lustre server building blocks, one for the metadata and management services, and one for object storage.		[[#Figure 2\|Figure 2]] shows a blue-print for typical high-availability Lustre server building blocks, one for the metadata and management services, and one for object storage.

	<div id="Figure 1-2">		<div id="Figure 2">
	[[File:Lustre HA Server Building Blocks hires v1.png\|thumb\|center\|512px\|Figure 1-2. Lustre Server High-Availability Building Blocks]]		[[File:Lustre HA Server Building Blocks hires v1.png\|thumb\|center\|512px\|Figure 2. Lustre Server High-Availability Building Blocks]]
	</div>		</div>

Malcolm at 03:41, 8 September 2017

2017-09-08T03:41:20Z

← Older revision		Revision as of 20:41, 7 September 2017
Line 1:		Line 1:
			== Introduction ==
	Lustre* is a global single-namespace, POSIX-compliant, distributed parallel file system designed for scalability, high-performance, and high-availability. Lustre runs on Linux-based operating systems and employs a client-server network architecture. Storage is provided by a set of servers that can scale to populations measuring up to several hundred hosts. Lustre servers for a single file system instance can, in aggregate, present up to tens of petabytes of storage to thousands of compute clients, with more than a terabyte-per-second of combined throughput.		Lustre* is a global single-namespace, POSIX-compliant, distributed parallel file system designed for scalability, high-performance, and high-availability. Lustre runs on Linux-based operating systems and employs a client-server network architecture. Storage is provided by a set of servers that can scale to populations measuring up to several hundred hosts. Lustre servers for a single file system instance can, in aggregate, present up to tens of petabytes of storage to thousands of compute clients, with more than a terabyte-per-second of combined throughput.

Line 6:		Line 7:

	== Overview of Lustre Architecture ==		== Overview of Lustre Architecture ==


			<div id="Figure 1-1">
			[[File: Lustre File System Overview (DNE) lowres v1.png\|thumb\|center\|512px\|Figure 1-1. Lustre File System Architecture]]
			</div>


	The Lustre file system architecture is designed as a scalable storage platform for computer networks and is based on distributed, object-based storage. The namespace hierarchy is stored separately from file content. Services in Lustre are separated into those supporting metadata operations, and those supporting file content operations.		The Lustre file system architecture is designed as a scalable storage platform for computer networks and is based on distributed, object-based storage. The namespace hierarchy is stored separately from file content. Services in Lustre are separated into those supporting metadata operations, and those supporting file content operations.
Line 128:		Line 135:
	A single Lustre file system can scale linearly based on the number of building blocks. The minimum HA configuration for Lustre is a metadata and management building block that provides the MDS and MGS services, plus a single object storage building block for the OSS services. Using these basic units, one can create file systems with hundreds of OSSs as well as several MDSs, using HA building blocks to provide a reliable, high-performance platform.		A single Lustre file system can scale linearly based on the number of building blocks. The minimum HA configuration for Lustre is a metadata and management building block that provides the MDS and MGS services, plus a single object storage building block for the OSS services. Using these basic units, one can create file systems with hundreds of OSSs as well as several MDSs, using HA building blocks to provide a reliable, high-performance platform.

	[[#Figure 1\|Figure 1]] shows a blue-print for typical high-availability Lustre server building blocks, one for the metadata and management services, and one for object storage.		[[#Figure 1-2\|Figure 1-2]] shows a blue-print for typical high-availability Lustre server building blocks, one for the metadata and management services, and one for object storage.

	<div id="Figure 1">		<div id="Figure 1-2">
	[[File:Lustre HA Server Building Blocks hires v1.png\|thumb\|center\|~~450px~~\|Figure 1. Lustre Server High-Availability Building Blocks]]		[[File:Lustre HA Server Building Blocks hires v1.png\|thumb\|center\|512px\|Figure 1-2. Lustre Server High-Availability Building Blocks]]
	</div>		</div>

Malcolm: /* Lustre Scalability */

2017-08-10T05:07:45Z

Lustre Scalability

← Older revision		Revision as of 22:07, 9 August 2017
Line 65:		Line 65:
	\|style="text-align:center;"\|31.25PB		\|style="text-align:center;"\|31.25PB
	\|style="text-align:center;"\|512PB*		\|style="text-align:center;"\|512PB*
	\|\|-		\|
			\|-
	! scope="row" style="text-align:left;"\|Maximum filesystem size		! scope="row" style="text-align:left;"\|Maximum filesystem size
	\|style="text-align:center;"\|512PB		\|style="text-align:center;"\|512PB

Malcolm: /* Lustre Scalability */

2017-08-10T05:07:07Z

Lustre Scalability

← Older revision		Revision as of 22:07, 9 August 2017
Line 64:		Line 64:
	! scope="row" style="text-align:left;"\|Maximum file size		! scope="row" style="text-align:left;"\|Maximum file size
	\|style="text-align:center;"\|31.25PB		\|style="text-align:center;"\|31.25PB
			\|style="text-align:center;"\|512PB*
			\|\|-
			! scope="row" style="text-align:left;"\|Maximum filesystem size
			\|style="text-align:center;"\|512PB
	\|style="text-align:center;"\|8EB*		\|style="text-align:center;"\|8EB*
	\|		\|

Malcolm at 05:01, 10 August 2017

2017-08-10T05:01:37Z

← Older revision		Revision as of 22:01, 9 August 2017
Line 32:		Line 32:

	LNet supports routing, which provides maximum flexibility for connecting different network topologies. LNet routing provides an efficient protocol for bridging different networks, or employing different fabric technologies, such as Intel® OPA and InfiniBand. In larger file systems, with very large client counts, LNet can also support multihoming to improve performance and reliability. For more details and guidance, see the document Configuring LNet Routers for File Systems based on Intel* EE for Lustre* Software.		LNet supports routing, which provides maximum flexibility for connecting different network topologies. LNet routing provides an efficient protocol for bridging different networks, or employing different fabric technologies, such as Intel® OPA and InfiniBand. In larger file systems, with very large client counts, LNet can also support multihoming to improve performance and reliability. For more details and guidance, see the document Configuring LNet Routers for File Systems based on Intel* EE for Lustre* Software.

			== Lustre Scalability ==

			{\| class="wikitable"
			\|-
			! style="width:25%;"\|
			! style="width:25%;"\|Value using LDISKFS as backend
			! style="width:25%;"\|Value using ZFS as backend
			! style="width:25%;"\|Notes
			\|-
			! scope="row" style="text-align:left;"\|Maximum stripe count
			\|style="text-align:center;"\|2000
			\|style="text-align:center;"\|2000
			\|style="text-align:center;"\| Limit is 160 for ldiskfs if "ea_inode" feature is not enabled on MDT
			\|-
			! scope="row" style="text-align:left;"\|Maximum stripe size
			\|style="text-align:center;"\|< 4GB
			\|style="text-align:center;"\|< 4GB
			\|
			\|-
			! scope="row" style="text-align:left;"\|Minimum stripe size
			\|style="text-align:center;"\|64KB
			\|style="text-align:center;"\|64KB
			\|
			\|-
			! scope="row" style="text-align:left;"\|Maximum object size
			\|style="text-align:center;"\|16TB
			\|style="text-align:center;"\|256TB
			\|
			\|-
			! scope="row" style="text-align:left;"\|Maximum file size
			\|style="text-align:center;"\|31.25PB
			\|style="text-align:center;"\|8EB*
			\|
			\|-
			! scope="row" style="text-align:left;"\|Maximum number of files or subdirectories in a single directory
			\|style="text-align:center;"\|10M for 48-byte filenames.<br/>5M for 128-byte filenames.
			\|style="text-align:center;"\|2<sup>48</sup>
			\|
			\|-
			! scope="row" style="text-align:left;"\|Maximum number of files in the file system
			\|style="text-align:center;"\|4 billion per MDT
			\|style="text-align:center;"\|256 trillion per MDT
			\|
			\|-
			! scope="row" style="text-align:left;"\|Maximum filename length
			\|style="text-align:center;"\|255 bytes
			\|style="text-align:center;"\|255 bytes
			\|
			\|-
			! scope="row" style="text-align:left;"\|Maximum pathname length
			\|style="text-align:center;"\|4096 bytes
			\|style="text-align:center;"\|4096 bytes
			\|style="text-align:center;"\|Limited by Linux VFS
			\|}
			<nowiki>*</nowiki> Theoretical value.

	== High Availability and Data Storage Reliability ==		== High Availability and Data Storage Reliability ==

← Older revision		Revision as of 22:07, 9 August 2017
Line 65:		Line 65:
	\|style="text-align:center;"\|31.25PB		\|style="text-align:center;"\|31.25PB
	\|style="text-align:center;"\|512PB*		\|style="text-align:center;"\|512PB*
	\|\|-		\|
			\|-
	! scope="row" style="text-align:left;"\|Maximum filesystem size		! scope="row" style="text-align:left;"\|Maximum filesystem size
	\|style="text-align:center;"\|512PB		\|style="text-align:center;"\|512PB

← Older revision		Revision as of 22:07, 9 August 2017
Line 64:		Line 64:
	! scope="row" style="text-align:left;"\|Maximum file size		! scope="row" style="text-align:left;"\|Maximum file size
	\|style="text-align:center;"\|31.25PB		\|style="text-align:center;"\|31.25PB
			\|style="text-align:center;"\|512PB*
			\|\|-
			! scope="row" style="text-align:left;"\|Maximum filesystem size
			\|style="text-align:center;"\|512PB
	\|style="text-align:center;"\|8EB*		\|style="text-align:center;"\|8EB*
	\|		\|

← Older revision		Revision as of 22:01, 9 August 2017
Line 32:		Line 32:

	LNet supports routing, which provides maximum flexibility for connecting different network topologies. LNet routing provides an efficient protocol for bridging different networks, or employing different fabric technologies, such as Intel® OPA and InfiniBand. In larger file systems, with very large client counts, LNet can also support multihoming to improve performance and reliability. For more details and guidance, see the document Configuring LNet Routers for File Systems based on Intel* EE for Lustre* Software.		LNet supports routing, which provides maximum flexibility for connecting different network topologies. LNet routing provides an efficient protocol for bridging different networks, or employing different fabric technologies, such as Intel® OPA and InfiniBand. In larger file systems, with very large client counts, LNet can also support multihoming to improve performance and reliability. For more details and guidance, see the document Configuring LNet Routers for File Systems based on Intel* EE for Lustre* Software.

			== Lustre Scalability ==

			{\| class="wikitable"
			\|-
			! style="width:25%;"\|
			! style="width:25%;"\|Value using LDISKFS as backend
			! style="width:25%;"\|Value using ZFS as backend
			! style="width:25%;"\|Notes
			\|-
			! scope="row" style="text-align:left;"\|Maximum stripe count
			\|style="text-align:center;"\|2000
			\|style="text-align:center;"\|2000
			\|style="text-align:center;"\| Limit is 160 for ldiskfs if "ea_inode" feature is not enabled on MDT
			\|-
			! scope="row" style="text-align:left;"\|Maximum stripe size
			\|style="text-align:center;"\|< 4GB
			\|style="text-align:center;"\|< 4GB
			\|
			\|-
			! scope="row" style="text-align:left;"\|Minimum stripe size
			\|style="text-align:center;"\|64KB
			\|style="text-align:center;"\|64KB
			\|
			\|-
			! scope="row" style="text-align:left;"\|Maximum object size
			\|style="text-align:center;"\|16TB
			\|style="text-align:center;"\|256TB
			\|
			\|-
			! scope="row" style="text-align:left;"\|Maximum file size
			\|style="text-align:center;"\|31.25PB
			\|style="text-align:center;"\|8EB*
			\|
			\|-
			! scope="row" style="text-align:left;"\|Maximum number of files or subdirectories in a single directory
			\|style="text-align:center;"\|10M for 48-byte filenames.<br/>5M for 128-byte filenames.
			\|style="text-align:center;"\|2<sup>48</sup>
			\|
			\|-
			! scope="row" style="text-align:left;"\|Maximum number of files in the file system
			\|style="text-align:center;"\|4 billion per MDT
			\|style="text-align:center;"\|256 trillion per MDT
			\|
			\|-
			! scope="row" style="text-align:left;"\|Maximum filename length
			\|style="text-align:center;"\|255 bytes
			\|style="text-align:center;"\|255 bytes
			\|
			\|-
			! scope="row" style="text-align:left;"\|Maximum pathname length
			\|style="text-align:center;"\|4096 bytes
			\|style="text-align:center;"\|4096 bytes
			\|style="text-align:center;"\|Limited by Linux VFS
			\|}
			<nowiki>*</nowiki> Theoretical value.

	== High Availability and Data Storage Reliability ==		== High Availability and Data Storage Reliability ==

← Older revision		Revision as of 08:12, 11 May 2026
Line 1:		Line 1:
	== Overview ==		== Overview ==
	Lustre* is an open-source, global single-namespace, POSIX-compliant, distributed parallel file system designed for scalability, high-performance, and high-availability. Lustre runs on Linux-based operating systems and employs a client-server network architecture. Storage is provided by a set of servers that can scale to populations measuring up to several hundred hosts. Lustre servers for a single file system instance can, in aggregate, present up to hundreds of petabytes of storage to thousands of compute clients, with multiple terabytes per second of combined throughput.		Lustre* is an open-source, global single-namespace, POSIX-compliant, distributed parallel file system designed for scalability, high-performance, and high-availability. Lustre runs on Linux-based operating systems and employs a client-server network architecture. Storage is provided by a set of servers that can scale to populations measuring up to several hundred hosts. Lustre servers for a single file system instance can, in aggregate, present up to hundreds of petabytes of storage to thousands of compute clients, with multiple terabytes per second of combined throughput.

	Lustre is a file system that scales to meet the requirements of applications running on a range of systems from small-scale HPC environments up to the very largest supercomputers and has been created using object-based storage building blocks to maximize scalability.		Lustre is a file system that scales to meet the requirements of applications running on a range of systems from small-scale HPC environments up to the very largest supercomputers and has been created using object-based storage building blocks to maximize scalability.
Line 32:		Line 32:
	Persistent configuration information for all Lustre nodes is recorded by the MGS onto a storage device called a Management Target (MGT). The MGS can be paired with an MDS in a high-availability configuration, with each server connected to shared storage. Multiple Lustre file systems can be managed by a single MGS, but the low resource requirements for the MGS/MGT usually mean each filesystem will have its own MGS/MGT.		Persistent configuration information for all Lustre nodes is recorded by the MGS onto a storage device called a Management Target (MGT). The MGS can be paired with an MDS in a high-availability configuration, with each server connected to shared storage. Multiple Lustre file systems can be managed by a single MGS, but the low resource requirements for the MGS/MGT usually mean each filesystem will have its own MGS/MGT.

	'''Clients:''' Applications access and use file system data by interfacing with Lustre clients. A Lustre client is represented as a file system mount point on a host and presents applications with a unified namespace for all of the files and data in the file system, using standard POSIX semantics. A Lustre file system mounted on the client operating system looks much like any other POSIX file system; each Lustre instance is presented as a separate mount point on the client’s operating system, and each client can mount several different Lustre file system instances concurrently.		'''Clients:''' Applications access and use file system data by interfacing with Lustre clients. A Lustre client is represented as a file system mount point on a host and presents applications with a unified namespace for all of the files and data in the file system, using standard POSIX semantics. A Lustre file system mounted on the client operating system looks much like any other POSIX file system; each Lustre instance is presented as a separate mount point on the client’s operating system, and each client can mount several different Lustre file system instances concurrently. Lustre uses a cache-coherent distributed lock manager to control file I/O, ensuring that all clients can access files in parallel with both read and write concurrency. Clients communicate with the OSS servers directly for data I/O; the MDS does not participate in file I/O once the file is opened. The full POSIX test suite passes on Lustre clients in an identical manner to a local ext4 file system, with limited exceptions. Lustre also supports mmap() file I/O.

	'''Lustre Networking (LNet):''' LNet is the high-speed data network protocol that clients use to access the file system. LNet is designed to meet the needs of large-scale compute clusters and is optimized for very large node counts and high throughput. LNet supports Ethernet, InfiniBand, Intel® Omni-Path Architecture (OPA), and specific compute fabrics such as as Cray Gemini.		'''Lustre Networking (LNet):''' LNet is the high-speed data network protocol that clients use to access the file system. LNet is designed to meet the needs of large-scale compute clusters and is optimized for very large node counts and high throughput. LNet supports Ethernet, InfiniBand, Intel® Omni-Path Architecture (OPA), and specific compute fabrics such as as Cray Gemini.

	LNet abstracts network details from the file system itself. LNet allows for full RDMA throughput and zero copy communications when available.		LNet abstracts network details from the file system itself. LNet allows for full RDMA throughput and zero copy communications when available.

	LNet supports routing, which provides maximum flexibility for connecting different network topologies. LNet routing provides an efficient protocol for bridging different networks, or employing different fabric technologies, such as Ethernet, InfiniBand, and OPA. To improve performance and reliability, LNet can also balance traffic across multiple network interfaces in either/both the client and server nodes, in a Multi-Rail configuration. For more details and guidance, see the document Configuring LNet Routers for File Systems.		LNet supports routing, which provides maximum flexibility for connecting different network topologies. LNet routing provides an efficient protocol for bridging different networks, or employing different fabric technologies, such as Ethernet, InfiniBand, and OPA. To improve performance and reliability, LNet can also balance traffic across multiple network interfaces in either/both the client and server nodes, in a Multi-Rail configuration. For more details and guidance, see the document Configuring LNet Routers for File Systems.
Line 119:		Line 119:
	The OpenZFS file system reduces the administrative complexity of maintaining software-based storage by taking a holistic view of both the file system and storage management. ZFS integrates volume management features with an advanced file system that scales efficiently and provides enhancements including end-to-end checksums for protection against data corruption, versatility in storage configuration, online data integrity verification, and a copy-on-write architecture that eliminates the need to perform offline repairs. There is no fsck in ZFS.		The OpenZFS file system reduces the administrative complexity of maintaining software-based storage by taking a holistic view of both the file system and storage management. ZFS integrates volume management features with an advanced file system that scales efficiently and provides enhancements including end-to-end checksums for protection against data corruption, versatility in storage configuration, online data integrity verification, and a copy-on-write architecture that eliminates the need to perform offline repairs. There is no fsck in ZFS.

	Advances in software-based storage architectures are also influencing storage hardware design, creating hybrid server and storage enclosures that combine storage trays with standard servers into a single high-density chassis. These integrated systems can offer higher density per rack and less complex physical integration (including reduced cabling).		Advances in software-based storage architectures are also influencing storage hardware design, creating hybrid server and storage enclosures that combine storage trays with standard servers into a single high-density chassis. These integrated systems can offer higher density per rack and less complex physical integration (including reduced cabling).

	=== High Availability for Lustre Service Continuity ===		=== High Availability for Lustre Service Continuity ===

	Lustre servers are responsible for transacting I/O requests from applications running on a network of computers, and for managing the block storage used to maintain a persistent record of the data. Lustre clients do not have direct connections to the block storage and are often completely diskless, with no local data persistence. Because data is not replicated between Lustre servers, loss of access to a server means loss of access to the data managed by that server, which in turn means that a subset of the data managed by the Lustre file system will not be available to clients.		Lustre servers are responsible for transacting I/O requests from applications running on a network of computers, and for managing the block storage used to maintain a persistent record of the data. Lustre clients do not have direct connections to the block storage and are often completely diskless, with no local data persistence. Because data is not replicated between Lustre servers, loss of access to a server means loss of access to the data managed by that server, which in turn means that a subset of the data managed by the Lustre file system will not be available to clients.

	To protect against service failure, Lustre data is usually held on multi-ported, dedicated storage to which two or more servers are connected. The storage is subdivided into volumes or LUNs, with each LUN representing a Lustre storage target (MGT, MDT or OST). Each server that is attached to the enclosure has equal access to the storage targets and can be configured to present the storage targets to the network, although only one server is permitted to access an individual storage target in the enclosure at any given time. Lustre uses an inter-node failover model for maintaining service availability, meaning that if a server develops a fault, then any Lustre storage target managed by the failed server can be transferred to a surviving server that is connected to the same storage array.		To protect against service failure, Lustre data is usually held on multi-ported, dedicated storage to which two or more servers are connected. The storage is subdivided into volumes or LUNs, with each LUN representing a Lustre storage target (MGT, MDT or OST). Each server that is attached to the enclosure has equal access to the storage targets and can be configured to present the storage targets to the network, although only one server is permitted to access an individual storage target in the enclosure at any given time. Lustre uses an inter-node failover model for maintaining service availability, meaning that if a server develops a fault, then any Lustre storage target managed by the failed server can be transferred to a surviving server that is connected to the same storage array.