Understanding Lustre Internals - Revision history

Hornc: Moved source code 20 block to appropriate section

2025-03-04T16:43:12Z

Moved source code 20 block to appropriate section

← Older revision		Revision as of 09:43, 4 March 2025
Line 1,177:		Line 1,177:

	Current version of Lustre supports only file name encryption whereas in future Lustre plans to extend the encryption capability for file contents as well.		Current version of Lustre supports only file name encryption whereas in future Lustre plans to extend the encryption capability for file contents as well.

	~~=== CPU Partition Table Management ===~~

	Libcfs includes APIs and data structures that help with CPU partition table management in Lustre. A CPU partition is a virtual processing unit and can have 1-N cores or 1-N NUMA nodes. Therefore a CPU partition is also viewed as a pool of processors.

	<span id="code:kernel_encr_apis" label="code:kernel_encr_apis">'''Source code 20: Kernel APIs for data encryption defined in libcfs/include/libcfs/crypto/llcrypt.h'''</span>		<span id="code:kernel_encr_apis" label="code:kernel_encr_apis">'''Source code 20: Kernel APIs for data encryption defined in libcfs/include/libcfs/crypto/llcrypt.h'''</span>
Line 1,210:		Line 1,206:
	struct llcrypt_str *disk_link)		struct llcrypt_str *disk_link)
	</syntaxhighlight>		</syntaxhighlight>

			=== CPU Partition Table Management ===

			Libcfs includes APIs and data structures that help with CPU partition table management in Lustre. A CPU partition is a virtual processing unit and can have 1-N cores or 1-N NUMA nodes. Therefore a CPU partition is also viewed as a pool of processors.

	A CPU partition table (CPT) consists of a set of CPU partitions. CPTs can have two modes of operation, NUMA and SMP denoted by <code>CFS_CPU_MODE_NUMA</code> and <code>CFS_CPU_MODE_SMP</code> respectively. Users can specify total number of CPU partitions while creating a CPT and ID of a CPU partition always starts from 0. For example: if there are 8 cores in the system, creating a CPT:		A CPU partition table (CPT) consists of a set of CPU partitions. CPTs can have two modes of operation, NUMA and SMP denoted by <code>CFS_CPU_MODE_NUMA</code> and <code>CFS_CPU_MODE_SMP</code> respectively. Users can specify total number of CPU partitions while creating a CPT and ID of a CPU partition always starts from 0. For example: if there are 8 cores in the system, creating a CPT:

Adilger: /* Lustre Features */ ORNL Orion is 690PB

2024-09-23T14:13:32Z

Lustre Features: ORNL Orion is 690PB

← Older revision		Revision as of 07:13, 23 September 2024
Line 83:		Line 83:
	\|''Single'' ''File'' ''(max size)'': 32 PiB (ldiskfs) or 2<sup>63</sup> bytes (ZFS)		\|''Single'' ''File'' ''(max size)'': 32 PiB (ldiskfs) or 2<sup>63</sup> bytes (ZFS)
	<br />''Aggregate'': 512 PiB total capacity, 1 trillion files		<br />''Aggregate'': 512 PiB total capacity, 1 trillion files
	\|''Single'' ''File'' ''(max size)'': multi-TiB<br />''Aggregate'': 55 PiB capacity, 8 billion files		\|''Single'' ''File'' ''(max size)'': multi-TiB<br />''Aggregate'': 690 PiB capacity, 20 billion files
	\|-		\|-
	\|}		\|}

Adilger: update performance results based on modern hardware

2024-09-22T21:41:22Z

update performance results based on modern hardware

← Older revision		Revision as of 14:41, 22 September 2024
Line 50:		Line 50:
	\|-		\|-
	\|Client Performance		\|Client Performance
	\|''Single client'': 90% of network bandwidth<br />''Aggregate'': 10 TB/s		\|''Single client'': 90% of network bandwidth
	\|''Single client'': ~~4.5~~ GB/s (~~FDR~~ IB~~, OPA1~~), ~~1000~~ metadata ops/sec<br />''Aggregate'': 2.5 TB/s		<br />''Aggregate'': 10 TB/s
			\|''Single client'': 20 GB/s (Multiple EDR IB), 50000 metadata ops/sec
			<br />''Aggregate'': 2.5 TB/s
	\|-		\|-
	\|OSS Scalability		\|OSS Scalability
	\|''Single OSS'': 1-32 OSTs per OSS<br />''Single OST (ldiskfs)'': ~~300M~~ objects, ~~256TiB~~ per OST<br />''Single OST (ZFS)'': 500M objects, 256TiB per OST<br />''OSS count'': 1000 OSSs w/ up to 4000 OSTs		\|''Single OSS'': 1-32 OSTs per OSS
	\|''Single OSS (ldiskfs)'': ~~32x8TiB OSTs per OSS, 8x32TiB~~ OSTs per OSS<br />''Single OSS (ZFS)'': ~~1x72TiB~~ OST per OSS<br />''OSS count'': 450 OSSs w/ ~~1000 4TiB~~ OSTs, 192 OSSs w/ 1344 ~~8TiB~~ OSTs, 768 OSSs w/ 768 ~~72TiB~~ OSTs		<br />''Single OST (ldiskfs)'': 1B objects, 1024TiB per OST
			<br />''Single OST (ZFS)'': 500M objects, 256TiB per OST
			<br />''OSS count'': 1000 OSSs w/ up to 4000 OSTs
			\|''Single OSS (ldiskfs)'': 2x1024TiB OSTs per OSS
			<br />''Single OSS (ZFS)'': 1x1024TiB OST per OSS
			<br />''OSS count'': 450 OSSs w/ 900 OSTs, 192 OSSs w/ 1344 OSTs, 768 OSSs w/ 768 OSTs
	\|-		\|-
	\|OSS Performance		\|OSS Performance
	\|''Single OSS'': 15 GB/s<br />''Aggregate'': 10 TB/s		\|''Single OSS'': 65 GB/s write/90 GB/s read
	~~\|''Single OSS'': 10 GB/s~~<br />''~~Aggregate~~'': ~~2.5 TB/s~~		<br />''Aggregate'': 10 TB/s
			<br />''IOPS'': 500K write, 3M read
	\|-		\|-
	\|MDS Scalability		\|MDS Scalability
	\|''Single MDS'': 1-4 MDTs per MDS<br />''Single MDT (ldiskfs)'': 4 billion files, 8 TiB per MDT<br />''Single MDT (ZFS)'': 64 billion files, 64 TiB per MDT<br />''MDS count'': 256 MDSs w/ up to 265 MDTs		\|''Single MDS'': 1-4 MDTs per MDS
	\|''Single MDS'': 3 billion files<br />''MDS count'': 7 MDSs w/ ~~7x 2~~ TiB MDTs in production (256 MDSs w/ 256 64 GiB MDTs in testing)		<br />''Single MDT (ldiskfs)'': 4 billion files, 10 TiB per MDT
			<br />''Single MDT (ZFS)'': 64 billion files, 64 TiB per MDT
			<br />''MDS count'': 256 MDSs w/ up to 265 MDTs
			\|''Single MDS'': 4 billion files
			<br />''MDS count'': 40 MDSs w/ 40x 8 TiB MDTs in production (256 MDSs w/ 256 64 GiB MDTs in testing)
	\|-		\|-
	\|MDS Performance		\|MDS Performance
	\|~~50,000~~ create ops/sec~~,<br />200,000 metadata stat ops~~/~~sec~~		\|''Single MDS'': 300k create ops/sec / MDS<br />2 M metadata stat ops/sec
	~~\|15,000 create ops/sec,~~<br />~~50,000~~ metadata stat ops/sec
	\|-		\|-
	\|File system Scalability		\|File system Scalability
	\|''Single'' ''File'' ''(max size)'': 32 PiB (ldiskfs) or 2<sup>63</sup> bytes (ZFS)<br />''Aggregate'': 512 PiB total capacity, 1 trillion files		\|''Single'' ''File'' ''(max size)'': 32 PiB (ldiskfs) or 2<sup>63</sup> bytes (ZFS)
			<br />''Aggregate'': 512 PiB total capacity, 1 trillion files
	\|''Single'' ''File'' ''(max size)'': multi-TiB<br />''Aggregate'': 55 PiB capacity, 8 billion files		\|''Single'' ''File'' ''(max size)'': multi-TiB<br />''Aggregate'': 55 PiB capacity, 8 billion files
	\|-		\|-

AnjusG: /* Authors */

2023-06-05T12:00:26Z

Authors

← Older revision		Revision as of 05:00, 5 June 2023
Line 1,448:		Line 1,448:

	== Authors ==		== Authors ==
	Anjus George, ORNL		* Anjus George, ORNL
	Rick Mohr, ORNL		* Rick Mohr, ORNL

	== Notes ==		== Notes ==

	<references />		<references />

AnjusG: /* Presentations */

2023-06-05T11:59:57Z

Presentations

← Older revision		Revision as of 04:59, 5 June 2023
Line 1,446:		Line 1,446:
	* [[Media:Anjus_George_LUG2022_tutorial.pdf\|Lustre MGC and Obdclass Deep Dive]]([https://vimeo.com/709544388#t=3m52s video]) - Lustre User Group (LUG 2022) tutorial		* [[Media:Anjus_George_LUG2022_tutorial.pdf\|Lustre MGC and Obdclass Deep Dive]]([https://vimeo.com/709544388#t=3m52s video]) - Lustre User Group (LUG 2022) tutorial
	* [[Media:LUG2022-Understanding_Lustre_Internals-George.pdf\|Understanding Lustre File System Internals – A Documentation Initiative]] ([https://www.youtube.com/watch?v=OQuCG1Vjs0s video]) - Lustre User Group (LUG 2022) technical talk		* [[Media:LUG2022-Understanding_Lustre_Internals-George.pdf\|Understanding Lustre File System Internals – A Documentation Initiative]] ([https://www.youtube.com/watch?v=OQuCG1Vjs0s video]) - Lustre User Group (LUG 2022) technical talk

			== Authors ==
			Anjus George, ORNL
			Rick Mohr, ORNL

	== Notes ==		== Notes ==

	<references />		<references />

AnjusG: /* Presentations */

2022-10-07T04:44:47Z

Presentations

← Older revision		Revision as of 21:44, 6 October 2022
Line 1,445:		Line 1,445:
	== Presentations ==		== Presentations ==
	* [[Media:Anjus_George_LUG2022_tutorial.pdf\|Lustre MGC and Obdclass Deep Dive]]([https://vimeo.com/709544388#t=3m52s video]) - Lustre User Group (LUG 2022) tutorial		* [[Media:Anjus_George_LUG2022_tutorial.pdf\|Lustre MGC and Obdclass Deep Dive]]([https://vimeo.com/709544388#t=3m52s video]) - Lustre User Group (LUG 2022) tutorial
	* [[Media:LUG2022-Understanding_Lustre_Internals-George.pdf\|Understanding Lustre File System Internals – A Documentation Initiative]]~~'''~~ ([https://www.youtube.com/watch?v=OQuCG1Vjs0s video]) - Lustre User Group (LUG 2022) technical talk		* [[Media:LUG2022-Understanding_Lustre_Internals-George.pdf\|Understanding Lustre File System Internals – A Documentation Initiative]] ([https://www.youtube.com/watch?v=OQuCG1Vjs0s video]) - Lustre User Group (LUG 2022) technical talk

	== Notes ==		== Notes ==

	<references />		<references />

AnjusG: /* Presentations */

2022-10-07T04:44:30Z

Presentations

← Older revision		Revision as of 21:44, 6 October 2022
Line 1,444:		Line 1,444:

	== Presentations ==		== Presentations ==
	* [[Media:Anjus_George_LUG2022_tutorial.pdf\|Lustre MGC and Obdclass Deep Dive]]([https://vimeo.com/709544388#t=3m52s video])		* [[Media:Anjus_George_LUG2022_tutorial.pdf\|Lustre MGC and Obdclass Deep Dive]]([https://vimeo.com/709544388#t=3m52s video]) - Lustre User Group (LUG 2022) tutorial
	* [[Media:LUG2022-Understanding_Lustre_Internals-George.pdf\|Understanding Lustre File System Internals – A Documentation Initiative]]''' ([https://www.youtube.com/watch?v=OQuCG1Vjs0s video])		* [[Media:LUG2022-Understanding_Lustre_Internals-George.pdf\|Understanding Lustre File System Internals – A Documentation Initiative]]''' ([https://www.youtube.com/watch?v=OQuCG1Vjs0s video]) - Lustre User Group (LUG 2022) technical talk

	== Notes ==		== Notes ==

	<references />		<references />

AnjusG: /* Presentations */

2022-10-07T04:43:31Z

Presentations

← Older revision		Revision as of 21:43, 6 October 2022
Line 1,444:		Line 1,444:

	== Presentations ==		== Presentations ==
	[[Media:Anjus_George_LUG2022_tutorial.pdf\|Lustre MGC and Obdclass Deep Dive]]([https://vimeo.com/709544388#t=3m52s video])		* [[Media:Anjus_George_LUG2022_tutorial.pdf\|Lustre MGC and Obdclass Deep Dive]]([https://vimeo.com/709544388#t=3m52s video])
	[[Media:LUG2022-Understanding_Lustre_Internals-George.pdf\|Understanding Lustre File System Internals – A Documentation Initiative]]''' ([https://www.youtube.com/watch?v=OQuCG1Vjs0s video])		* [[Media:LUG2022-Understanding_Lustre_Internals-George.pdf\|Understanding Lustre File System Internals – A Documentation Initiative]]''' ([https://www.youtube.com/watch?v=OQuCG1Vjs0s video])

	== Notes ==		== Notes ==

	<references />		<references />

AnjusG: /* Publications */

2022-10-07T04:42:40Z

Publications

← Older revision		Revision as of 21:42, 6 October 2022
Line 1,442:		Line 1,442:

	An initial version of this documentation is published as a Technical Report in osti.gov. The technical report can be found here: [https://info.ornl.gov/sites/publications/Files/Pub166872.pdf Understanding Lustre Internals, Second Edition]. Please [https://www.osti.gov/biblio/1824954-understanding-lustre-internals-second-edition cite this documentation] if it helps your work on Lustre file systems.		An initial version of this documentation is published as a Technical Report in osti.gov. The technical report can be found here: [https://info.ornl.gov/sites/publications/Files/Pub166872.pdf Understanding Lustre Internals, Second Edition]. Please [https://www.osti.gov/biblio/1824954-understanding-lustre-internals-second-edition cite this documentation] if it helps your work on Lustre file systems.

			== Presentations ==
			[[Media:Anjus_George_LUG2022_tutorial.pdf\|Lustre MGC and Obdclass Deep Dive]]([https://vimeo.com/709544388#t=3m52s video])
			[[Media:LUG2022-Understanding_Lustre_Internals-George.pdf\|Understanding Lustre File System Internals – A Documentation Initiative]]''' ([https://www.youtube.com/watch?v=OQuCG1Vjs0s video])

	== Notes ==		== Notes ==

	<references />		<references />

AnjusG: /* Publications */

2022-08-09T02:48:12Z

Publications

← Older revision		Revision as of 19:48, 8 August 2022
Line 1,441:		Line 1,441:
	== Publications ==		== Publications ==

	An initial version of this documentation is published as a Technical Report in osti.gov. The technical report can be found here: [https://info.ornl.gov/sites/publications/Files/Pub166872.pdf Understanding Lustre Internals, Second Edition]		An initial version of this documentation is published as a Technical Report in osti.gov. The technical report can be found here: [https://info.ornl.gov/sites/publications/Files/Pub166872.pdf Understanding Lustre Internals, Second Edition]. Please [https://www.osti.gov/biblio/1824954-understanding-lustre-internals-second-edition cite this documentation] if it helps your work on Lustre file systems.

	== Notes ==		== Notes ==

	<references />		<references />