Configuring Lustre File Striping - Revision history

Adilger: /* Displaying Layout Information for a File */ improve description

2019-02-06T20:28:06Z

Displaying Layout Information for a File: improve description

← Older revision		Revision as of 13:28, 6 February 2019
Line 64:		Line 64:
	== Displaying Layout Information for a File ==		== Displaying Layout Information for a File ==

	The '''lfs getstripe''' command can be used to display information that shows which specific OSTs a file is distributed over. For example, the output from the following command indicates that the file ''test2'' is striped over all six active OSTs in the filesystem, both because of the '''lmm_stripe_count:''' line, and because it shows 6 separate '''l_fid:''' objects allocated for the file ('''-y''' formats the output nicely in YAML format):		The '''lfs getstripe''' command can be used to display information that shows which specific OSTs a file is distributed over. For example, the output from the following command indicates that the file ''test2'' is striped over all six active OSTs in the filesystem, both because of the '''lmm_stripe_count:''' line, and because it shows 6 separate '''l_fid:''' objects allocated for the file starting at OST0002 because the '''test1''' file had just allocated objects on OST0000 and OST0001 (the '''-y''' option formats the output nicely in YAML format):

	<pre>		<pre>
Line 88:		Line 88:
	</pre>		</pre>

	In contrast, the output from the following command, which shows a '''lmm_stripe_count:''' of two and lists only two '''l_fid''' entries, indicates that the file ''test1'' is stored on two OSTs:		In contrast, the output from the following command, which shows a '''lmm_stripe_count:''' of two and lists only two '''l_fid''' entries, indicates that the file ''test1'' is stored on two OSTs, namely OST0000 and OST0001:

	<pre>		<pre>

Adilger: /* Setting Up Striping */ improve example a bit

2019-02-06T20:23:22Z

Setting Up Striping: improve example a bit

← Older revision		Revision as of 13:23, 6 February 2019
Line 22:		Line 22:
	root@client# lfs setstripe -S 4M -c 2 /mnt/testfs		root@client# lfs setstripe -S 4M -c 2 /mnt/testfs

	If a new file is created with these filesystem default settings, the following results are seen:		If a new 2000MB file is created with these filesystem default settings, the following results are seen:

	<pre>		<pre>
Line 36:		Line 36:
	testfs-OST0005_UUID 2.0G 83.3M 1.8G 4% /mnt/testfs[OST:5]		testfs-OST0005_UUID 2.0G 83.3M 1.8G 4% /mnt/testfs[OST:5]

	filesystem_summary: 11.8G 2.6G 8.6G 20% /mnt/testfs		filesystem_summary: 11.8G 2.5G 8.8G 20% /mnt/testfs
	</pre>		</pre>

	In this example, the entire file was written to two OSTs with no usage of the other four OSTs. ~~This~~ is ~~fine~~, since other files will be created on those other OSTs. Note that the layout ('''stripe_count''', '''stripe_size''', OSTs) of a file is fixed at the ~~time that a~~ file is first created. To change the layout of a file after it is created, the '''lfs migrate''' command (which takes the same parameters as '''lfs setstripe''') is needed to move the file data to different OSTs.		In this example, the entire file was written to the first two OSTs (1000MB per OST) with no usage of the other four OSTs. That only two OSTs are used is expected/requested, since other files will be created on those other OSTs to balance space and bandwidth usage. Note that the layout ('''stripe_count''', '''stripe_size''', OSTs) of a file is fixed when the file is first '''created''' (opened). To change the layout of a file after it is created, the '''lfs migrate''' command (which takes the same parameters as '''lfs setstripe''') is needed to move the file data to different OSTs.

	Continuing with this example, if a new file is created with an explicit '''stripe_count''' of '''-1''' to specify striping over all available OSTs instead of using the filesystem default:		Continuing with this example, if a new 1000MB file is created with an explicit '''stripe_count''' of '''-1''' to specify striping over all available OSTs instead of using the filesystem default:

	root@client# lfs setstripe -c -1 /mnt/testfs/test2		root@client# lfs setstripe -c -1 /mnt/testfs/test2

	Now, when a file is written, the new stripe setting evenly distributes about 160MB of ~~new~~ data over each the available OSTs. Using a widely-striped file is good if the file is very large, or a lot of clients will be accessing it concurrently, but typically it is best to have individual files striped over only 1 or 2 OSTs for minimal overhead, and let multiple processes creating separate files handle the parallelism across different OSTs.		Now, when this file is written, the new stripe setting evenly distributes about 160MB of the filek data over each the available OSTs. Using a widely-striped file is good if the file is very large, or a lot of clients will be accessing it concurrently, but typically it is best to have individual files striped over only 1 or 2 OSTs for minimal overhead, and let multiple processes creating separate files handle the parallelism across different OSTs.

	<pre>		<pre>
Line 59:		Line 59:
	testfs-OST0005_UUID 2.0G 247.3M 1.6G 12% /mnt/testfs[OST:5]		testfs-OST0005_UUID 2.0G 247.3M 1.6G 12% /mnt/testfs[OST:5]

	filesystem_summary: 11.8G 4.1G 9.7G 12% /mnt/testfs		filesystem_summary: 11.8G 3.5G 7.7G 12% /mnt/testfs
	</pre>		</pre>

Adilger: /* Setting Up Striping */ fix stripe size example to use -s

2019-02-05T09:58:53Z

Setting Up Striping: fix stripe size example to use -s

← Older revision		Revision as of 02:58, 5 February 2019
Line 20:		Line 20:
	The command to set the above default layout on the file system looked like this:		The command to set the above default layout on the file system looked like this:

	root@client# lfs setstripe -s 4M -c 2 /mnt/testfs		root@client# lfs setstripe -S 4M -c 2 /mnt/testfs

	If a new file is created with these filesystem default settings, the following results are seen:		If a new file is created with these filesystem default settings, the following results are seen:

Adilger: /* Setting Up Striping */ minor updates.

2019-02-05T09:57:42Z

Setting Up Striping: minor updates.

← Older revision		Revision as of 02:57, 5 February 2019
Line 7:		Line 7:
	== Setting Up Striping ==		== Setting Up Striping ==

	To see the layout of a particular file, or the default layout to be used for new files in a particular directory, use the command '''lfs getstripe {file\|directory\|~~filesystem~~}'''. If run against the filesystem root directory, it will show the default layout for all files created in the filesystem that do not otherwise specify a layout at creation time or inherit it from a layout on the parent directory. For example, running the command on the filesystem root directory shows the following (the '''-d''' option limits the output to the specified directory):		To see the layout of a particular file, or the default layout to be used for new files in a particular directory, use the command '''lfs getstripe {file\|directory\|root}'''. If run against the filesystem root directory, it will show the default layout for all files created in the filesystem that do not otherwise specify a layout at creation time or inherit it from a layout on the parent directory. For example, running the command on the filesystem root directory shows the following (the '''-d''' option limits the output to the specified directory):

	<pre>		<pre>
Line 14:		Line 14:
	</pre>		</pre>

	In this example, the default '''stripe_count''' is 2 (that is, data blocks are striped alternately over two OSTs), the default '''stripe_size''' is 4 MB (that is, each OST reads or writes 4MB of data for the first file stripe before going to the second file stripe), and files do not start on a specific OST index (that is, the MDS will balance new file creations across all OSTs in the filesystem for maximum performance).		In this example, the default '''stripe_count''' is 2 (that is, data blocks are striped alternately over two OSTs), the default '''stripe_size''' is 4 MB (that is, each OST reads or writes 4MB of data for the first file stripe before going to the second file stripe on the next OST), and files do not start on a specific OST index (that is, the MDS will balance new file creations across all OSTs in the filesystem for maximum performance).

			Note that the '''stripe_size''' does '''not''' affect the allocation size of the file on disk (which is controlled by the underlying OST filesystem, typically 4KB for ldiskfs), but only the distribution of file data across OSTs.

	The command to set the above default layout on the file system looked like this:		The command to set the above default layout on the file system looked like this:
Line 37:		Line 39:
	</pre>		</pre>

	In this example, the entire file was written to two OSTs with no usage of the other four OSTs. This is fine, since other files will be created on those other OSTs. Note that the layout (stripe_count, stripe_size, OSTs) of a file is fixed at the time that a file is first created. To change the layout of a file after it is created, the '''lfs migrate''' command (which takes the same parameters as '''lfs setstripe''') is needed to move the file data to different OSTs.		In this example, the entire file was written to two OSTs with no usage of the other four OSTs. This is fine, since other files will be created on those other OSTs. Note that the layout ('''stripe_count''', '''stripe_size''', OSTs) of a file is fixed at the time that a file is first created. To change the layout of a file after it is created, the '''lfs migrate''' command (which takes the same parameters as '''lfs setstripe''') is needed to move the file data to different OSTs.

	Continuing with this example, if a new file is created with an explicit '''stripe_count''' of '''-1''' to specify striping over all available OSTs instead of using the filesystem default:		Continuing with this example, if a new file is created with an explicit '''stripe_count''' of '''-1''' to specify striping over all available OSTs instead of using the filesystem default:

Adilger: improve and modernize the examples on this page, fix links to Lustre manual

2018-12-14T00:35:14Z

improve and modernize the examples on this page, fix links to Lustre manual

Show changes

KenRawlings: adding lustre striping guide from nics

2016-06-02T14:50:23Z

adding lustre striping guide from nics

← Older revision		Revision as of 07:50, 2 June 2016
Line 3:		Line 3:
	!Note: This page originated on the old Lustre wiki. It was identified as likely having value and was migrated to the new wiki. It is in the process of being reviewed/updated and may currently have content that is out of date.		!Note: This page originated on the old Lustre wiki. It was identified as likely having value and was migrated to the new wiki. It is in the process of being reviewed/updated and may currently have content that is out of date.
	\|}		\|}

			* [https://www.nics.tennessee.edu/computing-resources/file-systems/lustre-striping-guide Lustre Striping Guide]

	One of the main factors leading to the high performance of Lustre™ file systems is the ability to stripe data over multiple OSTs. The stripe count can be set on a file system, directory, or file level. An example showing the use of striping is provided below.		One of the main factors leading to the high performance of Lustre™ file systems is the ability to stripe data over multiple OSTs. The stripe count can be set on a file system, directory, or file level. An example showing the use of striping is provided below.

KenRawlings: Initial Creation From Migration

2016-06-02T14:44:46Z

Initial Creation From Migration

New page

{| class='wikitable'
|-
!Note: This page originated on the old Lustre wiki. It was identified as likely having value and was migrated to the new wiki. It is in the process of being reviewed/updated and may currently have content that is out of date.
|}

One of the main factors leading to the high performance of Lustre™ file systems is the ability to stripe data over multiple OSTs. The stripe count can be set on a file system, directory, or file level. An example showing the use of striping is provided below.

For additional information, see [http://wiki.lustre.org/manual/LustreManual20_HTML/ManagingStripingFreeSpace.html#50438209_pgfId-5529 Chapter 18: ''Managing File Striping and Free Space''] in the [http://wiki.lustre.org/manual/LustreManual20_HTML/index.html ''Lustre Operations Manual'']

== Setting Up Striping ==

To see the current stripe size, use the command ''lfs getstripe [file, dir, fs]''. This command will produce output similar to the following:

<pre>
root@LustreClient01 lustre]# lfs getstripe /mnt/lustre
OBDS:
0: lustre-OST0000_UUID ACTIVE
1: lustre-OST0001_UUID ACTIVE
2: lustre-OST0002_UUID ACTIVE
3: lustre-OST0003_UUID ACTIVE
4: lustre-OST0004_UUID ACTIVE
5: lustre-OST0005_UUID ACTIVE
/mnt/lustre
(Default) stripe_count: 2 stripe_size: 4M stripe_offset: 0
</pre>

In this example, the default stripe count is 2 (that is, data blocks are striped over two OSTs), the default stripe size is 4 MB (the stripe size can be set in K, M or G), and all writes start from the first OST.

'''''Note:''''' When setting the stripe, the offset is set before the stripe count.

The command to set a new stripe pattern on the file system may look like this:

[root@LustreClient01 lustre]# lfs setstripe -s 4M -i 0 -c 1 /mnt/lustre

This example command sets the stripe of ''/mnt/lustre'' to 4 MB blocks starting at OST0 and spanning over one OST. If a new file is created with these settings, the following results are seen:

<pre>
[root@LustreClient01 lustre]# dd if=/dev/zero of=/mnt/lustre/test1 bs=10M count=100

root@LustreClient01 lustre]# lfs df -h
UUID bytes Used Available Use% Mounted on
lustre-MDT0000_UUID 4.4G 214.5M 3.9G 4% /mnt/lustre[MDT:0]
lustre-OST0000_UUID 2.0G 1.1G 830.1M 53% /mnt/lustre[OST:0]
lustre-OST0001_UUID 2.0G 83.3M 1.8G 4% /mnt/lustre[OST:1]
lustre-OST0002_UUID 2.0G 83.3M 1.8G 4% /mnt/lustre[OST:2]
lustre-OST0003_UUID 2.0G 83.3M 1.8G 4% /mnt/lustre[OST:3]
lustre-OST0004_UUID 2.0G 83.3M 1.8G 4% /mnt/lustre[OST:4]
lustre-OST0005_UUID 2.0G 83.3M 1.8G 4% /mnt/lustre[OST:5]

filesystem summary: 11.8G 1.5G 9.7G 12% /mnt/lustre
</pre>

In this example, the entire file was written to the first OST with a very uneven distribution of data blocks.

Continuing with this example, the file is removed and the stripe count is changed to a value of ''-1'' to specify striping over all available OSTs:

[root@LustreClient01 lustre]# lfs setstripe -s 4M -i 0 -c -1 /mnt/lustre

Now, when a file is created, the new stripe setting evenly distributes the data over all the available OSTs:

<pre>
[root@LustreClient01 lustre]# dd if=/dev/zero of=/mnt/lustre/test1 bs=10M count=100
100+0 records in
100+0 records out
1048576000 bytes (1.0 GB) copied, 20.2589 seconds, 51.8 MB/s

[root@LustreClient01 lustre]# lfs df -h
UUID bytes Used Available Use% Mounted on
lustre-MDT0000_UUID 4.4G 214.5M 3.9G 4% /mnt/lustre[MDT:0]
lustre-OST0000_UUID 2.0G 251.3M 1.6G 12% /mnt/lustre[OST:0]
lustre-OST0001_UUID 2.0G 251.3M 1.6G 12% /mnt/lustre[OST:1]
lustre-OST0002_UUID 2.0G 251.3M 1.6G 12% /mnt/lustre[OST:2]
lustre-OST0003_UUID 2.0G 251.3M 1.6G 12% /mnt/lustre[OST:3]
lustre-OST0004_UUID 2.0G 247.3M 1.6G 12% /mnt/lustre[OST:4]
lustre-OST0005_UUID 2.0G 247.3M 1.6G 12% /mnt/lustre[OST:5]

filesystem summary: 11.8G 1.5G 9.7G 12% /mnt/lustre
</pre>

== Displaying Stripe Information for a File ==

The ''lfs getstripe'' command can be used to display information that shows over which OSTs a file is distributed. For example, the output from the following command (showing multiple ''obdidx'' entries) indicates that the file ''test1'' is striped over all six active OSTs in the configuration:

<pre>
[root@LustreClient01 ~]# lfs getstripe /mnt/lustre/test1
OBDS:
0: lustre-OST0000_UUID ACTIVE
1: lustre-OST0001_UUID ACTIVE
2: lustre-OST0002_UUID ACTIVE
3: lustre-OST0003_UUID ACTIVE
4: lustre-OST0004_UUID ACTIVE
5: lustre-OST0005_UUID ACTIVE
/mnt/lustre/test1
obdidx objid objid group
0 8 0x8 0
1 4 0x4 0
2 5 0x5 0
3 5 0x5 0
4 4 0x4 0
5 2 0x2 0
</pre>

In contrast, the output from the following command, which lists just a single ''obdidx'' entry, indicates that the file ''test2'' is contained on a single OST:

<pre>
[root@LustreClient01 ~]# lfs getstripe /mnt/lustre/test_2
OBDS:
0: lustre-OST0000_UUID ACTIVE
1: lustre-OST0001_UUID ACTIVE
2: lustre-OST0002_UUID ACTIVE
3: lustre-OST0003_UUID ACTIVE
4: lustre-OST0004_UUID ACTIVE
5: lustre-OST0005_UUID ACTIVE
/mnt/lustre/test_2
obdidx objid objid group
2 8 0x8 0
</pre>