Handling Full OSTs - Revision history

Adilger: /* Disabling MDS Object Creation on OST */ add "-P" for permanent setting

2023-05-24T23:28:55Z

Disabling MDS Object Creation on OST: add "-P" for permanent setting

← Older revision		Revision as of 16:28, 24 May 2023
Line 50:		Line 50:
	</pre>		</pre>

	The MDS connections to OST0002 will no longer create objects there. This process should be repeated for other MDS nodes if ~~needed~~. If a new file is now written to the file system, the write will be successful as the stripes are allocated across the remaining active OSTs.		If an OST is going to be removed permanently, then use the '''lfs set_param -P ...''' to set the parameters permanently, otherwise the ```max_create_count``` setting will be reset if the MDS is restarted or unmounted.

			The MDS connections to OST0002 will no longer create objects there. This process should be repeated for other MDS nodes and MDTa if present. If a new file is now written to the file system, the write will be successful as the stripes are allocated across the remaining active OSTs.

	2. Once the OST is no longer full (e.g. objects deleted or migrated off the full OST), it should be enabled again:		2. Once the OST is no longer full (e.g. objects deleted or migrated off the full OST), it should be enabled again:
Line 59:		Line 61:
	</pre>		</pre>

	'''Note:''' for releases 2.10.6 and earlier, the '''create_count''' must also be set to a non-zero value:		'''Note:''' for releases 2.10.6 and earlier, the '''create_count''' must also be set to a non-zero value after '''max_create_count''' is restored:
	<pre>		<pre>
	[root@mds01 ~]# lctl set_param osp.testfs-OST0002-*.create_count=128		[root@mds01 ~]# lctl set_param osp.testfs-OST0002-*.create_count=128

Adilger: Mention that full OSTs will still be used for wide-striped files.

2023-03-09T19:27:32Z

Mention that full OSTs will still be used for wide-striped files.

← Older revision		Revision as of 12:27, 9 March 2023
Line 1:		Line 1:
	<small>''(Updated: Mar 2018)''</small>		<small>''(Updated: Mar 2018)''</small>
	__TOC__		__TOC__
	Sometimes the OSTs in a file system have unbalanced usage, either due to the addition of new OSTs, or because of user error such as explicitly specifying the same starting OST index (e.g. '''-i 0''') for a large number of files, or when creating a single large file on one OST. If an OST is full and an attempt is made to write more information to that OST (e.g. extending an existing file), an error may occur. The procedures below describe how to ~~handle~~ a full OST.		Sometimes the OSTs in a file system have unbalanced usage, either due to the addition of new OSTs, or because of user error such as explicitly specifying the same starting OST index (e.g. '''-i 0''') for a large number of files, or when creating a single large file on one OST. If an OST is full and an attempt is made to write more information to that OST (e.g. extending an existing file), an error may occur.

			If the filesystem usage is always growing and old files are not regularly deleted, it is recommended to plan tha additional OST capacity is added before the filesystem hits 80% usage, and at least 25% capacity is added each time, so that the MDS can effectively balance space usage before it becomes critically low on any OST.

			The MDS will automatically reduce allocations on OSTs that have less free space than other OSTs (free space difference over 17%, controlled by '''lod..qos_threshold_rr'''). In many cases, this will be sufficient to put more new files on OSTs with more free space. This will work well for files that are striped over a subset of OSTs (within a pool, if any), to allow the MDS to skip the OSTs that are low on space. However, if there is something that is ''forcing'' the MDS to continue to allocate objects on those OSTs (eg. using '''-c -1''' to stripe over all available OSTs), then the MDS will continue to use the more-full OSTs until they are almost totally full (up to 99.9% by default, controlled by '''osp..reserved_mb_low''').

			The procedures below describe options on how to manually reduce the usage of a full OST.

	== Checking File System Usage ==		== Checking File System Usage ==
Line 21:		Line 27:
	</pre>		</pre>

	In this case, ''~~OST:2~~'' is almost full and when an attempt is made to write additional ~~information~~ to the file system (with uniform striping over all the OSTs), the write command fails ~~as follows~~:		In this case, ''OST0002'' is almost full and when an attempt is made to write additional data to the file system (with uniform striping over all the OSTs), the write command fails after the OST0002 stripe uses the remaining 100MiB on that OST:

	<pre>		<pre>
Line 56:		Line 62:
	<pre>		<pre>
	[root@mds01 ~]# lctl set_param osp.testfs-OST0002-*.create_count=128		[root@mds01 ~]# lctl set_param osp.testfs-OST0002-*.create_count=128
	osp.testfs-OST0002-MDT0000.~~max_create_count~~=128		osp.testfs-OST0002-MDT0000.create_count=128
	osp.testfs-OST0002-MDT0001.~~max_create_count~~=128		osp.testfs-OST0002-MDT0001.create_count=128
	</pre>		</pre>

Adilger: update to use modern commands and output formats

2019-03-19T00:56:10Z

update to use modern commands and output formats

Show changes

Adilger: /* Taking a Full OST Offline */ update for 2.9+ process to handle full OSTs

2019-01-17T05:47:24Z

Taking a Full OST Offline: update for 2.9+ process to handle full OSTs

← Older revision		Revision as of 22:47, 16 January 2019
Line 36:		Line 36:
	</pre>		</pre>

	== ~~Taking a Full~~ OST ~~Offline~~ ==		== Disabling MDS Object Creation on OST ==

	To enable continued use of the file system, the full OST has to ~~be taken offline or, more specifically, rendered read-only using the ''lctl'' command~~. This is done on ~~the~~ MDS, since the ~~MSD~~ allocates ~~space~~ for ~~writing~~.		To enable continued use of the file system, the full OST has to have object creation disabled. This needs to be done on all MDS nodes, since the MDS allocates OST objects for new files.

	1. ''Log in to the MDS server:''		1. ''Log in to the MDS server:''

	<pre>		<pre>
	[root@LustreClient01 ~]# ssh root@~~192.168.0.10~~		[root@LustreClient01 ~]# ssh root@mds01
	root@~~192.168.0.10~~'s password:		root@mds01's password:
	Last login: Wed Nov 26 13:35:12 2008 from ~~192.168.0.6~~		Last login: Wed Nov 26 13:35:12 2008 from LustreClient01
	</pre>		</pre>

	2. ''Use the'' lctl dl ''command to ~~show~~ the ~~status of all file system components~~:''		2. ''Use the'' lctl set_param'' command to disable object creation on the OST:''

	<pre>		<pre>
	[root@~~mds~~ ~]# lctl dl		[root@mds01 ~]# lctl set_param osp.testfs-OST0002-*.max_create_count=0
	~~0 UP mgs MGS MGS 9~~		osp.testfs-OST0002-MDT0000.max_create_count=0
	~~1 UP mgc MGC192~~.~~168~~.0.~~10@tcp e384bb0e~~-~~680b-ce25-7bc9-81655dd1e813 5~~		osp.testfs-OST0002-MDT0001.max_create_count=0
	~~2 UP mdt MDS MDS_uuid 3~~
	~~3 UP lov lustre-mdtlov lustre-mdtlov_UUID 4~~
	~~4 UP mds lustre~~-MDT0000 ~~lustre-MDT0000_UUID 5~~
	~~5 UP osc lustre-OST0000-osc lustre-mdtlov_UUID 5~~
	~~6 UP osc lustre-OST0001-osc lustre-mdtlov_UUID 5~~
	~~7 UP osc lustre~~-OST0002-~~osc lustre-mdtlov_UUID 5~~
	~~8 UP osc lustre-OST0003-osc lustre-mdtlov_UUID 5~~
	~~9 UP osc lustre-OST0004-osc lustre-mdtlov_UUID 5~~
	~~10 UP osc lustre-OST0005-osc lustre-mdtlov_UUID 5~~
	</pre>		</pre>

	3. ~~''Use the'' lctl deactivate ''command to take the full OST offline:''~~		The MDS connections to OST0002 will no longer create objects there. This process should be repeated for other MDS nodes if needed. If a new file is now written to the file system, the write will be successful as the stripes are allocated across the remaining active OSTs.

	~~[root@mds ~]# lctl --device 7 deactivate~~

	4. ~~''Display the status of~~ the file system ~~components:''~~

			3. ''Once the OST is no longer full (e.g. objects deleted or migrated off the OST), it should be enabled again:''
	<pre>		<pre>
	[root@~~mds~~ ~]# lctl dl		[root@mds01 ~]# lctl set_param osp.testfs-OST0002-*.max_create_count=20000
	~~0 UP mgs MGS MGS 9~~		osp.testfs-OST0002-MDT0000.max_create_count=20000
	~~1 UP mgc MGC192~~.~~168.0.10@tcp e384bb0e~~-~~680b~~-~~ce25-7bc9-81655dd1e813 5~~		osp.testfs-OST0002-MDT0001.max_create_count=20000
	~~2 UP mdt MDS MDS_uuid 3~~
	~~3 UP lov lustre~~-~~mdtlov lustre-mdtlov_UUID 4~~
	~~4 UP mds lustre~~-MDT0000 ~~lustre-MDT0000_UUID 5~~
	~~5 UP osc lustre-OST0000-osc lustre-mdtlov_UUID 5~~
	~~6 UP osc lustre-OST0001-osc lustre-mdtlov_UUID 5~~
	~~7 IN osc lustre~~-OST0002-~~osc lustre-mdtlov_UUID 5~~
	~~8 UP osc lustre-OST0003-osc lustre-mdtlov_UUID 5~~
	~~9 UP osc lustre-OST0004-osc lustre-mdtlov_UUID 5~~
	~~10 UP osc lustre-OST0005-osc lustre-mdtlov_UUID 5~~
	</pre>		</pre>

	~~The device list shows that ''OST2'' is now inactive. If a new file is now written to the file system, the write will be successful as the stripes are allocated across the remaining active OSTs.~~
	== Migrating Data within a File System ==		== Migrating Data within a File System ==
	As stripes cannot be moved within the file system, data must be migrated manually by copying and renaming the file, removing the original file, and renaming the new file with the original file name.		As stripes cannot be moved within the file system, data must be migrated manually by copying and renaming the file, removing the original file, and renaming the new file with the original file name.

KenRawlings: Initial creation as part of accelerated wiki migration

2016-08-25T14:41:16Z

Initial creation as part of accelerated wiki migration

New page

{| class='wikitable'
|-
!Note: This page originated on the old Lustre wiki. It was identified as likely having value and was migrated to the new wiki. It is in the process of being reviewed/updated and may currently have content that is out of date.
|}

<small>''(Updated: Oct 2009)''</small>
__TOC__
Sometimes a Lustre™ file system becomes unbalanced, often due to changed stripe settings. If an OST is full and an attempt is made to write more information to the file system, an error occurs. The procedures below describe how to handle a full OST.

== Checking File System Usage ==
The example below shows an unbalanced file system:

<pre>
root@LustreClient01 ~]# lfs df -h
UUID bytes Used Available Use% Mounted on
lustre-MDT0000_UUID 4.4G 214.5M 3.9G 4% /mnt/lustre[MDT:0]
lustre-OST0000_UUID 2.0G 751.3M 1.1G 37% /mnt/lustre[OST:0]
lustre-OST0001_UUID 2.0G 755.3M 1.1G 37% /mnt/lustre[OST:1]
lustre-OST0002_UUID 2.0G 1.7G 155.1M 86% /mnt/lustre[OST:2] <-
lustre-OST0003_UUID 2.0G 751.3M 1.1G 37% /mnt/lustre[OST:3]
lustre-OST0004_UUID 2.0G 747.3M 1.1G 37% /mnt/lustre[OST:4]
lustre-OST0005_UUID 2.0G 743.3M 1.1G 36% /mnt/lustre[OST:5]

filesystem summary: 11.8G 5.4G 5.8G 45% /mnt/lustre
</pre>

In this case, ''OST:2'' is almost full and when an attempt is made to write additional information to the file system (even with uniform striping over all the OSTs), the write command fails as follows:

<pre>
[root@LustreClient01 ~]# lfs setstripe /mnt/lustre 4M 0 -1
[root@LustreClient01 ~]# dd if=/dev/zero of=/mnt/lustre/test_3 bs=10M count=100
dd: writing `/mnt/lustre/test_3': No space left on device
98+0 records in
97+0 records out
1017192448 bytes (1.0 GB) copied, 23.2411 seconds, 43.8 MB/s
</pre>

== Taking a Full OST Offline ==

To enable continued use of the file system, the full OST has to be taken offline or, more specifically, rendered read-only using the ''lctl'' command. This is done on the MDS, since the MSD allocates space for writing.

1. ''Log in to the MDS server:''

<pre>
[root@LustreClient01 ~]# ssh root@192.168.0.10
root@192.168.0.10's password:
Last login: Wed Nov 26 13:35:12 2008 from 192.168.0.6
</pre>

2. ''Use the'' lctl dl ''command to show the status of all file system components:''

<pre>
[root@mds ~]# lctl dl
0 UP mgs MGS MGS 9
1 UP mgc MGC192.168.0.10@tcp e384bb0e-680b-ce25-7bc9-81655dd1e813 5
2 UP mdt MDS MDS_uuid 3
3 UP lov lustre-mdtlov lustre-mdtlov_UUID 4
4 UP mds lustre-MDT0000 lustre-MDT0000_UUID 5
5 UP osc lustre-OST0000-osc lustre-mdtlov_UUID 5
6 UP osc lustre-OST0001-osc lustre-mdtlov_UUID 5
7 UP osc lustre-OST0002-osc lustre-mdtlov_UUID 5
8 UP osc lustre-OST0003-osc lustre-mdtlov_UUID 5
9 UP osc lustre-OST0004-osc lustre-mdtlov_UUID 5
10 UP osc lustre-OST0005-osc lustre-mdtlov_UUID 5
</pre>

3. ''Use the'' lctl deactivate ''command to take the full OST offline:''

[root@mds ~]# lctl --device 7 deactivate

4. ''Display the status of the file system components:''

<pre>
[root@mds ~]# lctl dl
0 UP mgs MGS MGS 9
1 UP mgc MGC192.168.0.10@tcp e384bb0e-680b-ce25-7bc9-81655dd1e813 5
2 UP mdt MDS MDS_uuid 3
3 UP lov lustre-mdtlov lustre-mdtlov_UUID 4
4 UP mds lustre-MDT0000 lustre-MDT0000_UUID 5
5 UP osc lustre-OST0000-osc lustre-mdtlov_UUID 5
6 UP osc lustre-OST0001-osc lustre-mdtlov_UUID 5
7 IN osc lustre-OST0002-osc lustre-mdtlov_UUID 5
8 UP osc lustre-OST0003-osc lustre-mdtlov_UUID 5
9 UP osc lustre-OST0004-osc lustre-mdtlov_UUID 5
10 UP osc lustre-OST0005-osc lustre-mdtlov_UUID 5
</pre>

The device list shows that ''OST2'' is now inactive. If a new file is now written to the file system, the write will be successful as the stripes are allocated across the remaining active OSTs.
== Migrating Data within a File System ==
As stripes cannot be moved within the file system, data must be migrated manually by copying and renaming the file, removing the original file, and renaming the new file with the original file name.

1. ''Identify the file(s) to be moved.'' In the example below, output from the ''getstripe'' command indicates that the file ''test_2'' is located entirely on OST2:

<pre>
[root@LustreClient01 ~]# lfs getstripe /mnt/lustre/test_2
OBDS:
0: lustre-OST0000_UUID ACTIVE
1: lustre-OST0001_UUID ACTIVE
2: lustre-OST0002_UUID ACTIVE
3: lustre-OST0003_UUID ACTIVE
4: lustre-OST0004_UUID ACTIVE
5: lustre-OST0005_UUID ACTIVE
/mnt/lustre/test_2
obdidx objid objid group
2 8 0x8 0
</pre>

2. ''Move the file(s):''

<pre>
[root@LustreClient01 ~]# cp /mnt/lustre/test_2 /mnt/lustre/test_2.tmp
[root@LustreClient01 ~]# rm /mnt/lustre/test_2
rm: remove regular file `/mnt/lustre/test_2'? Y
</pre>

3. ''Check the file system balance.'' The ''df'' output in the example below shows a more balanced system compared to the ''df'' output in the example in [[Handling Full OSTs]].

<pre>
[root@LustreClient01 ~]# lfs df -h
UUID bytes Used Available Use% Mounted on
lustre-MDT0000_UUID 4.4G 214.5M 3.9G 4% /mnt/lustre[MDT:0]
lustre-OST0000_UUID 2.0G 1.3G 598.1M 65% /mnt/lustre[OST:0]
lustre-OST0001_UUID 2.0G 1.3G 594.1M 65% /mnt/lustre[OST:1]
lustre-OST0002_UUID 2.0G 913.4M 1000.0M 45% /mnt/lustre[OST:2]
lustre-OST0003_UUID 2.0G 1.3G 602.1M 65% /mnt/lustre[OST:3]
lustre-OST0004_UUID 2.0G 1.3G 606.1M 64% /mnt/lustre[OST:4]
lustre-OST0005_UUID 2.0G 1.3G 610.1M 64% /mnt/lustre[OST:5]

filesystem summary: 11.8G 7.3G 3.9G 61% /mnt/lustre
</pre>

4. ''Change the name of the file back to the original filename'' so it can be found by clients.

<pre>
[root@LustreClient01 ~]# mv test2.tmp test2
[root@LustreClient01 ~]# ls /mnt/lustre
test1 test_2 test3 test_3 test4 test_4 test_x
</pre>

5. Reactivate the OST from the MDS for further writes:

<pre>
[root@mds ~]# lctl --device 7 activate

[root@mds ~]# lctl dl
0 UP mgs MGS MGS 9
1 UP mgc MGC192.168.0.10@tcp e384bb0e-680b-ce25-7bc9-81655dd1e813 5
2 UP mdt MDS MDS_uuid 3
3 UP lov lustre-mdtlov lustre-mdtlov_UUID 4
4 UP mds lustre-MDT0000 lustre-MDT0000_UUID 5
5 UP osc lustre-OST0000-osc lustre-mdtlov_UUID 5
6 UP osc lustre-OST0001-osc lustre-mdtlov_UUID 5
7 UP osc lustre-OST0002-osc lustre-mdtlov_UUID 5
8 UP osc lustre-OST0003-osc lustre-mdtlov_UUID 5
9 UP osc lustre-OST0004-osc lustre-mdtlov_UUID 5
10 UP osc lustre-OST0005-osc lustre-mdtlov_UUID
</pre>

[[Category: NeedsReview]]