Configuring Lustre File Striping
| Note: This page originated on the old Lustre wiki. It was identified as likely having value and was migrated to the new wiki. It is in the process of being reviewed/updated and may currently have content that is out of date. |
|---|
One of the main factors leading to the high performance of Lustre™ file systems is the ability to stripe data over multiple OSTs. The stripe count can be set on a file system, directory, or file level. An example showing the use of striping is provided below.
For additional information, see Chapter 18: Managing File Striping and Free Space in the Lustre Operations Manual
Setting Up Striping
To see the current stripe size, use the command lfs getstripe [file, dir, fs]. This command will produce output similar to the following:
root@LustreClient01 lustre]# lfs getstripe /mnt/lustre OBDS: 0: lustre-OST0000_UUID ACTIVE 1: lustre-OST0001_UUID ACTIVE 2: lustre-OST0002_UUID ACTIVE 3: lustre-OST0003_UUID ACTIVE 4: lustre-OST0004_UUID ACTIVE 5: lustre-OST0005_UUID ACTIVE /mnt/lustre (Default) stripe_count: 2 stripe_size: 4M stripe_offset: 0
In this example, the default stripe count is 2 (that is, data blocks are striped over two OSTs), the default stripe size is 4 MB (the stripe size can be set in K, M or G), and all writes start from the first OST.
Note: When setting the stripe, the offset is set before the stripe count.
The command to set a new stripe pattern on the file system may look like this:
[root@LustreClient01 lustre]# lfs setstripe -s 4M -i 0 -c 1 /mnt/lustre
This example command sets the stripe of /mnt/lustre to 4 MB blocks starting at OST0 and spanning over one OST. If a new file is created with these settings, the following results are seen:
[root@LustreClient01 lustre]# dd if=/dev/zero of=/mnt/lustre/test1 bs=10M count=100 root@LustreClient01 lustre]# lfs df -h UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 4.4G 214.5M 3.9G 4% /mnt/lustre[MDT:0] lustre-OST0000_UUID 2.0G 1.1G 830.1M 53% /mnt/lustre[OST:0] lustre-OST0001_UUID 2.0G 83.3M 1.8G 4% /mnt/lustre[OST:1] lustre-OST0002_UUID 2.0G 83.3M 1.8G 4% /mnt/lustre[OST:2] lustre-OST0003_UUID 2.0G 83.3M 1.8G 4% /mnt/lustre[OST:3] lustre-OST0004_UUID 2.0G 83.3M 1.8G 4% /mnt/lustre[OST:4] lustre-OST0005_UUID 2.0G 83.3M 1.8G 4% /mnt/lustre[OST:5] filesystem summary: 11.8G 1.5G 9.7G 12% /mnt/lustre
In this example, the entire file was written to the first OST with a very uneven distribution of data blocks.
Continuing with this example, the file is removed and the stripe count is changed to a value of -1 to specify striping over all available OSTs:
[root@LustreClient01 lustre]# lfs setstripe -s 4M -i 0 -c -1 /mnt/lustre
Now, when a file is created, the new stripe setting evenly distributes the data over all the available OSTs:
[root@LustreClient01 lustre]# dd if=/dev/zero of=/mnt/lustre/test1 bs=10M count=100 100+0 records in 100+0 records out 1048576000 bytes (1.0 GB) copied, 20.2589 seconds, 51.8 MB/s [root@LustreClient01 lustre]# lfs df -h UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 4.4G 214.5M 3.9G 4% /mnt/lustre[MDT:0] lustre-OST0000_UUID 2.0G 251.3M 1.6G 12% /mnt/lustre[OST:0] lustre-OST0001_UUID 2.0G 251.3M 1.6G 12% /mnt/lustre[OST:1] lustre-OST0002_UUID 2.0G 251.3M 1.6G 12% /mnt/lustre[OST:2] lustre-OST0003_UUID 2.0G 251.3M 1.6G 12% /mnt/lustre[OST:3] lustre-OST0004_UUID 2.0G 247.3M 1.6G 12% /mnt/lustre[OST:4] lustre-OST0005_UUID 2.0G 247.3M 1.6G 12% /mnt/lustre[OST:5] filesystem summary: 11.8G 1.5G 9.7G 12% /mnt/lustre
Displaying Stripe Information for a File
The lfs getstripe command can be used to display information that shows over which OSTs a file is distributed. For example, the output from the following command (showing multiple obdidx entries) indicates that the file test1 is striped over all six active OSTs in the configuration:
[root@LustreClient01 ~]# lfs getstripe /mnt/lustre/test1
OBDS:
0: lustre-OST0000_UUID ACTIVE
1: lustre-OST0001_UUID ACTIVE
2: lustre-OST0002_UUID ACTIVE
3: lustre-OST0003_UUID ACTIVE
4: lustre-OST0004_UUID ACTIVE
5: lustre-OST0005_UUID ACTIVE
/mnt/lustre/test1
obdidx objid objid group
0 8 0x8 0
1 4 0x4 0
2 5 0x5 0
3 5 0x5 0
4 4 0x4 0
5 2 0x2 0
In contrast, the output from the following command, which lists just a single obdidx entry, indicates that the file test2 is contained on a single OST:
[root@LustreClient01 ~]# lfs getstripe /mnt/lustre/test_2
OBDS:
0: lustre-OST0000_UUID ACTIVE
1: lustre-OST0001_UUID ACTIVE
2: lustre-OST0002_UUID ACTIVE
3: lustre-OST0003_UUID ACTIVE
4: lustre-OST0004_UUID ACTIVE
5: lustre-OST0005_UUID ACTIVE
/mnt/lustre/test_2
obdidx objid objid group
2 8 0x8 0