Creating Lustre Object Storage Services (OSS)

Syntax Overview
The syntax for creating an OST is:

 mkfs.lustre --ost \ [--reformat] \ --fsname \ --index  \ --mgsnode  [--mgsnode  …] \ [ --servicenode  [--servicenode  …]] \ [ --failnode  [--failnode  …]] \ [ --backfstype=ldiskfs|zfs ] \ [ --mkfsoptions ] \ | / command requires the file system name, OST index, list of MGS server NIDs, and the NIDS for each of the OSS machines that can mount the OST (used for HA failover). The command also specifies the back-end file system type and device path or ZFS pool.

The MGS NIDs are supplied using the  flag. If there is more than one potential location for the MGS (i.e., it is part of a high availability failover cluster configuration), then the option is repeated for as many failover nodes as are configured (usually there are two).

Ordering of the MGS nodes in the command line is significant: the first  flag must reference the NID of the current active or primary MGS server. If this is not the case, then the first time that the OST tries to join the cluster, it will fail. The first time mount of a storage target does not currently check the failover locations when trying to establish a connection with the MGS. When adding new storage targets to Lustre, the MGS must be running on its primary NID.

Each OST must also be supplied with the name of the Lustre file system it will join (maximum 8 characters), and an index number  unique to the file system.

The list of service nodes or failover nodes  must be specified for any high availability configuration. Although there are more compact declarations for defining the nodes, for simplicity, list the NID of each server that can mount the storage as a separate  entry.

The next example uses the  syntax to create an OST that can be run on two servers as an HA failover resource:

 [root@rh7z-oss1 system]# mkfs.lustre --ost \ >  --fsname demo \ >  --index 0 \ >  --mgsnode 192.168.227.11@tcp1 \ >  --mgsnode 192.168.227.12@tcp1 \ >  --servicenode 192.168.227.21@tcp1 \ >  --servicenode 192.168.227.22@tcp1 \ >  /dev/dm-3

The command line formats a new OST that will be used by an OSS for storage. The OST will be part of a file system called, with index number   (zero). There are two NIDs defined as the nodes able to host the OSS service, denoted by the  options, and two NIDs supplied for the MGS that the OSS will register with.

The  syntax is similar, but is used to define only a failover target for the storage service. For example:

 [root@rh7z-oss1 system]# mkfs.lustre --ost \ >  --fsname demo \ >  --index 0 \ >  --mgsnode 192.168.227.11@tcp1 \ >  --mgsnode 192.168.227.12@tcp1 \ >  --failnode 192.168.227.22@tcp1 \ >  /dev/dm-3

Here, the failover host is identified as, one server in an HA pair (and which, for the purpose of this example, has the hostname  ). The  command was executed on   (NID:  ), and the   command must also be run from this host when the service starts for the very first time.

OST Formatted as an LDISKFS OSD
The syntax for creating an LDISKFS-based OST is:

 mkfs.lustre --ost \ [--reformat] \ --fsname \ --index  \ --mgsnode  [--mgsnode  …] \ [ --servicenode  [--servicenode  …]] \ [ --failnode <NID> [--failnode <NID> …]] \ [ --backfstype=ldiskfs ] \ [ --mkfsoptions ] \

The next example uses the  syntax to create an OST that can be run on two servers as an HA failover resource:

<pre style="overflow-x:auto;"> [root@rh7z-oss1 system]# mkfs.lustre --ost \ >  --fsname demo \ >  --index 0 \ >  --mgsnode 192.168.227.11@tcp1 \ >  --mgsnode 192.168.227.12@tcp1 \ >  --servicenode 192.168.227.21@tcp1 \ >  --servicenode 192.168.227.22@tcp1 \ >  --backfstype=ldiskfs \ >  /dev/dm-3

The  syntax is similar, but is used to define only a failover target for the storage service. For example:

<pre style="overflow-x:auto;"> [root@rh7z-oss1 system]# mkfs.lustre --ost \ >  --fsname demo \ >  --index 0 \ >  --mgsnode 192.168.227.11@tcp1 \ >  --mgsnode 192.168.227.12@tcp1 \ >  --failnode 192.168.227.22@tcp1 \ >  --backfstype=ldiskfs \ >  /dev/dm-3

The above examples are repeated from the main introduction to the syntax, but are included here to maintain symmetry with the rest of the text. However, note that the  flag has been set to , which tells   to format the device as an LDISKFS OSD.

Formatting an OST using only the mkfs.lustre command
Note: For the greatest flexibility and control when creating ZFS-based Lustre storage targets, do not use this approach – instead, create the zpool separately from formatting the Lustre OSD. See Formatting an OST using zpool and mkfs.lustre.

The syntax for creating a ZFS-based OST using only the  command is:

<pre style="overflow-x:auto;"> mkfs.lustre --ost \ [--reformat] \ --fsname \ --index <n> \ --mgsnode <MGS NID> [--mgsnode <MGS NID> …] \ [ --servicenode <NID> [--servicenode <NID> …]] \ [ --failnode <NID> [--failnode <NID> …]] \ --backfstype=zfs \ [ --mkfsoptions ] \ / syntax to create an OST that can be run on two servers as an HA failover resource:

<pre style="overflow-x:auto;"> [root@rh7z-oss1 system]# mkfs.lustre --ost \ >  --fsname demo \ >  --index 0 \ >  --mgsnode 192.168.227.11@tcp1 \ >  --mgsnode 192.168.227.12@tcp1 \ >  --servicenode 192.168.227.21@tcp1 \ >  --servicenode 192.168.227.22@tcp1 \ >  --backfstype=zfs \ >  --mkfsoptions "recordsize=1024K" \ >  demo-ost0pool/ost0 \ >  raidz2 sda sdb sdc sdd sde sdf

The command line formats a new ZFS-based OST that will be used by an OSS for storage. The OST will be part of to a file system called, with index number   (zero). The back-end storage is a ZFS pool called  comprising a RAIDZ2 vdev constructed from six physical devices, and creates a ZFS file system dataset called. Two server NIDs are supplied as service nodes for the OSS,  and , and there are NIDs for the MGS primary and failover hosts.

In the above command-line, notice that there is a custom property for the recordsize, set using. The syntax of string needs some explanation.

The  command takes the string provided as an argument to , prefixes the string with   and inserts it into the   command line that creates the dataset. This can be observed in the output of the  command.

If only one property is being altered, then just create the  pair string without any additional flags or arguments. However, if more than one property needs to be set, then separate each property with. For example:

<pre style="overflow-x:auto;"> --mkfsoptions="recordsize=1024K -o mountpoint=none"

Refer to ZFS recordsize Property for recommendations on how to set the recordsize appropriately.

The  syntax is similar, but is used to define only a failover target for the storage service. For example:

[root@rh7z-oss1 system]# mkfs.lustre --ost \ >  --fsname demo --index 0 \ >  --mgsnode 192.168.227.11@tcp1 --mgsnode 192.168.227.12@tcp1 \ >  --failnode 192.168.227.22@tcp1 \ >  --backfstype=zfs \ >  demo-ost0pool/ost0 mirror sdb sdd

Note:: When creating a ZFS-based OSD using only the  command, it is not possible to set or change some properties of the zpool or its vdevs, such as the   property. For this reason, it is highly recommended that the zpools be created independently of the  command, as shown in the next section.

Formatting an OST using zpool and mkfs.lustre
To create a ZFS-based OST, create a zpool to contain the OST file system dataset, then use  to create the actual file system dataset inside the zpool:

<pre style="overflow-x:auto;"> zpool create [-f] \ -O canmount=off \ -O recordsize=<RK> \ [-o ashift=<n>] \ -o cachefile=/etc/zfs/ .spec | -o cachefile=none \ \

mkfs.lustre --ost \ [--reformat] \ --fsname \ --index <n> \ --mgsnode <MGS NID> [--mgsnode <MGS NID> …] \ [ --servicenode <NID> [--servicenode <NID> …]] \ [ --failnode <NID> [--failnode <NID> …]] \ --backfstype=zfs \ [ --mkfsoptions ] \ / command or   to verify that the file system dataset has been formatted correctly. For example:

<pre style="overflow-x:auto;"> [root@rh7-oss1 ~]# zfs get all -s local,inherited NAME               PROPERTY              VALUE                                    SOURCE demo-ost0pool      recordsize            1M                                       local demo-ost0pool      canmount              off                                      local demo-ost0pool/ost0 recordsize            1M                                       inherited from demo-ost0pool demo-ost0pool/ost0 canmount              off                                      local demo-ost0pool/ost0 xattr                 sa                                       local demo-ost0pool/ost0 lustre:mgsnode        192.168.227.11@tcp1:192.168.227.12@tcp1  local demo-ost0pool/ost0 lustre:flags          4194                                     local demo-ost0pool/ost0 lustre:fsname         demo                                     local demo-ost0pool/ost0 lustre:version        1                                        local demo-ost0pool/ost0 lustre:failover.node  192.168.227.21@tcp1:192.168.227.22@tcp1  local demo-ost0pool/ost0 lustre:index          0                                        local demo-ost0pool/ost0 lustre:svname         demo:OST0000                             local

Starting the OSS Service
The  command is used to start all Lustre storage services, including the OSS. The syntax is:

<pre style="overflow-x:auto;"> mount -t lustre [-o ] \ | / command syntax is very similar for both LDISKFS and ZFS storage targets. The main difference is the format of the path to the storage. For LDISKFS, the path will resolve to a block device, such as  or , whereas for ZFS, the path resolves to a dataset in a zpool, e.g..

The mount point directory must exist before the mount command is executed. The recommended convention for the mount point of the OST storage is, where   is the name of the file system and   is the index number of the OST.

The following example starts a ZFS-based OSS:

<pre style="overflow-x:auto;"> [root@rh7z-oss1 ~]# zfs list -o name,used,avail,refer NAME                USED  AVAIL  REFER demo-ost0pool       173M  48.0G    19K demo-ost0pool/ost0  173M  48.0G   173M

[root@rh7z-oss1 lz]# mkdir -p /lustre/demo/ost0 [root@rh7z-oss1 lz]# mount -t lustre demo-ost0pool/ost0 /lustre/demo/ost0

[root@rh7z-oss1 lz]# df -ht lustre File system         Size  Used Avail Use% Mounted on demo-ost0pool/ost0   49G  2.9M   49G   1% /lustre/demo/ost0

As with all Lustre storage targets, only the  command can start Lustre services.

To verify that the OSS is running, check that the device has been mounted, then get the Lustre device list with  and review the running processes:

<pre style="overflow-x:auto;"> [root@rh7z-oss1 lz]# lctl dl 0 UP osd-zfs demo-OST0000-osd demo-OST0000-osd_UUID 5 1 UP mgc MGC192.168.227.11@tcp1 4106d169-ed51-cd92-3361-12800a73962d 5 2 UP ost OSS OSS_uuid 3 3 UP obdfilter demo-OST0000 demo-OST0000_UUID 7 4 UP lwp demo-MDT0000-lwp-OST0000 demo-MDT0000-lwp-OST0000_UUID 5

[root@rh7z-oss1 lz]# ps -ef | awk '/ost/ && !/awk/' root     1932     1  0 Apr04 ? 00:00:00 /usr/libexec/postfix/master -w postfix  1984  1932  0 Apr04 ? 00:00:00 pickup -l -t unix -u postfix  1985  1932  0 Apr04 ? 00:00:00 qmgr -l -t unix -u root    24709     2  0 00:03 ? 00:00:00 [ll_ost00_000] root    24710     2  0 00:03 ? 00:00:00 [ll_ost00_001] root    24711     2  0 00:03 ? 00:00:00 [ll_ost00_002] root    24712     2  0 00:03 ? 00:00:00 [ll_ost_create00] root    24713     2  0 00:03 ? 00:00:00 [ll_ost_create00] root    24714     2  0 00:03 ? 00:00:00 [ll_ost_io00_000] root    24715     2  0 00:03 ? 00:00:00 [ll_ost_io00_001] root    24716     2  0 00:03 ? 00:00:00 [ll_ost_io00_002] root    24717     2  0 00:03 ? 00:00:00 [ll_ost_seq00_00] root    24718     2  0 00:03 ? 00:00:00 [ll_ost_seq00_00] root    24719     2  0 00:03 ? 00:00:00 [ll_ost_out00_00] root    24720     2  0 00:03 ? 00:00:00 [ll_ost_out00_00]

Stopping the OSS Service
To stop a Lustre service, umount the corresponding target:

<pre style="overflow-x:auto;"> umount

The mount point must correspond to the mount point used with the  command. For example:

<pre style="overflow-x:auto;"> [root@rh7z-oss1 lz]# df -ht lustre File system         Size  Used Avail Use% Mounted on demo-ost0pool/ost0   49G  2.9M   49G   1% /lustre/demo/ost0 [root@rh7z-oss1 ~]# umount /lustre/demo/ost0 [root@rh7z-oss1 ~]# df -ht lustre df: no file systems processed [root@rh7z-oss1 ~]# lctl dl [root@rh7z-oss1 ~]#

The regular  command is the correct way to stop a given Lustre service and unmount the associated storage, for both LDISKFS and ZFS-based Lustre storage volumes.

Do not use the  command to stop a Lustre service. Attempting to use  commands to unmount a storage target that is mounted as part of an active Lustre service will return an error.