Mounting a Lustre File System on Client Nodes - Revision history

Adilger: /* Starting and stopping the Lustre Client */ include hostnames in mount examples, fix line wrapping

2024-06-11T18:44:49Z

Starting and stopping the Lustre Client: include hostnames in mount examples, fix line wrapping

← Older revision		Revision as of 11:44, 11 June 2024
Line 13:		Line 13:

	<pre style="overflow-x:auto;">		<pre style="overflow-x:auto;">
	mount -t lustre \		mount -t lustre [-o OPTIONS ] MGSNID[:MGSNID]:/FSNAME /lustre/FSNAME
	[-o ~~<options>~~ ] \
	~~<MGS NID>~~[:~~<MGS NID>~~]:/~~<fsname> \~~
	/lustre/~~<fsname>~~
	</pre>		</pre>

Line 22:		Line 19:

	<pre style="overflow-x:auto;">		<pre style="overflow-x:auto;">
	umount ~~<path>~~		umount /lustre/FSNAME
	</pre>		</pre>

	The <code>mount</code> and <code>umount</code> commands require super-user privileges to run.		The <code>mount</code> and <code>umount</code> commands require super-user privileges to run.

	The mount point directory must exist before the mount command is executed. The recommended convention for the mount point of the client is <code>/lustre/~~<fsname>/~~</code>, where <code>~~<fsname>~~</code> is the name of the file system.		The mount point directory must exist before the mount command is executed. The recommended convention for the mount point of the client is <code>/lustre/FSNAME</code>, where <code>FSNAME</code> is the name of the file system.

	When the <code>mount</code> command is invoked, the client first registers with the MGS to retrieve the configuration information, also referred to as the log, for the file system that it wants to mount. A single MGS can store the configuration information for more than one file system.		When the <code>mount</code> command is invoked, the client first registers with the MGS to retrieve the configuration information, also referred to as the log, for the file system that it wants to mount. A single MGS can store the configuration information for more than one file system.
Line 35:		Line 32:
	<pre style="overflow-x:auto;">		<pre style="overflow-x:auto;">
	mkdir -p /lustre/demo		mkdir -p /lustre/demo
	mount -t lustre \		mount -t lustre 192.168.227.11@tcp1:192.168.227.12@tcp1:/demo /lustre/demo
	192.168.227.11@tcp1:192.168.227.12@tcp1:/demo \
	/lustre/demo
	</pre>		</pre>

	The client will try to connect to the MGS in the order of the NID addresses supplied on the command line. If connection to the first NID fails, the client will attempt a connection using the next NID.		The client will try to connect to the MGS in the order of the NID addresses supplied on the command line. If connection to the first NID fails, the client will attempt a connection using the next NID. Note that if there are DNS addresses for the MGS NIDs the hostname can also be used and will be resolved by the <code>mount</code> command.

	To verify that the file system is mounted on the client, use the <code>df</code> command:		To verify that the file system is mounted on the client, use the <code>df</code> command:
Line 47:		Line 42:
	[root@rh7z-c3 ~]# df -ht lustre		[root@rh7z-c3 ~]# df -ht lustre
	File system Size Used Avail Use% Mounted on		File system Size Used Avail Use% Mounted on
	~~192.168.227.11~~@tcp1:~~192.168.227.12~~@tcp1:/demo 49G 2.9M 49G 1% /lustre/demo		demo-mgs1@tcp1:demo-mgs2@tcp1:/demo 49G 2.9M 49G 1% /lustre/demo

	The lctl dl command provides detail on the connections to the Lustre services:		The lctl dl command provides detail on the connections to the Lustre services:
Line 61:		Line 56:

	<pre style="overflow-x:auto;">		<pre style="overflow-x:auto;">
	[root@rh7z-c3 ~]# mount -t lustre \		[root@rh7z-c3 ~]# mount -t lustre demo-mgs1@tcp1:demo-mgs2@tcp1:/demo /lustre/demo
	~~> 192.168.227.11~~@tcp1:~~192.168.227.12~~@tcp1:/demo \
	> /lustre/demo
	mount.lustre: mount 192.168.227.11@tcp1:192.168.227.12@tcp1:/demo at /lustre/demo failed: Input/output error		mount.lustre: mount 192.168.227.11@tcp1:192.168.227.12@tcp1:/demo at /lustre/demo failed: Input/output error
	Is the MGS running?		Is the MGS running?
Line 78:		Line 71:
	</pre>		</pre>

	The most common cause of failure is an improperly configured network interface, or LNet NID. Verify that the LNet protocol is able to communicate with the MGS with <code>lctl ping</code>:		The most common cause of failure is an improperly configured network interface, or LNet NID. Verify that the LNet protocol is able to communicate with the MGS node with <code>lctl ping</code> to the hostname or NID of that server:

	<pre style="overflow-x:auto;">		<pre style="overflow-x:auto;">
	lctl ping ~~<MGS NID>~~		lctl ping MGSNID
	</pre>		</pre>

	If the ping fails, the command will return an ~~I/O~~ error:		If the ping fails, the command will return an error:

	<pre style="overflow-x:auto;">		<pre style="overflow-x:auto;">
	[root@rh7z-c3 ~]# lctl ping ~~192.168.227.11~~@tcp1		[root@rh7z-c3 ~]# lctl ping demo-mgs1@tcp1
	failed to ping 192.168.227.11@tcp1: Input/output error		failed to ping 192.168.227.11@tcp1: Input/output error
			[root@rh7z-c3 ~]# lctl ping demo-mgs2@tcp1
			failed to ping 192.168.227.12@tcp1: Connection refused
	</pre>		</pre>

Line 95:		Line 90:
	If the ping succeeds, but the mount still fails, verify that the Lustre services are running on the target host. Also check to see if there are any services running on the client that might be interfering with communication, such as a firewall or SELinux. While SELinux is supported in Lustre 2.8 onwards, older releases of Lustre are not compatible. Temporarily disabling the firewall and SELinux can help narrow down the root cause of issues with Lustre communications.		If the ping succeeds, but the mount still fails, verify that the Lustre services are running on the target host. Also check to see if there are any services running on the client that might be interfering with communication, such as a firewall or SELinux. While SELinux is supported in Lustre 2.8 onwards, older releases of Lustre are not compatible. Temporarily disabling the firewall and SELinux can help narrow down the root cause of issues with Lustre communications.

	If there are no OSS services online, but the MGS and the MDS for MDT0 are running, then the client mount command will hang indefinitely until an OSS service starts up.		If there are no OSS services online, but the MGS and the MDS for MDT0 are running, then the client mount command will hang indefinitely until at least one OSS service starts up, otherwise the filesystem will not be usable.

	There are options specific to Lustre that can be applied to the Lustre client <code>mount</code> command. The most common of these are <code>flock</code>, <code>localflock</code> and <code>user_xattr</code>:		There are options specific to Lustre that can be applied to the Lustre client <code>mount</code> command. The most common of these are <code>flock</code>, <code>localflock</code> and <code>user_xattr</code>:

	* <code>flock</code>: enable support for cluster-wide, coherent ~~posix~~ file locks ('flock's). Must be applied to the <code>mount</code> commands for all clients that will be accessing common data requiring lock functionality. This has no measurable performance impact in modern (2.x) versions of Lustre (this page previously stated there was a performance impact; this is not correct for modern versions). This is the recommended mode, and is enabled by default in Lustre 2.13 and newer.		* <code>flock</code>: enable support for cluster-wide, coherent POSIX file locks ('flock'). Must be applied to the <code>mount</code> commands for all clients that will be accessing common data requiring lock functionality. This has no measurable performance impact in modern (2.x) versions of Lustre (this page previously stated there was a performance impact; this is not correct for modern versions). This is the recommended mode, and is enabled by default in Lustre 2.13 and newer.
	* <code>localflock</code>: enable client-local <code>flock</code> support. This option is a holdover from older versions where full flock support had a performance impact. It is potentially unsafe and should be used with caution - it is only suitable for applications that require posix file locks, but don’t run on multiple hosts (or where the data will not be accessed in a manner that would require locking across multiple hosts). This generally implies that you have a multi-threaded app using flocks that is known to be running on only one node. If the app is ever run across multiple hosts, it will not receive the expected locking behavior, which would generally result in application failures.		* <code>localflock</code>: enable client-local <code>flock</code> support. This option is a holdover from older versions where full flock support had a performance impact. It is potentially unsafe and should be used with caution - it is only suitable for applications that require posix file locks, but don’t run on multiple hosts (or where the data will not be accessed in a manner that would require locking across multiple hosts). This generally implies that you have a multi-threaded app using flocks that is known to be running on only one node. If the app is ever run across multiple hosts, it will not receive the expected locking behavior, which would generally result in application failures.
	* <code>user_xattr</code>: Enable support for user extended attributes.		* <code>user_xattr</code>: Enable support for user extended attributes.

	Additionally, consider using the <code>_netdev</code> mount option when mounting the Lustre client~~, especially when adding an entry into~~ <code>/etc/fstab</code>. This option indicates to the operating system that the file system has a dependency on the network such that it should not be mounted before the network is online and should be unmounted on shutdown prior to stopping the network stack. An example entry for <code>/etc/fstab</code>:		Additionally, consider using the <code>_netdev</code> mount option when mounting the Lustre client automatically at mount time from the <code>/etc/fstab</code> file. This option indicates to the operating system that the file system has a dependency on the network such that it should not be mounted before the network is online and should be unmounted on shutdown prior to stopping the network stack. An example entry for <code>/etc/fstab</code>:

	<pre style="overflow-x:auto;">		<pre style="overflow-x:auto;">
	~~192.168.227.11~~@tcp1:~~192.168.227.12~~@tcp1:/demo /lustre/demo lustre defaults,_netdev 0 0		demo-mgs1@tcp1:demo-mgs2@tcp1:/demo /lustre/demo lustre defaults,_netdev 0 0
	</pre>		</pre>

	For systemd based systems, the following is recommended to ensure ~~correct startup and shutdown~~. Be sure you have properly [[Dynamic_LNet_Configuration_and_lnetctl\| setup <code>lnet</code> service in systemd]] before using these options:		For <code>systemd</code> based systems, the following options are recommended to ensure the mount command is not run before the network is configured. Be sure you have properly [[Dynamic_LNet_Configuration_and_lnetctl\| setup <code>lnet</code> service in systemd]] before using these options:

	<pre style="overflow-x:auto;">		<pre style="overflow-x:auto;">
	~~192.168.227.11~~@tcp1:~~192.168.227.12~~@tcp1:/demo /lustre/demo lustre defaults,_netdev,noauto,x-systemd.automount,x-systemd.requires=lnet.service		demo-mgs1@tcp1:demo-mgs2@tcp1:/demo /lustre/demo lustre defaults,_netdev,noauto,x-systemd.automount,x-systemd.requires=lnet.service
	0 0		0 0
	</pre>		</pre>

AurelienDegremont: /* Starting and stopping the Lustre Client */

2020-03-05T16:19:20Z

Starting and stopping the Lustre Client

← Older revision		Revision as of 09:19, 5 March 2020
Line 109:		Line 109:
	</pre>		</pre>

	For systemd based systems, the following is recommended to ensure correct startup and shutdown:		For systemd based systems, the following is recommended to ensure correct startup and shutdown. Be sure you have properly [[Dynamic_LNet_Configuration_and_lnetctl\| setup <code>lnet</code> service in systemd]] before using these options:

	<pre style="overflow-x:auto;">		<pre style="overflow-x:auto;">

Paf at 15:25, 16 July 2019

2019-07-16T15:25:09Z

← Older revision		Revision as of 08:25, 16 July 2019
Line 99:		Line 99:
	There are options specific to Lustre that can be applied to the Lustre client <code>mount</code> command. The most common of these are <code>flock</code>, <code>localflock</code> and <code>user_xattr</code>:		There are options specific to Lustre that can be applied to the Lustre client <code>mount</code> command. The most common of these are <code>flock</code>, <code>localflock</code> and <code>user_xattr</code>:

	* <code>flock</code>: enable support for cluster-wide, coherent file locks. Must be applied to the <code>mount</code> commands for all clients that will be accessing common data requiring lock functionality. ~~Cluster-wide locking will have~~ a ~~detrimental~~ impact ~~on file system performance, and should only be enabled when absolutely required~~. ~~For some applications, the locking~~ is ~~only necessary on a sub-set of nodes. For example,~~ the ~~CTDB cluster framework used by Samba to provide a parallel~~, ~~high-availability SMB gateway, relies on locking of a shared file when coordinating cluster start-up~~ and ~~recovery. However, only the CTDB nodes need to mount the~~ Lustre ~~file system with the <code>flock</code> option~~. ~~This is an example of application or domain-specific lock requirements~~.		* <code>flock</code>: enable support for cluster-wide, coherent posix file locks ('flock's). Must be applied to the <code>mount</code> commands for all clients that will be accessing common data requiring lock functionality. This has no measurable performance impact in modern (2.x) versions of Lustre (this page previously stated there was a performance impact; this is not correct for modern versions). This is the recommended mode, and is enabled by default in Lustre 2.13 and newer.
	* <code>localflock</code>: enable client-local <code>flock</code> support. This is ~~much faster than cluster-wide <code>~~flock~~</code>~~ support~~, but~~ is only suitable for applications that require locks, but don’t run on multiple hosts (or where the data will not be accessed in a manner that would require locking across multiple hosts).		* <code>localflock</code>: enable client-local <code>flock</code> support. This option is a holdover from older versions where full flock support had a performance impact. It is potentially unsafe and should be used with caution - it is only suitable for applications that require posix file locks, but don’t run on multiple hosts (or where the data will not be accessed in a manner that would require locking across multiple hosts). This generally implies that you have a multi-threaded app using flocks that is known to be running on only one node. If the app is ever run across multiple hosts, it will not receive the expected locking behavior, which would generally result in application failures.
	* <code>user_xattr</code>: Enable support for user extended attributes.		* <code>user_xattr</code>: Enable support for user extended attributes.

Utopiabound: /* Starting and stopping the Lustre Client */ Add info for client mounts with systemd (C.f. https://jira.hpdd.intel.com/browse/LU-8293)

2018-05-07T16:39:03Z

Starting and stopping the Lustre Client: Add info for client mounts with systemd (C.f. https://jira.hpdd.intel.com/browse/LU-8293)

← Older revision		Revision as of 09:39, 7 May 2018
Line 107:		Line 107:
	<pre style="overflow-x:auto;">		<pre style="overflow-x:auto;">
	192.168.227.11@tcp1:192.168.227.12@tcp1:/demo /lustre/demo lustre defaults,_netdev 0 0		192.168.227.11@tcp1:192.168.227.12@tcp1:/demo /lustre/demo lustre defaults,_netdev 0 0
			</pre>

			For systemd based systems, the following is recommended to ensure correct startup and shutdown:

			<pre style="overflow-x:auto;">
			192.168.227.11@tcp1:192.168.227.12@tcp1:/demo /lustre/demo lustre defaults,_netdev,noauto,x-systemd.automount,x-systemd.requires=lnet.service
			0 0
	</pre>		</pre>

Malcolm: Created page with "All end-user application I/O happens via a service called the Lustre client. The client is responsible for providing a POSIX interface to applications, creating a coherent pre..."

2017-08-16T08:26:42Z

Created page with "All end-user application I/O happens via a service called the Lustre client. The client is responsible for providing a POSIX interface to applications, creating a coherent pre..."

New page

All end-user application I/O happens via a service called the Lustre client. The client is responsible for providing a POSIX interface to applications, creating a coherent presentation of the metadata (file system name space) and object data (file content) to applications running on the client operating system. All Lustre file system IO is transacted over a network protocol.

Client specifications are entirely application-driven and vary widely across the spectrum of applications, organisations and industries. Lustre clients must be running a Linux operating system, and the client software is comprised of kernel modules, with some user-space tools to assist with configuration and management.

== Starting and stopping the Lustre Client ==

'''Pre-requisites''':

# [[Installing the Lustre Software|Lustre software has been installed]] on the machine
# [[:Category:Lustre Networking (LNet)|LNet has been configured]]

Start a Lustre client using the <code>mount</code> command, the basic syntax of which is:

<pre style="overflow-x:auto;">
mount -t lustre \
[-o <options> ] \
<MGS NID>[:<MGS NID>]:/<fsname> \
/lustre/<fsname>
</pre>

To stop the Lustre client, unmount the file system:

<pre style="overflow-x:auto;">
umount <path>
</pre>

The <code>mount</code> and <code>umount</code> commands require super-user privileges to run.

The mount point directory must exist before the mount command is executed. The recommended convention for the mount point of the client is <code>/lustre/<fsname>/</code>, where <code><fsname></code> is the name of the file system.

When the <code>mount</code> command is invoked, the client first registers with the MGS to retrieve the configuration information, also referred to as the log, for the file system that it wants to mount. A single MGS can store the configuration information for more than one file system.

The following example shows the command line used to mount a file system named <code>demo</code>:

<pre style="overflow-x:auto;">
mkdir -p /lustre/demo
mount -t lustre \
192.168.227.11@tcp1:192.168.227.12@tcp1:/demo \
/lustre/demo
</pre>

The client will try to connect to the MGS in the order of the NID addresses supplied on the command line. If connection to the first NID fails, the client will attempt a connection using the next NID.

To verify that the file system is mounted on the client, use the <code>df</code> command:

<pre style="overflow-x:auto;">
[root@rh7z-c3 ~]# df -ht lustre
File system Size Used Avail Use% Mounted on
192.168.227.11@tcp1:192.168.227.12@tcp1:/demo 49G 2.9M 49G 1% /lustre/demo

The lctl dl command provides detail on the connections to the Lustre services:
[root@rh7z-c3 ~]# lctl dl
0 UP mgc MGC192.168.227.11@tcp1 7f07b5f9-27e3-0b09-7456-d83ae184d204 5
1 UP lov demo-clilov-ffff8800bab6a000 c04fa65d-3f0b-9cbf-b373-6a894da8e0be 4
2 UP lmv demo-clilmv-ffff8800bab6a000 c04fa65d-3f0b-9cbf-b373-6a894da8e0be 4
3 UP mdc demo-MDT0000-mdc-ffff8800bab6a000 c04fa65d-3f0b-9cbf-b373-6a894da8e0be 5
4 UP osc demo-OST0000-osc-ffff8800bab6a000 c04fa65d-3f0b-9cbf-b373-6a894da8e0be 5
</pre>

If the MGS is unavailable, the <code>mount</code> command will return an error, similar to the following example:

<pre style="overflow-x:auto;">
[root@rh7z-c3 ~]# mount -t lustre \
> 192.168.227.11@tcp1:192.168.227.12@tcp1:/demo \
> /lustre/demo
mount.lustre: mount 192.168.227.11@tcp1:192.168.227.12@tcp1:/demo at /lustre/demo failed: Input/output error
Is the MGS running?
</pre>

More detailed information on the failure will be in the syslog and kernel ring buffer:

<pre style="overflow-x:auto;">
[ 9996.909126] Lustre: 10199:0:(client.c:1967:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1459822631/real 1459822631] req@ffff8800ad298000 x1530734379532292/t0(0) o250->MGC192.168.227.11@tcp1@192.168.227.11@tcp1:26/25 lens 400/544 e 0 to 1 dl 1459822636 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
[10021.923403] Lustre: 10199:0:(client.c:1967:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1459822656/real 1459822656] req@ffff880138238000 x1530734379532308/t0(0) o250->MGC192.168.227.11@tcp1@192.168.227.12@tcp1:26/25 lens 400/544 e 0 to 1 dl 1459822661 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
[10027.155495] LustreError: 15c-8: MGC192.168.227.11@tcp1: The configuration from log 'demo-client' failed (-5). This may be the result of communication errors between this node and the MGS, a bad configuration, or other errors. See the syslog for more information.
[10027.207044] Lustre: Unmounted demo-client
[10027.214618] LustreError: 10212:0:(obd_mount.c:1342:lustre_fill_super()) Unable to mount (-5)
</pre>

The most common cause of failure is an improperly configured network interface, or LNet NID. Verify that the LNet protocol is able to communicate with the MGS with <code>lctl ping</code>:

<pre style="overflow-x:auto;">
lctl ping <MGS NID>
</pre>

If the ping fails, the command will return an I/O error:

<pre style="overflow-x:auto;">
[root@rh7z-c3 ~]# lctl ping 192.168.227.11@tcp1
failed to ping 192.168.227.11@tcp1: Input/output error
</pre>

Check the LNet settings before continuing.

If the ping succeeds, but the mount still fails, verify that the Lustre services are running on the target host. Also check to see if there are any services running on the client that might be interfering with communication, such as a firewall or SELinux. While SELinux is supported in Lustre 2.8 onwards, older releases of Lustre are not compatible. Temporarily disabling the firewall and SELinux can help narrow down the root cause of issues with Lustre communications.

If there are no OSS services online, but the MGS and the MDS for MDT0 are running, then the client mount command will hang indefinitely until an OSS service starts up.

There are options specific to Lustre that can be applied to the Lustre client <code>mount</code> command. The most common of these are <code>flock</code>, <code>localflock</code> and <code>user_xattr</code>:

* <code>flock</code>: enable support for cluster-wide, coherent file locks. Must be applied to the <code>mount</code> commands for all clients that will be accessing common data requiring lock functionality. Cluster-wide locking will have a detrimental impact on file system performance, and should only be enabled when absolutely required. For some applications, the locking is only necessary on a sub-set of nodes. For example, the CTDB cluster framework used by Samba to provide a parallel, high-availability SMB gateway, relies on locking of a shared file when coordinating cluster start-up and recovery. However, only the CTDB nodes need to mount the Lustre file system with the <code>flock</code> option. This is an example of application or domain-specific lock requirements.
* <code>localflock</code>: enable client-local <code>flock</code> support. This is much faster than cluster-wide <code>flock</code> support, but is only suitable for applications that require locks, but don’t run on multiple hosts (or where the data will not be accessed in a manner that would require locking across multiple hosts).
* <code>user_xattr</code>: Enable support for user extended attributes.

Additionally, consider using the <code>_netdev</code> mount option when mounting the Lustre client, especially when adding an entry into <code>/etc/fstab</code>. This option indicates to the operating system that the file system has a dependency on the network such that it should not be mounted before the network is online and should be unmounted on shutdown prior to stopping the network stack. An example entry for <code>/etc/fstab</code>:

<pre style="overflow-x:auto;">
192.168.227.11@tcp1:192.168.227.12@tcp1:/demo /lustre/demo lustre defaults,_netdev 0 0
</pre>

Refer to the <code>mount.lustre(8)</code> man page for more information on the available options.

[[Category:Lustre Systems Administration]]