Dynamic LNet Configuration and lnetctl - Revision history

Malcolm at 04:30, 11 September 2017

2017-09-11T04:30:12Z

← Older revision		Revision as of 21:30, 10 September 2017
Line 129:		Line 129:
	Configuration applied by <code>lnetctl</code> will be lost when the <code>lnet</code> kernel module is unloaded or the host is rebooted. To preserve the configuration, it can be exported to a file so that it can be re-imported when the host is restarted.		Configuration applied by <code>lnetctl</code> will be lost when the <code>lnet</code> kernel module is unloaded or the host is rebooted. To preserve the configuration, it can be exported to a file so that it can be re-imported when the host is restarted.

	To save an LNet configuration, use:		To save an LNet configuration, use either <code>lnetctl export</code> or <code>lnetctl net show</code>. The <code>export</code> sub-command syntax is:

	<pre style="overflow-x:auto;">		<pre style="overflow-x:auto;">
	lnetctl export [{file}]		lnetctl export [{file}]
			</pre>

			For example:

			<pre style="overflow-x:auto;">
			lnetctl export /etc/lnet.conf
	</pre>		</pre>

Line 160:		Line 166:
	</pre>		</pre>

	The output for the <code>export</code> command is the same as for <code>lnetctl net show --~~verbose~~</code>.		The output for the <code>export</code> command is the same as the output of <code>lnetctl net show --verbose</code>. The output also contains a lot of additional information that may not be required, such as the loopback device configuration. Because the <code>loopback</code> NID is always created by the LNet module, regardless of the supplied configuration, it is not needed in the exported file output. Unfortunately, there is not an easy way to exclude interfaces from the exported configuration and the <code>loopback</code> NID has to be removed by hand. In addition, the <code>lnetctl export</code> output includes some statistical data that is not relevant to describing a configuration, and some advanced options that may not be required for many installs.

			For a simpler configuration, use the <code>lnetctl net show</code> command:

			<pre style="overflow-x:auto;">
			lnetctl net show [--net <lnet>] [> {file}]
			</pre>

			For example, to capture the configuration of LNet <code>tcp0</code>:

			<pre style="overflow-x:auto;">
			lnetctl net show --net tcp0 > /etc/lnet.conf
			</pre>

			The file will not contain any tuning parameters, but for many, the defaults will be sufficient.

			In systems that support <code>systemd</code>, there is an LNet service definition, <code>lnet.service</code>, that can be enabled to load the LNet configuration on system boot. This is of most use to LNet routers, but can be enabled on any host. The <code>lnet</code> service will read the configuration from <code>/etc/lnet.conf</code>.

	All <code>lnetctl</code> output is formatted in YAML (Yet Another Markup Language), a relatively simple data structure syntax that is intended to be portable and has the benefit of being straightforward to interpret. Libraries for parsing YAML exist for a wide variety of programming languages.		All <code>lnetctl</code> output is formatted in YAML (Yet Another Markup Language), a relatively simple data structure syntax that is intended to be portable and has the benefit of being straightforward to interpret. Libraries for parsing YAML exist for a wide variety of programming languages.
Line 270:		Line 292:
	Mar 7 04:06:38 rh7z-pe kernel: LNetError: 2678:0:(api-ni.c:1252:lnet_startup_lndni()) Net tcp1 is not unique		Mar 7 04:06:38 rh7z-pe kernel: LNetError: 2678:0:(api-ni.c:1252:lnet_startup_lndni()) Net tcp1 is not unique
	</pre>		</pre>

	Note that <code>lnetctl export</code> will include the <code>loopback</code> interface in the output that it generates. Because the <code>loopback</code> NID is always created by the LNet module, regardless of the supplied configuration, it is not needed in the exported file output and LNet will actually generate an error when an attempt is made to import a configuration that contains the <code>loopback</code> NID. The error will not prevent the rest of the configuration file from being processed, but it does add unneeded noise to the host’s log files and can be disconcerting to users who are not expecting this behavior. Unfortunately, there is not an easy way to exclude interfaces from the exported configuration and the <code>loopback</code> NID has to be removed by hand.

	The <code>import --del</code> sub-command will attempt to delete any NIDS configured on the host that match the specification described in the YAML configuration file. If the command is successful, then it will not return any output to the shell, but there will be an entry logged to the system console and to syslog.		The <code>import --del</code> sub-command will attempt to delete any NIDS configured on the host that match the specification described in the YAML configuration file. If the command is successful, then it will not return any output to the shell, but there will be an entry logged to the system console and to syslog.

Malcolm at 05:47, 31 August 2017

2017-08-31T05:47:00Z

← Older revision		Revision as of 22:47, 30 August 2017
Line 331:		Line 331:

	[[Category:Lustre Networking (LNet)]]		[[Category:Lustre Networking (LNet)]]
			[[Category:Lustre Systems Administration]]

Malcolm: Created page with "Administrators of Lustre file systems will be most familiar with using the `modprobe` interface to configure LNet devices (described Static LNet Configuration),..."

2017-08-10T03:04:55Z

Created page with "Administrators of Lustre file systems will be most familiar with using the <code>modprobe</code> interface to configure LNet devices (described Static LNet Configuration),..."

New page

Administrators of Lustre file systems will be most familiar with using the <code>modprobe</code> interface to configure LNet devices (described [[Static LNet Configuration]]), and this is the only mechanism available for LNet configuration in versions of Lustre prior to 2.7.0. In these older release (2.6.0 and earlier), configuring a new LNet or changing the properties of an existing LNet interface means unloading and reloading the kernel module, which effectively incurs a temporary outage on the the node being altered.

However, with the release of Lustre version 2.7.0, there is a new utility, called <code>lnetctl</code>, that offers a more flexible and powerful way to manage LNet configuration. With this new utility, LNet can also be updated while the kernel module is still running, a feature referred to as Dynamic LNet Configuration (DLC). Amongst other things, Dynamic LNet means tuning can be applied to an LNet interface on a host without incurring an outage.

The <code>lnetctl</code> utility can completely replace the older <code>modprobe</code> configuration method, but it is not mandatory; administrators can choose the tools that best match their needs. <code>lnetctl</code> is straightforward to use and helps to guide administrators to a valid, working configuration. The <code>lnetctl</code> man page has a comprehensive description of the configuration commands and options, as does the Lustre Operations Manual.

Dynamic LNet configuration is powerful and versatile, and provides administrators with easy tools to view and alter the running configuration of LNet on a host.

The LNet modules need to be loaded into the kernel prior to making any configuration changes using <code>lnetctl</code>. To load the <code>lnet</code> modules, use the <code>modprobe</code> command:

<pre style="overflow-x:auto;">
modprobe [-v] lnet
</pre>

For example:

<pre style="overflow-x:auto;">
[root@rh7z-pe ~]# modprobe -v lnet
insmod /lib/modules/3.10.0-327.13.1.el7_lustre.x86_64/extra/kernel/net/lustre/libcfs.ko
insmod /lib/modules/3.10.0-327.13.1.el7_lustre.x86_64/extra/kernel/net/lustre/lnet.ko
</pre>

After the modules are loaded, run the <code>lnet configure</code> sub-command to initialize the LNet service in the kernel:

<pre style="overflow-x:auto;">
lnetctl lnet configure [--all]
</pre>

If the <code>--all</code> flag is used, <code>lnetctl</code> will try to load any LNet interfaces referenced as either <code>networks</code> or <code>ip2nets</code> options in the <code>modprobe</code> configuration files. Otherwise, <code>lnetctl lnet configure</code> will not attempt to initialize any networks. When using Dynamic LNet Configuration to manage the LNet interface for a host, using the <code>--all</code> flag is not recommended unless one is migrating from a legacy configuration with complex module options defined.

If there is no other configuration on the server, LNet will have a reference to the <code>loopback</code> interface and nothing else. For example:

<pre style="overflow-x:auto;">
[root@rh7z-pe ~]# lnetctl net show
net:
- net: lo
nid: 0@lo
status: up
</pre>

If the LNet kernel module is not loaded, then <code>lnetctl</code> will report an error when any of the commands are executed:

<pre style="overflow-x:auto;">
[root@rh7z-pe ~]# lnetctl lnet configure
opening /dev/lnet failed: No such device
hint: the kernel modules may not be loaded
configure:
- lnet:
errno: -19
descr: "LNet configure error: No such device"
</pre>

After loading the kernel module and running the <code>lnetctl lnet configure</code> command, <code>lnetctl</code> can now be used to create new LNet interfaces on the host. The syntax is explained in the <code>lnetctl</code> man page, but the basic format follows:

<pre style="overflow-x:auto;">
lnetctl net add --net <lnet name> --if <net interface> [<options> …]
</pre>

Here is a simple example that creates a new NID configuration for LNet <code>tcp1</code> using the TCP/IP sockets LND on <code>eth1</code>:

<pre style="overflow-x:auto;">
lnetctl net add --net tcp1 --if eth1
</pre>

This next example creates a new NID for an InfiniBand device using the OFED LND:

<pre style="overflow-x:auto;">
lnetctl net add --net o2ib0 --if ib0
</pre>

To delete a NID from the LNet configuration, use the <code>lctl net del</code> command:

<pre style="overflow-x:auto;">
lnetctl net del --net <lnet name>
</pre>

For example:

<pre style="overflow-x:auto;">
lnetctl net del --net tcp1
</pre>

To review the currently configured NIDs, use:

<pre style="overflow-x:auto;">
lnetctl net show [--verbose]
</pre>

For example:

<pre style="overflow-x:auto;">
[root@rh7z-pe ~]# lnetctl net show
net:
- net: lo
nid: 0@lo
status: up
- net: tcp
nid: 192.168.207.2@tcp
status: up
interfaces:
0: eth1
</pre>

When invoked without any options, the <code>lnetctl</code> command also provides a command shell for interactive configuration:

<pre style="overflow-x:auto;">
[root@rh7z-pe ~]# lnetctl
lnetctl > help

Available commands are:
lnet
route
net
routing
set
import
export
stats
peer_credits
help
exit
quit
For more help type: help command-name
lnetctl > net
net {add | del | show | help}
lnetctl >
</pre>

Configuration applied by <code>lnetctl</code> will be lost when the <code>lnet</code> kernel module is unloaded or the host is rebooted. To preserve the configuration, it can be exported to a file so that it can be re-imported when the host is restarted.

To save an LNet configuration, use:

<pre style="overflow-x:auto;">
lnetctl export [{file}]
</pre>

If a file is not specified, the configuration is directed to <code><stdout></code>. For example:

<pre style="overflow-x:auto;">
[root@rh7z-pe ~]# lnetctl export
net:
- net: lo
nid: 0@lo
status: up
tunables:
peer_timeout: 0
peer_credits: 0
peer_buffer_credits: 0
credits: 0
- net: tcp1
nid: 192.168.207.2@tcp1
status: up
interfaces:
0: eth1
tunables:
peer_timeout: 180
peer_credits: 8
peer_buffer_credits: 0
credits: 256
</pre>

The output for the <code>export</code> command is the same as for <code>lnetctl net show --verbose</code>.

All <code>lnetctl</code> output is formatted in YAML (Yet Another Markup Language), a relatively simple data structure syntax that is intended to be portable and has the benefit of being straightforward to interpret. Libraries for parsing YAML exist for a wide variety of programming languages.

The <code>export</code> command is a great way to migrate hosts that have been using the kernel module options files via <code>modprobe</code> to use Dynamic LNet configuration. If a host has a running configuration that was provided using kernel module options, it can be exported using <code>lnetctl</code> into the YAML syntax.

There is, of course, a corresponding <code>import</code> command for <code>lnetctl</code>:

<pre style="overflow-x:auto;">
lnetctl import [--add|--del|--show] {file}
</pre>

The file used as input must be formatted in YAML. The import command will also parse YAML input from <code><stdin></code>. The following example shows a simple configuration in YAML syntax:

<pre style="overflow-x:auto;">
net:
- net: tcp1
interfaces:
0: eth1
</pre>

This YAML configuration is equivalent to the following, expressed as a <code>modprobe</code> option:

<pre style="overflow-x:auto;">
options lnet networks="tcp1(eth1)"
</pre>

When a configuration is exported by <code>lnetctl</code>, it will also include host-specific information such as the NID, interface status and tunables. However, the basic configuration need only describe the LNet (e.g. <code>tcp1</code>, <code>o2ib0</code>) and the interfaces, as above.

'''Note:''' To provide an effective and simple means to audit a host’s LNet configuration, consider using the <code>lnetctl --add</code> command to create the LNet configuration for the first time, then use <code>lnetctl export</code> to capture a persistent record. In that way, one can create a tailored audit per host to verify that the recorded and running configurations match.

The default behavior of the <code>import</code> command is to add the interfaces described in the configuration file. The following two commands are therefore equivalent:

<pre style="overflow-x:auto;">
lnetctl import --add {file}
lnetctl import {file}
</pre>

The following transcript shows an example of how the <code>lnetctl import --add</code> command works:

<pre style="overflow-x:auto;">
[root@rh7z-pe ~]# modprobe lnet # load the lnet kernel module
[root@rh7z-pe ~]# lnetctl lnet configure # start the lnet service
[root@rh7z-pe ~]# lnetctl net show
net:
- net: lo
nid: 0@lo
status: up
[root@rh7z-pe ~]# cat lnet.cf
net:
- net: tcp1
nid: 192.168.207.2@tcp1
status: up
interfaces:
0: eth1
tunables:
peer_timeout: 180
peer_credits: 8
peer_buffer_credits: 0
credits: 256
[root@rh7z-pe ~]# lnetctl import --add lnet.cf
[root@rh7z-pe ~]# lnetctl net show
net:
- net: lo
nid: 0@lo
status: up
- net: tcp1
nid: 192.168.207.2@tcp1
status: up
interfaces:
0: eth1
</pre>

If an LNet NID is already configured, then <code>lnetctl import [--add]</code> will return an error. For example, the following transcript shows an attempt to import a configuration for <code>tcp1(eth1)</code> on a host where the NID is already configured:

<pre style="overflow-x:auto;">
[root@rh7z-pe ~]# lnetctl net show
net:
- net: lo
nid: 0@lo
status: up
- net: tcp
nid: 192.168.207.2@tcp1
status: up
interfaces:
0: eth1
[root@rh7z-pe ~]# cat lnet.cf
net:
- net: tcp
nid: 192.168.207.2@tcp1
status: up
interfaces:
0: eth1
tunables:
peer_timeout: 180
peer_credits: 8
peer_buffer_credits: 0
credits: 256
[root@rh7z-pe ~]# lnetctl import lnet.cf
add:
- net:
errno: -17
descr: "cannot add network: File exists"
</pre>

The kernel will also report an error to the ring buffer and system console:

<pre style="overflow-x:auto;">
Mar 7 04:06:38 rh7z-pe kernel: LNetError: 2678:0:(api-ni.c:1252:lnet_startup_lndni()) Net tcp1 is not unique
</pre>

Note that <code>lnetctl export</code> will include the <code>loopback</code> interface in the output that it generates. Because the <code>loopback</code> NID is always created by the LNet module, regardless of the supplied configuration, it is not needed in the exported file output and LNet will actually generate an error when an attempt is made to import a configuration that contains the <code>loopback</code> NID. The error will not prevent the rest of the configuration file from being processed, but it does add unneeded noise to the host’s log files and can be disconcerting to users who are not expecting this behavior. Unfortunately, there is not an easy way to exclude interfaces from the exported configuration and the <code>loopback</code> NID has to be removed by hand.

The <code>import --del</code> sub-command will attempt to delete any NIDS configured on the host that match the specification described in the YAML configuration file. If the command is successful, then it will not return any output to the shell, but there will be an entry logged to the system console and to syslog.

For example, if a file called <code>lnet.cf</code> contained an entry for NID <code>192.168.207.2@tcp1</code>, then the command:

<pre style="overflow-x:auto;">
[root@rh7z-pe ~]# lnetctl import --del lnet.cf
</pre>

...will create an entry in the kernel ring buffer and syslog similar to the following:

<pre style="overflow-x:auto;">
Mar 7 04:46:00 rh7z-pe kernel: LNet: Removed LNI 192.168.207.2@tcp1
</pre>

Any mismatched entries will generate an error – that is, if there is an entry in the YAML configuration that does not match a configured NID on the host, <code>lnetctl</code> will return an error for each such entry:

<pre style="overflow-x:auto;">
[root@rh7z-pe ~]# lnetctl import --del ln-mismatch.cf
del:
- net:
errno: -22
descr: "cannot delete network: Invalid argument"
</pre>

The <code>lnetctl import --show</code> command is used to identify which of the interfaces defined in the configuration file are actually active LNet NIDs configured on the host. If there are no matching NIDs, <code>lnetctl import --show</code> will not return any output:

<pre style="overflow-x:auto;">
[root@rh7z-pe ~]# lnetctl net show
net:
- net: lo
nid: 0@lo
status: up
[root@rh7z-pe ~]# cat lnet.cf
net:
- net: tcp1
nid: 192.168.207.2@tcp1
status: up
interfaces:
0: eth1
tunables:
peer_timeout: 180
peer_credits: 8
peer_buffer_credits: 0
credits: 256
[root@rh7z-pe ~]# lnetctl import --show lnet.cf
[root@rh7z-pe ~]# lnetctl import --add lnet.cf
[root@rh7z-pe ~]# lnetctl import --show lnet.cf
net:
- net: tcp1
nid: 192.168.207.2@tcp1
status: up
interfaces:
0: eth1
</pre>

In the above example, the <code>lnet</code> kernel module on the host initially has only the <code>loopback</code> NID configured. When <code>lnetctl import --show</code> is run for the first time, it does not find a NID on the host that corresponds to the configuration, so it does not return any matches in its output. After the configuration is added to LNet, the second invocation of <code>lnetctl import --show</code> lists a match. The <code>lnetctl import --show</code> command is most useful when debugging more complex configurations with the YAML syntax.

[[Category:Lustre Networking (LNet)]]

Dynamic LNet Configuration and lnetctl - Revision history

Malcolm at 04:30, 11 September 2017

Malcolm at 05:47, 31 August 2017

Malcolm: Created page with "Administrators of Lustre file systems will be most familiar with using the modprobe interface to configure LNet devices (described Static LNet Configuration),..."

Malcolm: Created page with "Administrators of Lustre file systems will be most familiar with using the `modprobe` interface to configure LNet devices (described Static LNet Configuration),..."