Dynamic LNet Configuration and lnetctl
Administrators of Lustre file systems will be most familiar with using the modprobe interface to configure LNet devices (described Static LNet Configuration), and this is the only mechanism available for LNet configuration in versions of Lustre prior to 2.7.0. In these older release (2.6.0 and earlier), configuring a new LNet or changing the properties of an existing LNet interface means unloading and reloading the kernel module, which effectively incurs a temporary outage on the the node being altered.
However, with the release of Lustre version 2.7.0, there is a new utility, called lnetctl, that offers a more flexible and powerful way to manage LNet configuration. With this new utility, LNet can also be updated while the kernel module is still running, a feature referred to as Dynamic LNet Configuration (DLC). Amongst other things, Dynamic LNet means tuning can be applied to an LNet interface on a host without incurring an outage.
The lnetctl utility can completely replace the older modprobe configuration method, but it is not mandatory; administrators can choose the tools that best match their needs. lnetctl is straightforward to use and helps to guide administrators to a valid, working configuration. The lnetctl man page has a comprehensive description of the configuration commands and options, as does the Lustre Operations Manual.
Dynamic LNet configuration is powerful and versatile, and provides administrators with easy tools to view and alter the running configuration of LNet on a host.
The LNet modules need to be loaded into the kernel prior to making any configuration changes using lnetctl. To load the lnet modules, use the modprobe command:
modprobe [-v] lnet
For example:
[root@rh7z-pe ~]# modprobe -v lnet insmod /lib/modules/3.10.0-327.13.1.el7_lustre.x86_64/extra/kernel/net/lustre/libcfs.ko insmod /lib/modules/3.10.0-327.13.1.el7_lustre.x86_64/extra/kernel/net/lustre/lnet.ko
After the modules are loaded, run the lnet configure sub-command to initialize the LNet service in the kernel:
lnetctl lnet configure [--all]
If the --all flag is used, lnetctl will try to load any LNet interfaces referenced as either networks or ip2nets options in the modprobe configuration files. Otherwise, lnetctl lnet configure will not attempt to initialize any networks. When using Dynamic LNet Configuration to manage the LNet interface for a host, using the --all flag is not recommended unless one is migrating from a legacy configuration with complex module options defined.
If there is no other configuration on the server, LNet will have a reference to the loopback interface and nothing else. For example:
[root@rh7z-pe ~]# lnetctl net show
net:
- net: lo
nid: 0@lo
status: up
If the LNet kernel module is not loaded, then lnetctl will report an error when any of the commands are executed:
[root@rh7z-pe ~]# lnetctl lnet configure
opening /dev/lnet failed: No such device
hint: the kernel modules may not be loaded
configure:
- lnet:
errno: -19
descr: "LNet configure error: No such device"
After loading the kernel module and running the lnetctl lnet configure command, lnetctl can now be used to create new LNet interfaces on the host. The syntax is explained in the lnetctl man page, but the basic format follows:
lnetctl net add --net <lnet name> --if <net interface> [<options> …]
Here is a simple example that creates a new NID configuration for LNet tcp1 using the TCP/IP sockets LND on eth1:
lnetctl net add --net tcp1 --if eth1
This next example creates a new NID for an InfiniBand device using the OFED LND:
lnetctl net add --net o2ib0 --if ib0
To delete a NID from the LNet configuration, use the lctl net del command:
lnetctl net del --net <lnet name>
For example:
lnetctl net del --net tcp1
To review the currently configured NIDs, use:
lnetctl net show [--verbose]
For example:
[root@rh7z-pe ~]# lnetctl net show
net:
- net: lo
nid: 0@lo
status: up
- net: tcp
nid: 192.168.207.2@tcp
status: up
interfaces:
0: eth1
When invoked without any options, the lnetctl command also provides a command shell for interactive configuration:
[root@rh7z-pe ~]# lnetctl
lnetctl > help
Available commands are:
lnet
route
net
routing
set
import
export
stats
peer_credits
help
exit
quit
For more help type: help command-name
lnetctl > net
net {add | del | show | help}
lnetctl >
Configuration applied by lnetctl will be lost when the lnet kernel module is unloaded or the host is rebooted. To preserve the configuration, it can be exported to a file so that it can be re-imported when the host is restarted.
To save an LNet configuration, use either lnetctl export or lnetctl net show. The export sub-command syntax is:
lnetctl export [{file}]
For example:
lnetctl export /etc/lnet.conf
If a file is not specified, the configuration is directed to <stdout>. For example:
[root@rh7z-pe ~]# lnetctl export
net:
- net: lo
nid: 0@lo
status: up
tunables:
peer_timeout: 0
peer_credits: 0
peer_buffer_credits: 0
credits: 0
- net: tcp1
nid: 192.168.207.2@tcp1
status: up
interfaces:
0: eth1
tunables:
peer_timeout: 180
peer_credits: 8
peer_buffer_credits: 0
credits: 256
The output for the export command is the same as the output of lnetctl net show --verbose. The output also contains a lot of additional information that may not be required, such as the loopback device configuration. Because the loopback NID is always created by the LNet module, regardless of the supplied configuration, it is not needed in the exported file output. Unfortunately, there is not an easy way to exclude interfaces from the exported configuration and the loopback NID has to be removed by hand. In addition, the lnetctl export output includes some statistical data that is not relevant to describing a configuration, and some advanced options that may not be required for many installs.
For a simpler configuration, use the lnetctl net show command:
lnetctl net show [--net <lnet>] [> {file}]
For example, to capture the configuration of LNet tcp0:
lnetctl net show --net tcp0 > /etc/lnet.conf
The file will not contain any tuning parameters, but for many, the defaults will be sufficient.
In systems that support systemd, there is an LNet service definition, lnet.service, that can be enabled to load the LNet configuration on system boot. This is of most use to LNet routers, but can be enabled on any host. The lnet service will read the configuration from /etc/lnet.conf.
All lnetctl output is formatted in YAML (Yet Another Markup Language), a relatively simple data structure syntax that is intended to be portable and has the benefit of being straightforward to interpret. Libraries for parsing YAML exist for a wide variety of programming languages.
The export command is a great way to migrate hosts that have been using the kernel module options files via modprobe to use Dynamic LNet configuration. If a host has a running configuration that was provided using kernel module options, it can be exported using lnetctl into the YAML syntax.
There is, of course, a corresponding import command for lnetctl:
lnetctl import [--add|--del|--show] {file}
The file used as input must be formatted in YAML. The import command will also parse YAML input from <stdin>. The following example shows a simple configuration in YAML syntax:
net:
- net: tcp1
interfaces:
0: eth1
This YAML configuration is equivalent to the following, expressed as a modprobe option:
options lnet networks="tcp1(eth1)"
When a configuration is exported by lnetctl, it will also include host-specific information such as the NID, interface status and tunables. However, the basic configuration need only describe the LNet (e.g. tcp1, o2ib0) and the interfaces, as above.
Note: To provide an effective and simple means to audit a host’s LNet configuration, consider using the lnetctl --add command to create the LNet configuration for the first time, then use lnetctl export to capture a persistent record. In that way, one can create a tailored audit per host to verify that the recorded and running configurations match.
The default behavior of the import command is to add the interfaces described in the configuration file. The following two commands are therefore equivalent:
lnetctl import --add {file}
lnetctl import {file}
The following transcript shows an example of how the lnetctl import --add command works:
[root@rh7z-pe ~]# modprobe lnet # load the lnet kernel module
[root@rh7z-pe ~]# lnetctl lnet configure # start the lnet service
[root@rh7z-pe ~]# lnetctl net show
net:
- net: lo
nid: 0@lo
status: up
[root@rh7z-pe ~]# cat lnet.cf
net:
- net: tcp1
nid: 192.168.207.2@tcp1
status: up
interfaces:
0: eth1
tunables:
peer_timeout: 180
peer_credits: 8
peer_buffer_credits: 0
credits: 256
[root@rh7z-pe ~]# lnetctl import --add lnet.cf
[root@rh7z-pe ~]# lnetctl net show
net:
- net: lo
nid: 0@lo
status: up
- net: tcp1
nid: 192.168.207.2@tcp1
status: up
interfaces:
0: eth1
If an LNet NID is already configured, then lnetctl import [--add] will return an error. For example, the following transcript shows an attempt to import a configuration for tcp1(eth1) on a host where the NID is already configured:
[root@rh7z-pe ~]# lnetctl net show
net:
- net: lo
nid: 0@lo
status: up
- net: tcp
nid: 192.168.207.2@tcp1
status: up
interfaces:
0: eth1
[root@rh7z-pe ~]# cat lnet.cf
net:
- net: tcp
nid: 192.168.207.2@tcp1
status: up
interfaces:
0: eth1
tunables:
peer_timeout: 180
peer_credits: 8
peer_buffer_credits: 0
credits: 256
[root@rh7z-pe ~]# lnetctl import lnet.cf
add:
- net:
errno: -17
descr: "cannot add network: File exists"
The kernel will also report an error to the ring buffer and system console:
Mar 7 04:06:38 rh7z-pe kernel: LNetError: 2678:0:(api-ni.c:1252:lnet_startup_lndni()) Net tcp1 is not unique
The import --del sub-command will attempt to delete any NIDS configured on the host that match the specification described in the YAML configuration file. If the command is successful, then it will not return any output to the shell, but there will be an entry logged to the system console and to syslog.
For example, if a file called lnet.cf contained an entry for NID 192.168.207.2@tcp1, then the command:
[root@rh7z-pe ~]# lnetctl import --del lnet.cf
...will create an entry in the kernel ring buffer and syslog similar to the following:
Mar 7 04:46:00 rh7z-pe kernel: LNet: Removed LNI 192.168.207.2@tcp1
Any mismatched entries will generate an error – that is, if there is an entry in the YAML configuration that does not match a configured NID on the host, lnetctl will return an error for each such entry:
[root@rh7z-pe ~]# lnetctl import --del ln-mismatch.cf
del:
- net:
errno: -22
descr: "cannot delete network: Invalid argument"
The lnetctl import --show command is used to identify which of the interfaces defined in the configuration file are actually active LNet NIDs configured on the host. If there are no matching NIDs, lnetctl import --show will not return any output:
[root@rh7z-pe ~]# lnetctl net show
net:
- net: lo
nid: 0@lo
status: up
[root@rh7z-pe ~]# cat lnet.cf
net:
- net: tcp1
nid: 192.168.207.2@tcp1
status: up
interfaces:
0: eth1
tunables:
peer_timeout: 180
peer_credits: 8
peer_buffer_credits: 0
credits: 256
[root@rh7z-pe ~]# lnetctl import --show lnet.cf
[root@rh7z-pe ~]# lnetctl import --add lnet.cf
[root@rh7z-pe ~]# lnetctl import --show lnet.cf
net:
- net: tcp1
nid: 192.168.207.2@tcp1
status: up
interfaces:
0: eth1
In the above example, the lnet kernel module on the host initially has only the loopback NID configured. When lnetctl import --show is run for the first time, it does not find a NID on the host that corresponds to the configuration, so it does not return any matches in its output. After the configuration is added to LNet, the second invocation of lnetctl import --show lists a match. The lnetctl import --show command is most useful when debugging more complex configurations with the YAML syntax.