Dynamic LNet Configuration and lnetctl

Administrators of Lustre file systems will be most familiar with using the  interface to configure LNet devices (described Static LNet Configuration), and this is the only mechanism available for LNet configuration in versions of Lustre prior to 2.7.0. In these older release (2.6.0 and earlier), configuring a new LNet or changing the properties of an existing LNet interface means unloading and reloading the kernel module, which effectively incurs a temporary outage on the the node being altered.

However, with the release of Lustre version 2.7.0, there is a new utility, called, that offers a more flexible and powerful way to manage LNet configuration. With this new utility, LNet can also be updated while the kernel module is still running, a feature referred to as Dynamic LNet Configuration (DLC). Amongst other things, Dynamic LNet means tuning can be applied to an LNet interface on a host without incurring an outage.

The  utility can completely replace the older   configuration method, but it is not mandatory; administrators can choose the tools that best match their needs. is straightforward to use and helps to guide administrators to a valid, working configuration. The  man page has a comprehensive description of the configuration commands and options, as does the Lustre Operations Manual.

Dynamic LNet configuration is powerful and versatile, and provides administrators with easy tools to view and alter the running configuration of LNet on a host.

The LNet modules need to be loaded into the kernel prior to making any configuration changes using. To load the  modules, use the   command:

 modprobe [-v] lnet

For example:

 [root@rh7z-pe ~]# modprobe -v lnet insmod /lib/modules/3.10.0-327.13.1.el7_lustre.x86_64/extra/kernel/net/lustre/libcfs.ko insmod /lib/modules/3.10.0-327.13.1.el7_lustre.x86_64/extra/kernel/net/lustre/lnet.ko

After the modules are loaded, run the  sub-command to initialize the LNet service in the kernel:

 lnetctl lnet configure [--all]

If the  flag is used,   will try to load any LNet interfaces referenced as either   or   options in the   configuration files. Otherwise,  will not attempt to initialize any networks. When using Dynamic LNet Configuration to manage the LNet interface for a host, using the  flag is not recommended unless one is migrating from a legacy configuration with complex module options defined.

If there is no other configuration on the server, LNet will have a reference to the  interface and nothing else. For example:

 [root@rh7z-pe ~]# lnetctl net show net: - net: lo     nid: 0@lo status: up

If the LNet kernel module is not loaded, then  will report an error when any of the commands are executed:

 [root@rh7z-pe ~]# lnetctl lnet configure opening /dev/lnet failed: No such device hint: the kernel modules may not be loaded configure: - lnet: errno: -19 descr: "LNet configure error: No such device"

After loading the kernel module and running the  command,   can now be used to create new LNet interfaces on the host. The syntax is explained in the  man page, but the basic format follows:

 lnetctl net add --net --if [ …]

Here is a simple example that creates a new NID configuration for LNet  using the TCP/IP sockets LND on  :

 lnetctl net add --net tcp1 --if eth1

This next example creates a new NID for an InfiniBand device using the OFED LND:

 lnetctl net add --net o2ib0 --if ib0

To delete a NID from the LNet configuration, use the  command:

 lnetctl net del --net

For example:

 lnetctl net del --net tcp1

To review the currently configured NIDs, use:

 lnetctl net show [--verbose]

For example:

 [root@rh7z-pe ~]# lnetctl net show net: - net: lo     nid: 0@lo status: up   - net: tcp nid: 192.168.207.2@tcp status: up     interfaces: 0: eth1

When invoked without any options, the  command also provides a command shell for interactive configuration:

 [root@rh7z-pe ~]# lnetctl lnetctl > help

Available commands are: lnet route net routing set import export stats peer_credits help exit quit For more help type: help command-name lnetctl > net net {add | del | show | help} lnetctl >

Configuration applied by  will be lost when the   kernel module is unloaded or the host is rebooted. To preserve the configuration, it can be exported to a file so that it can be re-imported when the host is restarted.

To save an LNet configuration, use:

 lnetctl export [{file}]

If a file is not specified, the configuration is directed to. For example:

 [root@rh7z-pe ~]# lnetctl export net: - net: lo     nid: 0@lo status: up     tunables: peer_timeout: 0 peer_credits: 0 peer_buffer_credits: 0 credits: 0 - net: tcp1 nid: 192.168.207.2@tcp1 status: up     interfaces: 0: eth1 tunables: peer_timeout: 180 peer_credits: 8 peer_buffer_credits: 0 credits: 256

The output for the  command is the same as for.

All  output is formatted in YAML (Yet Another Markup Language), a relatively simple data structure syntax that is intended to be portable and has the benefit of being straightforward to interpret. Libraries for parsing YAML exist for a wide variety of programming languages.

The  command is a great way to migrate hosts that have been using the kernel module options files via   to use Dynamic LNet configuration. If a host has a running configuration that was provided using kernel module options, it can be exported using  into the YAML syntax.

There is, of course, a corresponding  command for  :

 lnetctl import [--add|--del|--show] {file}

The file used as input must be formatted in YAML. The import command will also parse YAML input from. The following example shows a simple configuration in YAML syntax:

<pre style="overflow-x:auto;"> net: - net: tcp1 interfaces: 0: eth1

This YAML configuration is equivalent to the following, expressed as a  option:

<pre style="overflow-x:auto;"> options lnet networks="tcp1(eth1)"

When a configuration is exported by, it will also include host-specific information such as the NID, interface status and tunables. However, the basic configuration need only describe the LNet (e.g.,  ) and the interfaces, as above.

Note: To provide an effective and simple means to audit a host’s LNet configuration, consider using the  command to create the LNet configuration for the first time, then use   to capture a persistent record. In that way, one can create a tailored audit per host to verify that the recorded and running configurations match.

The default behavior of the  command is to add the interfaces described in the configuration file. The following two commands are therefore equivalent:

<pre style="overflow-x:auto;"> lnetctl import --add {file} lnetctl import {file}

The following transcript shows an example of how the  command works:

<pre style="overflow-x:auto;"> [root@rh7z-pe ~]# modprobe lnet # load the lnet kernel module [root@rh7z-pe ~]# lnetctl lnet configure # start the lnet service [root@rh7z-pe ~]# lnetctl net show net: - net: lo     nid: 0@lo status: up [root@rh7z-pe ~]# cat lnet.cf net: - net: tcp1 nid: 192.168.207.2@tcp1 status: up     interfaces: 0: eth1 tunables: peer_timeout: 180 peer_credits: 8 peer_buffer_credits: 0 credits: 256 [root@rh7z-pe ~]# lnetctl import --add lnet.cf [root@rh7z-pe ~]# lnetctl net show net: - net: lo     nid: 0@lo status: up   - net: tcp1 nid: 192.168.207.2@tcp1 status: up     interfaces: 0: eth1

If an LNet NID is already configured, then  will return an error. For example, the following transcript shows an attempt to import a configuration for  on a host where the NID is already configured:

<pre style="overflow-x:auto;"> [root@rh7z-pe ~]# lnetctl net show net: - net: lo     nid: 0@lo status: up   - net: tcp nid: 192.168.207.2@tcp1 status: up     interfaces: 0: eth1 [root@rh7z-pe ~]# cat lnet.cf net: - net: tcp nid: 192.168.207.2@tcp1 status: up     interfaces: 0: eth1 tunables: peer_timeout: 180 peer_credits: 8 peer_buffer_credits: 0 credits: 256 [root@rh7z-pe ~]# lnetctl import lnet.cf add: - net: errno: -17 descr: "cannot add network: File exists"

The kernel will also report an error to the ring buffer and system console:

<pre style="overflow-x:auto;"> Mar 7 04:06:38 rh7z-pe kernel: LNetError: 2678:0:(api-ni.c:1252:lnet_startup_lndni) Net tcp1 is not unique

Note that  will include the   interface in the output that it generates. Because the  NID is always created by the LNet module, regardless of the supplied configuration, it is not needed in the exported file output and LNet will actually generate an error when an attempt is made to import a configuration that contains the   NID. The error will not prevent the rest of the configuration file from being processed, but it does add unneeded noise to the host’s log files and can be disconcerting to users who are not expecting this behavior. Unfortunately, there is not an easy way to exclude interfaces from the exported configuration and the  NID has to be removed by hand.

The  sub-command will attempt to delete any NIDS configured on the host that match the specification described in the YAML configuration file. If the command is successful, then it will not return any output to the shell, but there will be an entry logged to the system console and to syslog.

For example, if a file called  contained an entry for NID , then the command:

<pre style="overflow-x:auto;"> [root@rh7z-pe ~]# lnetctl import --del lnet.cf

...will create an entry in the kernel ring buffer and syslog similar to the following:

<pre style="overflow-x:auto;"> Mar 7 04:46:00 rh7z-pe kernel: LNet: Removed LNI 192.168.207.2@tcp1

Any mismatched entries will generate an error – that is, if there is an entry in the YAML configuration that does not match a configured NID on the host,  will return an error for each such entry:

<pre style="overflow-x:auto;"> [root@rh7z-pe ~]# lnetctl import --del ln-mismatch.cf del: - net: errno: -22 descr: "cannot delete network: Invalid argument"

The  command is used to identify which of the interfaces defined in the configuration file are actually active LNet NIDs configured on the host. If there are no matching NIDs,  will not return any output:

<pre style="overflow-x:auto;"> [root@rh7z-pe ~]# lnetctl net show net: - net: lo     nid: 0@lo status: up [root@rh7z-pe ~]# cat lnet.cf net: - net: tcp1 nid: 192.168.207.2@tcp1 status: up     interfaces: 0: eth1 tunables: peer_timeout: 180 peer_credits: 8 peer_buffer_credits: 0 credits: 256 [root@rh7z-pe ~]# lnetctl import --show lnet.cf [root@rh7z-pe ~]# lnetctl import --add lnet.cf [root@rh7z-pe ~]# lnetctl import --show lnet.cf net: - net: tcp1 nid: 192.168.207.2@tcp1 status: up     interfaces: 0: eth1

In the above example, the  kernel module on the host initially has only the   NID configured. When  is run for the first time, it does not find a NID on the host that corresponds to the configuration, so it does not return any matches in its output. After the configuration is added to LNet, the second invocation of  lists a match. The  command is most useful when debugging more complex configurations with the YAML syntax.