Static LNet Configuration

From Lustre Wiki
Jump to navigation Jump to search

In releases of Lustre prior to version 2.7.0, configuration of the LNet device driver is managed exclusively by supplying options to the lnet kernel module. The options are read from a modules options (modprobe) configuration file when the kernel module is first loaded. With the release of Lustre 2.7.0, administrators can instead make use of the Dynamic LNet Configuration feature, controlled by a utility called lnetctl, as an alternative to static configuration.

Refer to the article Lustre Networking (LNET) Overview for a general introduction to LNet.

This article describes static configuration of LNet.


An LNet interface can be configured by supplying parameters directly to the lnet module when it is loaded by the kernel. To do this, create a file in the normal modprobe configuration options directory (usually /etc/modprobe.d), and add an entry for the lnet module. By convention, Lustre module configuration options are recorded in a file called /etc/modprobe.d/lustre.conf (although tuning parameters for devices can be separated into device-specific files, such as /etc/modprobe.d/ko2iblnd.conf – see the section Optimizing o2iblnd Performance for an example of this, which shows an optimized configuration that is automatically applied when an Intel® Omni-Path interface is detected).

There are two ways to define an LNet configuration using module options: networks and ip2nets. The networks parameter is the easiest to understand and uses a very simple syntax, while ip2nets gives administrators the greatest flexibility at the expense of much greater complexity. The networks syntax is strongly recommended unless there is a specific requirement that can be more readily fulfilled by the ip2nets syntax.

There should be only one networks or ip2nets parameter defined for the lnet module. The parameters are mutually exclusive; choose one or the other but not both. If both options are referenced in the modules configuration, then the following warning will be reported in the kernel ring buffer (dmesg) and system console:

[46290.641911] LNetError: 101-0: Please specify EITHER 'networks' or 'ip2nets' but not both at once
[46290.645288] LNetError: 3483:0:(config.c:199:lnet_parse_networks()) networks string is undefined

The second line in the error output is a side-effect of the conflict between ip2nets and networks options. The key error report is the first line of the above output. Remove one or other of the networks or ip2nets options to resolve the conflict.

If there is more than one networks or ip2nets declaration defined through successive entries in one or more kernel module configuration files (i.e., multiple networks declarations or multiple ip2nets declarations), the last entry that is read by the module loader will be used. Make sure that there is no more than one LNet configuration defined in the modprobe directory (/etc/modprobe.d in RHEL and CentOS) when using this method to define LNet parameters.

Kernel module options are read only when the kernel module is loaded, passed to the kernel by the module loader. As a consequence, changes in the configuration will not be applied until the lnet module is stopped, unloaded from the kernel, and then reloaded.

LNet networks syntax

The simplest configuration has the following syntax:

options lnet networks="<lnd><#>(<dev>)[,…]"

For example, to add a TCP/IP (socklnd) LNet NID for an Ethernet device eth1:

options lnet networks="tcp0(eth1)"

Notice that the IP address is not mentioned in the configuration, only the device name. The syntax is simple and readable.

The next example creates an RDMA verbs (o2iblnd) LNet interface:

options lnet networks="o2ib0(ib0)"

If there is more than one network interface on the host that will carry LNet traffic, additional interfaces can be added to the networks variable:

options lnet networks="tcp0(eth1),o2ib0(ib0),o2ib1(ib1)"

The above example configures three LNet interfaces: a socklnd NID on eth1, and o2iblnd NIDs for ib0 and ib1.

To create a configuration where one physical interface belongs to two LNets, specify each LNet in a comma-separated format, with the same physical interface in brackets for each LNet. For example:

options lnet networks="tcp0(eth1),tcp1(eth1)"

This is not a common configuration, because its practical application is limited (both LNets will be on the same subnet and connect to the same physical interface, so there is no advantage in creating two LNets in this case).

LNet ip2nets syntax

The ip2nets configuration syntax makes use of regular expressions to programmatically create a host’s LNet configuration from a single global definition. The administrator creates pattern-matching rules based on the IPv4 address of the network interfaces of the target hosts, and the first matching rule is used to determine the configuration that will be applied. In this way, a single configuration file can be applied to both clients and servers across a heterogeneous network environment, and across multiple networks.

This flexibility comes at the cost of simplicity and readability. The ip2nets rules can be complex, making it difficult to interpret the effects of a rule on a given set of hosts. A syntax error introduced into the rules, such as an incorrectly scoped pattern matching expression, can have a wide-ranging, negative impact on the configuration of the entire host population, if it is distributed before the error is caught. Testing the ip2nets configuration is more complex as a result.

The general syntax is as follows:

options lnet ip2nets="<lnd>[<#>][(<dev>)][, <lnd>[<#>][(<dev>)]] <pattern>[; …]"

Each rule is separated by a semi-colon and rules are evaluated in left-to-right order when lnet is started. The first match for each LNet will be applied to the configuration. Evaluation continues until all unique LNets are created or the last rule has been evaluated. This allows a single rule to define multiple NIDs for a single server, each one for a different LNet on a different network interface.

It is simplest to illustrate how to use ip2nets with an example. The following describes a configuration for two different LNets on different types of network fabric:

options lnet ip2nets="tcp0(eth1) 10.10.100.*; o2ib0(ib0) 192.168.200.*"

Hosts having an IP address that matches the pattern 10.10.100.* will be configured with a NID on the tcp0 LNet, and hosts that match the 192.168.200.* pattern will be configured with an o2ib0 LNet. Should a host match both patterns, then both LNets will be configured for that host.

The pattern-matching syntax includes numeric ranges in the format [x-y], for example [2-20]. This can be further refined with a divisor to allow for stepping in the range: [2-20/2] will match all of the even numbers in the range, and [1-19/2] can be used to match all of the odd numbers in a range.

In the next example, some of the hosts will use eth0 while others use eth1 to connect to the LNet tcp0. This configuration is most likely when there are hardware differences between systems and they are using different NICs to connect to the same subnet:

options lnet ip2nets="tcp0(eth0) 10.10.100.[1-50]; tcp0(eth1) 10.10.100.[100-200]"

If the interface definition is omitted, then the host interface that matches the IP address will be used. So the above example could be re-written as:

options lnet ip2nets="tcp0 10.10.100.*"

If no match is found in the configuration, LNet will report an error:

[257641.245272] LNetError: 11a-a: ip2nets does not match any local IP interfaces
[257641.247813] LNetError: 8881:0:(config.c:199:lnet_parse_networks()) networks string is undefined

Contrary, perhaps, to expectations, if ip2nets does not find a match, the LNet kernel module will not fall back to the networks option or load a default configuration. A failure to match a pattern in the ip2nets options declaration is considered to be implicit confirmation that the target system is not meant to have an LNet configuration applied.

Here is a more complex demonstration:

options lnet ip2nets="tcp0(eth1) 10.70.207.[11,12]; \
    tcp0(eth2) 10.70.207.[21-24]; \
    tcp0 10.70.*.*"

Above, there are two hosts that will create an LNet NID on the eth1 device, four hosts using eth2, and the remainder of the 10.70/16 network will create an LNet NID for the tcp0 LNet on the first interface presented by the operating system, provided that the interface has an IPv4 address on the 10.70/16 subnet.

This last item requires further explanation. The syntax of configuration entry “tcp0 10.70.*.*” is ambiguous and open to misinterpretation by a human reader. Contrary to expectation, this does not mean “configure the interface that matches the pattern 10.70.*.* as LNet tcp0. It is in fact far more specific. It means “configure the first interface detected by the host operating system as LNet tcp0 if it also matches the pattern 10.70.*.*". If the condition is false, then no LNet NID will be configured.

One can also mix LNDs in the ip2nets syntax, should this be required:

options lnet ip2nets="o2ib0(ib0) 192.168.207.[11,12]; \
    o2ib0(ib2) 192.168.207.[21-24]; \
    tcp0 10.70.*.*"

Comments are also supported, but take care with placement of the comment relative to the semi-colon that separates rules. A comment marker instructs the parser to ignore everything between the comment marker and the next semi-colon in the string, or the end of the string, whichever comes first. If a comment is started after a semi-colon, that could have the effect of causing the parser to ignore an important part of the configuration.

For example, the following syntax is correct:

options lnet ip2nets="tcp0(eth1) 10.70.207.[11-12] # MGS/MDS; \
    tcp0(eth2) 10.70.207.[21-24] # OSS; \
    tcp0 10.70.*.* # Everything else"

In the next example, the second line will be ignored because the comment marker (#) is placed after the semi-colon, causing the next line to be treated as comment text, not configuration syntax:

options lnet ip2nets="tcp0(eth1) 10.70.207.[11-12] ; # MGS/MDS \
tcp0(eth2) 10.70.207.[21-24] # OSS; \
tcp0 10.70.*.* # Everything else"

All of the text highlighted in the example will be treated as a comment.