Static LNet Configuration

In releases of Lustre prior to version 2.7.0, configuration of the LNet device driver is managed exclusively by supplying options to the  kernel module. The options are read from a modules options configuration file when the kernel module is first loaded. With the release of Lustre 2.7.0, administrators can instead make use of the Dynamic LNet Configuration feature, controlled by a utility called , as an alternative to static configuration.

Refer to the article Lustre Networking (LNET) Overview for a general introduction to LNet.

This article describes static configuration of LNet.

Overview
An LNet interface can be configured by supplying parameters directly to the  module when it is loaded by the kernel. To do this, create a file in the normal  configuration options directory (usually  ), and add an entry for the   module. By convention, Lustre module configuration options are recorded in a file called  (although tuning parameters for devices can be separated into device-specific files, such as   – see the section Optimizing o2iblnd Performance for an example of this, which shows an optimized configuration that is automatically applied when an Intel® Omni-Path interface is detected).

There are two ways to define an LNet configuration using module options:  and. The  parameter is the easiest to understand and uses a very simple syntax, while   gives administrators the greatest flexibility at the expense of much greater complexity. The  syntax is strongly recommended unless there is a specific requirement that can be more readily fulfilled by the   syntax.

There should be only one  or   parameter defined for the   module. The parameters are mutually exclusive; choose one or the other but not both. If both options are referenced in the modules configuration, then the following warning will be reported in the kernel ring buffer and system console:

 [46290.641911] LNetError: 101-0: Please specify EITHER 'networks' or 'ip2nets' but not both at once [46290.645288] LNetError: 3483:0:(config.c:199:lnet_parse_networks) networks string is undefined

The second line in the error output is a side-effect of the conflict between  and   options. The key error report is the first line of the above output. Remove one or other of the  or   options to resolve the conflict.

If there is more than one  or   declaration defined through successive entries in one or more kernel module configuration files (i.e., multiple   declarations or multiple   declarations), the last entry that is read by the module loader will be used. Make sure that there is no more than one LNet configuration defined in the  directory (  in RHEL and CentOS) when using this method to define LNet parameters.

Kernel module options are read only when the kernel module is loaded, passed to the kernel by the module loader. As a consequence, changes in the configuration will not be applied until the  module is stopped, unloaded from the kernel, and then reloaded.

LNet syntax
The simplest configuration has the following syntax:

 options lnet networks=" <#>[,…]"

For example, to add a TCP/IP LNet NID for an Ethernet device  :

 options lnet networks="tcp0(eth1)"

Notice that the IP address is not mentioned in the configuration, only the device name. The syntax is simple and readable.

The next example creates an RDMA verbs LNet interface:

 options lnet networks="o2ib0(ib0)"

If there is more than one network interface on the host that will carry LNet traffic, additional interfaces can be added to the  variable:

 options lnet networks="tcp0(eth1),o2ib0(ib0),o2ib1(ib1)"

The above example configures three LNet interfaces: a  NID on , and   NIDs for   and.

To create a configuration where one physical interface belongs to two LNets, specify each LNet in a comma-separated format, with the same physical interface in brackets for each LNet. For example:

 options lnet networks="tcp0(eth1),tcp1(eth1)"

This is not a common configuration, because its practical application is limited (both LNets will be on the same subnet and connect to the same physical interface, so there is no advantage in creating two LNets in this case).

LNet syntax
The  configuration syntax makes use of regular expressions to programmatically create a host’s LNet configuration from a single global definition. The administrator creates pattern-matching rules based on the IPv4 address of the network interfaces of the target hosts, and the first matching rule is used to determine the configuration that will be applied. In this way, a single configuration file can be applied to both clients and servers across a heterogeneous network environment, and across multiple networks.

This flexibility comes at the cost of simplicity and readability. The  rules can be complex, making it difficult to interpret the effects of a rule on a given set of hosts. A syntax error introduced into the rules, such as an incorrectly scoped pattern matching expression, can have a wide-ranging, negative impact on the configuration of the entire host population, if it is distributed before the error is caught. Testing the  configuration is more complex as a result.

The general syntax is as follows:

 options lnet ip2nets=" [<#>][][, [<#>][]] [; …]"

Each rule is separated by a semi-colon and rules are evaluated in left-to-right order when  is started. The first match for each LNet will be applied to the configuration. Evaluation continues until all unique LNets are created or the last rule has been evaluated. This allows a single rule to define multiple NIDs for a single server, each one for a different LNet on a different network interface.

It is simplest to illustrate how to use  with an example. The following describes a configuration for two different LNets on different types of network fabric:

 options lnet ip2nets="tcp0(eth1) 10.10.100.*; o2ib0(ib0) 192.168.200.*"

Hosts having an IP address that matches the pattern  will be configured with a NID on the   LNet, and hosts that match the   pattern will be configured with an   LNet. Should a host match both patterns, then both LNets will be configured for that host.

The pattern-matching syntax includes numeric ranges in the format, for example. This can be further refined with a divisor to allow for stepping in the range:  will match all of the even numbers in the range, and   can be used to match all of the odd numbers in a range.

In the next example, some of the hosts will use  while others use   to connect to the LNet. This configuration is most likely when there are hardware differences between systems and they are using different NICs to connect to the same subnet:

 options lnet ip2nets="tcp0(eth0) 10.10.100.[1-50]; tcp0(eth1) 10.10.100.[100-200]"

If the interface definition is omitted, then the host interface that matches the IP address will be used. So the above example could be re-written as:

 options lnet ip2nets="tcp0 10.10.100.*"

If no match is found in the configuration, LNet will report an error:

 [257641.245272] LNetError: 11a-a: ip2nets does not match any local IP interfaces [257641.247813] LNetError: 8881:0:(config.c:199:lnet_parse_networks) networks string is undefined

Contrary, perhaps, to expectations, if  does not find a match, the LNet kernel module will not fall back to the   option or load a default configuration. A failure to match a pattern in the  options declaration is considered to be implicit confirmation that the target system is not meant to have an LNet configuration applied.

Here is a more complex demonstration:

 options lnet ip2nets="tcp0(eth1) 10.70.207.[11,12]; \   tcp0(eth2) 10.70.207.[21-24]; \    tcp0 10.70.*.*"

Above, there are two hosts that will create an LNet NID on the  device, four hosts using , and the remainder of the   network will create an LNet NID for the   LNet on the first interface presented by the operating system, provided that the interface has an IPv4 address on the   subnet.

This last item requires further explanation. The syntax of configuration entry  is ambiguous and open to misinterpretation by a human reader. Contrary to expectation, this does not mean “configure the interface that matches the pattern  as LNet  ”. It is in fact far more specific. It means “configure the first interface detected by the host operating system as LNet  if it also matches the pattern  ". If either condition is false, then no LNet NID will be configured.

One can also mix LNDs in the  syntax, should this be required:

 options lnet ip2nets="o2ib0(ib0) 192.168.207.[11,12]; \   o2ib0(ib2) 192.168.207.[21-24]; \    tcp0 10.70.*.*"

Comments are also supported, but take care with placement of the comment relative to the semi-colon that separates rules. A comment marker instructs the parser to ignore everything between the comment marker and the next semi-colon in the string, or the end of the string, whichever comes first. If a comment is started after a semi-colon, that could have the effect of causing the parser to ignore an important part of the configuration.

For example, the following syntax is correct:

 options lnet ip2nets="tcp0(eth1) 10.70.207.[11-12] # MGS/MDS; \   tcp0(eth2) 10.70.207.[21-24] # OSS; \    tcp0 10.70.*.* # Everything else"

In the next example, the second line will be ignored because the comment marker (#) is placed after the semi-colon, causing the next line to be treated as comment text, not configuration syntax:

 options lnet ip2nets="tcp0(eth1) 10.70.207.[11-12] ; # MGS/MDS \     tcp0(eth2) 10.70.207.[21-24] # OSS; \      tcp0 10.70.*.* # Everything else " All of the text highlighted in the example will be treated as a comment.