Static LNet Configuration
In releases of Lustre prior to version 2.7.0, configuration of the LNet device driver is managed exclusively by supplying options to the lnet
kernel module. The options are read from a modules options (modprobe
) configuration file when the kernel module is first loaded. With the release of Lustre 2.7.0, administrators can instead make use of the Dynamic LNet Configuration feature, controlled by a utility called lnetctl
, as an alternative to static configuration.
Refer to the article Lustre Networking (LNET) Overview for a general introduction to LNet.
This article describes static configuration of LNet.
Overview
An LNet interface can be configured by supplying parameters directly to the lnet
module when it is loaded by the kernel. To do this, create a file in the normal modprobe
configuration options directory (usually /etc/modprobe.d
), and add an entry for the lnet
module. By convention, Lustre module configuration options are recorded in a file called /etc/modprobe.d/lustre.conf
(although tuning parameters for devices can be separated into device-specific files, such as /etc/modprobe.d/ko2iblnd.conf
– see the section Optimizing o2iblnd Performance for an example of this, which shows an optimized configuration that is automatically applied when an Intel® Omni-Path interface is detected).
There are two ways to define an LNet configuration using module options: networks
and ip2nets
. The networks
parameter is the easiest to understand and uses a very simple syntax, while ip2nets
gives administrators the greatest flexibility at the expense of much greater complexity. The networks
syntax is strongly recommended unless there is a specific requirement that can be more readily fulfilled by the ip2nets
syntax.
There should be only one networks
or ip2nets
parameter defined for the lnet
module. The parameters are mutually exclusive; choose one or the other but not both. If both options are referenced in the modules configuration, then the following warning will be reported in the kernel ring buffer (dmesg
) and system console:
[46290.641911] LNetError: 101-0: Please specify EITHER 'networks' or 'ip2nets' but not both at once [46290.645288] LNetError: 3483:0:(config.c:199:lnet_parse_networks()) networks string is undefined
The second line in the error output is a side-effect of the conflict between ip2nets
and networks
options. The key error report is the first line of the above output. Remove one or other of the networks
or ip2nets
options to resolve the conflict.
If there is more than one networks
or ip2nets
declaration defined through successive entries in one or more kernel module configuration files (i.e., multiple networks
declarations or multiple ip2nets
declarations), the last entry that is read by the module loader will be used. Make sure that there is no more than one LNet configuration defined in the modprobe
directory (/etc/modprobe.d
in RHEL and CentOS) when using this method to define LNet parameters.
Kernel module options are read only when the kernel module is loaded, passed to the kernel by the module loader. As a consequence, changes in the configuration will not be applied until the lnet
module is stopped, unloaded from the kernel, and then reloaded.
LNet networks
syntax
The simplest configuration has the following syntax:
options lnet networks="<lnd><#>(<dev>)[,…]"
For example, to add a TCP/IP (socklnd
) LNet NID for an Ethernet device eth1
:
options lnet networks="tcp0(eth1)"
Notice that the IP address is not mentioned in the configuration, only the device name. The syntax is simple and readable.
The next example creates an RDMA verbs (o2iblnd
) LNet interface:
options lnet networks="o2ib0(ib0)"
If there is more than one network interface on the host that will carry LNet traffic, additional interfaces can be added to the networks
variable:
options lnet networks="tcp0(eth1),o2ib0(ib0),o2ib1(ib1)"
The above example configures three LNet interfaces: a socklnd
NID on eth1
, and o2iblnd
NIDs for ib0
and ib1
.
To create a configuration where one physical interface belongs to two LNets, specify each LNet in a comma-separated format, with the same physical interface in brackets for each LNet. For example:
options lnet networks="tcp0(eth1),tcp1(eth1)"
This is not a common configuration, because its practical application is limited (both LNets will be on the same subnet and connect to the same physical interface, so there is no advantage in creating two LNets in this case).
LNet ip2nets
syntax
The ip2nets
configuration syntax makes use of regular expressions to programmatically create a host’s LNet configuration from a single global definition. The administrator creates pattern-matching rules based on the IPv4 address of the network interfaces of the target hosts, and the first matching rule is used to determine the configuration that will be applied. In this way, a single configuration file can be applied to both clients and servers across a heterogeneous network environment, and across multiple networks.
This flexibility comes at the cost of simplicity and readability. The ip2nets
rules can be complex, making it difficult to interpret the effects of a rule on a given set of hosts. A syntax error introduced into the rules, such as an incorrectly scoped pattern matching expression, can have a wide-ranging, negative impact on the configuration of the entire host population, if it is distributed before the error is caught. Testing the ip2nets
configuration is more complex as a result.
The general syntax is as follows:
options lnet ip2nets="<lnd>[<#>][(<dev>)][, <lnd>[<#>][(<dev>)]] <pattern>[; …]"
Each rule is separated by a semi-colon and rules are evaluated in left-to-right order when lnet
is started. The first match for each LNet will be applied to the configuration. Evaluation continues until all unique LNets are created or the last rule has been evaluated. This allows a single rule to define multiple NIDs for a single server, each one for a different LNet on a different network interface.
It is simplest to illustrate how to use ip2nets
with an example. The following describes a configuration for two different LNets on different types of network fabric:
options lnet ip2nets="tcp0(eth1) 10.10.100.*; o2ib0(ib0) 192.168.200.*"
Hosts having an IP address that matches the pattern 10.10.100.*
will be configured with a NID on the tcp0
LNet, and hosts that match the 192.168.200.*
pattern will be configured with an o2ib0
LNet. Should a host match both patterns, then both LNets will be configured for that host.
The pattern-matching syntax includes numeric ranges in the format [x-y]
, for example [2-20]
. This can be further refined with a divisor to allow for stepping in the range: [2-20/2]
will match all of the even numbers in the range, and [1-19/2]
can be used to match all of the odd numbers in a range.
In the next example, some of the hosts will use eth0
while others use eth1
to connect to the LNet tcp0
. This configuration is most likely when there are hardware differences between systems and they are using different NICs to connect to the same subnet:
options lnet ip2nets="tcp0(eth0) 10.10.100.[1-50]; tcp0(eth1) 10.10.100.[100-200]"
If the interface definition is omitted, then the host interface that matches the IP address will be used. So the above example could be re-written as:
options lnet ip2nets="tcp0 10.10.100.*"
If no match is found in the configuration, LNet will report an error:
[257641.245272] LNetError: 11a-a: ip2nets does not match any local IP interfaces [257641.247813] LNetError: 8881:0:(config.c:199:lnet_parse_networks()) networks string is undefined
Contrary, perhaps, to expectations, if ip2nets
does not find a match, the LNet kernel module will not fall back to the networks
option or load a default configuration. A failure to match a pattern in the ip2nets
options declaration is considered to be implicit confirmation that the target system is not meant to have an LNet configuration applied.
Here is a more complex demonstration:
options lnet ip2nets="tcp0(eth1) 10.70.207.[11,12]; \ tcp0(eth2) 10.70.207.[21-24]; \ tcp0 10.70.*.*"
Above, there are two hosts that will create an LNet NID on the eth1
device, four hosts using eth2
, and the remainder of the 10.70/16
network will create an LNet NID for the tcp0
LNet on the first interface presented by the operating system, provided that the interface has an IPv4 address on the 10.70/16
subnet.
This last item requires further explanation. The syntax of configuration entry “tcp0 10.70.*.*”
is ambiguous and open to misinterpretation by a human reader. Contrary to expectation, this does not mean “configure the interface that matches the pattern 10.70.*.*
as LNet tcp0
”. It is in fact far more specific. It means “configure the first interface detected by the host operating system as LNet tcp0
if it also matches the pattern 10.70.*.*
". If the condition is false, then no LNet NID will be configured.
One can also mix LNDs in the ip2nets
syntax, should this be required:
options lnet ip2nets="o2ib0(ib0) 192.168.207.[11,12]; \ o2ib0(ib2) 192.168.207.[21-24]; \ tcp0 10.70.*.*"
Comments are also supported, but take care with placement of the comment relative to the semi-colon that separates rules. A comment marker instructs the parser to ignore everything between the comment marker and the next semi-colon in the string, or the end of the string, whichever comes first. If a comment is started after a semi-colon, that could have the effect of causing the parser to ignore an important part of the configuration.
For example, the following syntax is correct:
options lnet ip2nets="tcp0(eth1) 10.70.207.[11-12] # MGS/MDS; \ tcp0(eth2) 10.70.207.[21-24] # OSS; \ tcp0 10.70.*.* # Everything else"
In the next example, the second line will be ignored because the comment marker (#) is placed after the semi-colon, causing the next line to be treated as comment text, not configuration syntax:
options lnet ip2nets="tcp0(eth1) 10.70.207.[11-12] ; # MGS/MDS \
tcp0(eth2) 10.70.207.[21-24] # OSS; \
tcp0 10.70.*.* # Everything else"
All of the text highlighted in the example will be treated as a comment.