Malcolm: /* LNet ip2nets syntax */

2018-03-21T20:43:18Z

LNet ip2nets syntax

← Older revision		Revision as of 13:43, 21 March 2018
Line 117:		Line 117:
	Above, there are two hosts that will create an LNet NID on the <code>eth1</code> device, four hosts using <code>eth2</code>, and the remainder of the <code>10.70/16</code> network will create an LNet NID for the <code>tcp0</code> LNet on the first interface presented by the operating system, provided that the interface has an IPv4 address on the <code>10.70/16</code> subnet.		Above, there are two hosts that will create an LNet NID on the <code>eth1</code> device, four hosts using <code>eth2</code>, and the remainder of the <code>10.70/16</code> network will create an LNet NID for the <code>tcp0</code> LNet on the first interface presented by the operating system, provided that the interface has an IPv4 address on the <code>10.70/16</code> subnet.

	This last item requires further explanation. The syntax of configuration entry <code>“tcp0 10.70..”</code> is ambiguous and open to misinterpretation by a human reader. Contrary to expectation, this does not mean ''“configure the interface that matches the pattern <code>10.70..</code> as LNet <code>tcp0</code>”''. It is in fact far more specific. It means ''“configure the first interface detected by the host operating system as LNet <code>tcp0</code> if it also matches the pattern <code>10.70..</code>"''. If ~~either~~ condition is false, then no LNet NID will be configured.		This last item requires further explanation. The syntax of configuration entry <code>“tcp0 10.70..”</code> is ambiguous and open to misinterpretation by a human reader. Contrary to expectation, this does not mean ''“configure the interface that matches the pattern <code>10.70..</code> as LNet <code>tcp0</code>”''. It is in fact far more specific. It means ''“configure the first interface detected by the host operating system as LNet <code>tcp0</code> if it also matches the pattern <code>10.70..</code>"''. If the condition is false, then no LNet NID will be configured.

	One can also mix LNDs in the <code>ip2nets</code> syntax, should this be required:		One can also mix LNDs in the <code>ip2nets</code> syntax, should this be required:

Malcolm at 04:43, 8 September 2017

2017-09-08T04:43:06Z

← Older revision		Revision as of 21:43, 7 September 2017
Line 1:		Line 1:
	In releases of Lustre prior to version 2.7.0, configuration of the LNet device driver is managed exclusively by supplying options to the <code>lnet</code> kernel module. The options are read from a modules options (<code>modprobe</code>) configuration file when the kernel module is first loaded. With the release of Lustre 2.7.0, administrators can instead make use of the [[Dynamic LNet ~~configuration (DLC)~~]] feature, controlled by a utility called <code>lnetctl</code>, as an alternative to static configuration.		In releases of Lustre prior to version 2.7.0, configuration of the LNet device driver is managed exclusively by supplying options to the <code>lnet</code> kernel module. The options are read from a modules options (<code>modprobe</code>) configuration file when the kernel module is first loaded. With the release of Lustre 2.7.0, administrators can instead make use of the [[Dynamic LNet Configuration and lnetctl\| Dynamic LNet Configuration]] feature, controlled by a utility called <code>lnetctl</code>, as an alternative to static configuration.

	Refer to the article [[Lustre Networking (LNET) Overview]] for a general introduction to LNet.		Refer to the article [[Lustre Networking (LNET) Overview]] for a general introduction to LNet.

Malcolm at 05:46, 31 August 2017

2017-08-31T05:46:51Z

← Older revision		Revision as of 22:46, 30 August 2017
Line 147:		Line 147:

	[[Category:Lustre Networking (LNet)]]		[[Category:Lustre Networking (LNet)]]
			[[Category:Lustre Systems Administration]]

Malcolm: Created page with "In releases of Lustre prior to version 2.7.0, configuration of the LNet device driver is managed exclusively by supplying options to the `lnet` kernel module. The o..."

2017-08-10T02:57:11Z

Created page with "In releases of Lustre prior to version 2.7.0, configuration of the LNet device driver is managed exclusively by supplying options to the <code>lnet</code> kernel module. The o..."

New page

In releases of Lustre prior to version 2.7.0, configuration of the LNet device driver is managed exclusively by supplying options to the <code>lnet</code> kernel module. The options are read from a modules options (<code>modprobe</code>) configuration file when the kernel module is first loaded. With the release of Lustre 2.7.0, administrators can instead make use of the [[Dynamic LNet configuration (DLC)]] feature, controlled by a utility called <code>lnetctl</code>, as an alternative to static configuration.

Refer to the article [[Lustre Networking (LNET) Overview]] for a general introduction to LNet.

This article describes static configuration of LNet.

== Overview ==

An LNet interface can be configured by supplying parameters directly to the <code>lnet</code> module when it is loaded by the kernel. To do this, create a file in the normal <code>modprobe</code> configuration options directory (usually <code>/etc/modprobe.d</code>), and add an entry for the <code>lnet</code> module. By convention, Lustre module configuration options are recorded in a file called <code>/etc/modprobe.d/lustre.conf</code> (although tuning parameters for devices can be separated into device-specific files, such as <code>/etc/modprobe.d/ko2iblnd.conf</code> – see the section [[Optimizing o2iblnd Performance]] for an example of this, which shows an optimized configuration that is automatically applied when an Intel® Omni-Path interface is detected).

There are two ways to define an LNet configuration using module options: <code>networks</code> and <code>ip2nets</code>. The <code>networks</code> parameter is the easiest to understand and uses a very simple syntax, while <code>ip2nets</code> gives administrators the greatest flexibility at the expense of much greater complexity. The <code>networks</code> syntax is strongly recommended unless there is a specific requirement that can be more readily fulfilled by the <code>ip2nets</code> syntax.

There should be only one <code>networks</code> or <code>ip2nets</code> parameter defined for the <code>lnet</code> module. The parameters are mutually exclusive; choose one or the other but not both. If both options are referenced in the modules configuration, then the following warning will be reported in the kernel ring buffer (<code>dmesg</code>) and system console:

<pre style="overflow-x:auto;">
[46290.641911] LNetError: 101-0: Please specify EITHER 'networks' or 'ip2nets' but not both at once
[46290.645288] LNetError: 3483:0:(config.c:199:lnet_parse_networks()) networks string is undefined
</pre>

The second line in the error output is a side-effect of the conflict between <code>ip2nets</code> and <code>networks</code> options. The key error report is the first line of the above output. Remove one or other of the <code>networks</code> or <code>ip2nets</code> options to resolve the conflict.

If there is more than one <code>networks</code> or <code>ip2nets</code> declaration defined through successive entries in one or more kernel module configuration files (i.e., multiple <code>networks</code> declarations or multiple <code>ip2nets</code> declarations), the last entry that is read by the module loader will be used. Make sure that there is no more than one LNet configuration defined in the <code>modprobe</code> directory (<code>/etc/modprobe.d</code> in RHEL and CentOS) when using this method to define LNet parameters.

Kernel module options are read only when the kernel module is loaded, passed to the kernel by the module loader. As a consequence, changes in the configuration will not be applied until the <code>lnet</code> module is stopped, unloaded from the kernel, and then reloaded.

== LNet <code>networks</code> syntax ==

The simplest configuration has the following syntax:

<pre style="overflow-x:auto;">
options lnet networks="<lnd><#>(<dev>)[,…]"
</pre>

For example, to add a TCP/IP (<code>socklnd</code>) LNet NID for an Ethernet device <code>eth1</code>:

<pre style="overflow-x:auto;">
options lnet networks="tcp0(eth1)"
</pre>

Notice that the IP address is not mentioned in the configuration, only the device name. The syntax is simple and readable.

The next example creates an RDMA verbs (<code>o2iblnd</code>) LNet interface:

<pre style="overflow-x:auto;">
options lnet networks="o2ib0(ib0)"
</pre>

If there is more than one network interface on the host that will carry LNet traffic, additional interfaces can be added to the <code>networks</code> variable:

<pre style="overflow-x:auto;">
options lnet networks="tcp0(eth1),o2ib0(ib0),o2ib1(ib1)"
</pre>

The above example configures three LNet interfaces: a <code>socklnd</code> NID on <code>eth1</code>, and <code>o2iblnd</code> NIDs for <code>ib0</code> and <code>ib1</code>.

To create a configuration where one physical interface belongs to two LNets, specify each LNet in a comma-separated format, with the same physical interface in brackets for each LNet. For example:

<pre style="overflow-x:auto;">
options lnet networks="tcp0(eth1),tcp1(eth1)"
</pre>

This is not a common configuration, because its practical application is limited (both LNets will be on the same subnet and connect to the same physical interface, so there is no advantage in creating two LNets in this case).

== LNet <code>ip2nets</code> syntax ==

The <code>ip2nets</code> configuration syntax makes use of regular expressions to programmatically create a host’s LNet configuration from a single global definition. The administrator creates pattern-matching rules based on the IPv4 address of the network interfaces of the target hosts, and the first matching rule is used to determine the configuration that will be applied. In this way, a single configuration file can be applied to both clients and servers across a heterogeneous network environment, and across multiple networks.

This flexibility comes at the cost of simplicity and readability. The <code>ip2nets</code> rules can be complex, making it difficult to interpret the effects of a rule on a given set of hosts. A syntax error introduced into the rules, such as an incorrectly scoped pattern matching expression, can have a wide-ranging, negative impact on the configuration of the entire host population, if it is distributed before the error is caught. Testing the <code>ip2nets</code> configuration is more complex as a result.

The general syntax is as follows:

<pre style="overflow-x:auto;">
options lnet ip2nets="<lnd>[<#>][(<dev>)][, <lnd>[<#>][(<dev>)]] <pattern>[; …]"
</pre>

Each rule is separated by a semi-colon and rules are evaluated in left-to-right order when <code>lnet</code> is started. The first match for each LNet will be applied to the configuration. Evaluation continues until all unique LNets are created or the last rule has been evaluated. This allows a single rule to define multiple NIDs for a single server, each one for a different LNet on a different network interface.

It is simplest to illustrate how to use <code>ip2nets</code> with an example. The following describes a configuration for two different LNets on different types of network fabric:

<pre style="overflow-x:auto;">
options lnet ip2nets="tcp0(eth1) 10.10.100.*; o2ib0(ib0) 192.168.200.*"
</pre>

Hosts having an IP address that matches the pattern <code>10.10.100.*</code> will be configured with a NID on the <code>tcp0</code> LNet, and hosts that match the <code>192.168.200.*</code> pattern will be configured with an <code>o2ib0</code> LNet. Should a host match both patterns, then both LNets will be configured for that host.

The pattern-matching syntax includes numeric ranges in the format <code>[x-y]</code>, for example <code>[2-20]</code>. This can be further refined with a divisor to allow for stepping in the range: <code>[2-20/2]</code> will match all of the even numbers in the range, and <code>[1-19/2]</code> can be used to match all of the odd numbers in a range.

In the next example, some of the hosts will use <code>eth0</code> while others use <code>eth1</code> to connect to the LNet <code>tcp0</code>. This configuration is most likely when there are hardware differences between systems and they are using different NICs to connect to the same subnet:

<pre style="overflow-x:auto;">
options lnet ip2nets="tcp0(eth0) 10.10.100.[1-50]; tcp0(eth1) 10.10.100.[100-200]"
</pre>

If the interface definition is omitted, then the host interface that matches the IP address will be used. So the above example could be re-written as:

<pre style="overflow-x:auto;">
options lnet ip2nets="tcp0 10.10.100.*"
</pre>

If no match is found in the configuration, LNet will report an error:

<pre style="overflow-x:auto;">
[257641.245272] LNetError: 11a-a: ip2nets does not match any local IP interfaces
[257641.247813] LNetError: 8881:0:(config.c:199:lnet_parse_networks()) networks string is undefined
</pre>

Contrary, perhaps, to expectations, if <code>ip2nets</code> does not find a match, the LNet kernel module will not fall back to the <code>networks</code> option or load a default configuration. A failure to match a pattern in the <code>ip2nets</code> options declaration is considered to be implicit confirmation that the target system is not meant to have an LNet configuration applied.

Here is a more complex demonstration:

<pre style="overflow-x:auto;">
options lnet ip2nets="tcp0(eth1) 10.70.207.[11,12]; \
tcp0(eth2) 10.70.207.[21-24]; \
tcp0 10.70.*.*"
</pre>

Above, there are two hosts that will create an LNet NID on the <code>eth1</code> device, four hosts using <code>eth2</code>, and the remainder of the <code>10.70/16</code> network will create an LNet NID for the <code>tcp0</code> LNet on the first interface presented by the operating system, provided that the interface has an IPv4 address on the <code>10.70/16</code> subnet.

This last item requires further explanation. The syntax of configuration entry <code>“tcp0 10.70.*.*”</code> is ambiguous and open to misinterpretation by a human reader. Contrary to expectation, this does not mean ''“configure the interface that matches the pattern <code>10.70.*.*</code> as LNet <code>tcp0</code>”''. It is in fact far more specific. It means ''“configure the first interface detected by the host operating system as LNet <code>tcp0</code> if it also matches the pattern <code>10.70.*.*</code>"''. If either condition is false, then no LNet NID will be configured.

One can also mix LNDs in the <code>ip2nets</code> syntax, should this be required:

<pre style="overflow-x:auto;">
options lnet ip2nets="o2ib0(ib0) 192.168.207.[11,12]; \
o2ib0(ib2) 192.168.207.[21-24]; \
tcp0 10.70.*.*"
</pre>

Comments are also supported, but take care with placement of the comment relative to the semi-colon that separates rules. A comment marker instructs the parser to ignore everything between the comment marker and the next semi-colon in the string, or the end of the string, whichever comes first. If a comment is started after a semi-colon, that could have the effect of causing the parser to ignore an important part of the configuration.

For example, the following syntax is correct:

<pre style="overflow-x:auto;">
options lnet ip2nets="tcp0(eth1) 10.70.207.[11-12] # MGS/MDS; \
tcp0(eth2) 10.70.207.[21-24] # OSS; \
tcp0 10.70.*.* # Everything else"
</pre>

In the next example, the second line will be ignored because the comment marker (#) is placed after the semi-colon, causing the next line to be treated as comment text, not configuration syntax:

<div style="overflow-x:auto;font-family:monospace, Courier;color:black;background-color:#f9f9f9;border:1px solid #ddd;border-radius:2px;padding:1px 4px">
options lnet ip2nets="tcp0(eth1) 10.70.207.[11-12] ; <span style="background:yellow"># MGS/MDS \</span><br/>
<span style="background:yellow"> tcp0(eth2) 10.70.207.[21-24] # OSS; \</span><br/>
<span style="background:yellow"> tcp0 10.70.*.* # Everything else</span>"<br/>
</div>
All of the text highlighted in the example will be treated as a comment.

[[Category:Lustre Networking (LNet)]]

Static LNet Configuration - Revision history

Malcolm: /* LNet ip2nets syntax */

Malcolm at 04:43, 8 September 2017

Malcolm at 05:46, 31 August 2017

Malcolm: Created page with "In releases of Lustre prior to version 2.7.0, configuration of the LNet device driver is managed exclusively by supplying options to the lnet kernel module. The o..."

Malcolm: Created page with "In releases of Lustre prior to version 2.7.0, configuration of the LNet device driver is managed exclusively by supplying options to the `lnet` kernel module. The o..."