Malcolm at 05:47, 31 August 2017

2017-08-31T05:47:43Z

← Older revision		Revision as of 22:47, 30 August 2017
Line 40:		Line 40:

	[[Category:Lustre Networking (LNet)]]		[[Category:Lustre Networking (LNet)]]
			[[Category:Lustre Systems Administration]]

Malcolm: Created page with "If no explicit configuration is supplied to LNet, either through a modprobe options file or a YAML description for `lnetctl`, it will attempt to create a valid TCP/..."

2017-08-10T03:22:11Z

Created page with "If no explicit configuration is supplied to LNet, either through a modprobe options file or a YAML description for <code>lnetctl</code>, it will attempt to create a valid TCP/..."

New page

If no explicit configuration is supplied to LNet, either through a modprobe options file or a YAML description for <code>lnetctl</code>, it will attempt to create a valid TCP/IP (<code>socklnd</code>) NID for <code>tcp0</code> using the first network interface that is detected by the operating system (e.g. <code>eth0</code>) when the module is loaded and the LNet service started.

The order of interface detection is entirely at the discretion of the operating system, which means that there is no guarantee that the ordering of interfaces will be preserved between reboots and on the insertion of a new hardware device. It also means that the default behavior for a host will differ depending on its hardware configuration. Most operating systems do try to ensure that a device, after it is detected, maintains the same device name (<code>eth0</code>, <code>eth1</code>, etc.) between reboots. Nevertheless, it is strongly recommended that all configuration be stated explicitly: defining the configuration also defines the expected behavior of the system, making it easier to audit.

The <code>ip2nets</code> option for the LNet kernel module is a list of network definition and IP-match pairs. These pairs are processed in sequence. If there is a match for a local IP address, then that network definition is used for the node, and further pairs for that network are ignored. Multiple networks can be matched.

For example:

<pre style="overflow-x:auto;">
ip2nets="tcp(eth2) 134.32.1.[4-10/2]; tcp(eth1) *.*.*.*"
</pre>

This set of rules is used to create network <code>tcp0</code> (the <code>0</code> is implied, because the LNet network number is omitted). If a local IP address matches <code>134.32.1.[4-10/2]</code>, meaning it is one of <code>134.32.1.4</code>, <code>134.32.1.6</code>, <code>134.32.1.8</code>, or <code>134.32.1.10</code>, then <code>tcp0</code> is created using interface <code>eth2</code>. Otherwise the second pair is used, and because "<code>*.*.*.*</code>" matches every address, it always creates <code>tcp0</code> on <code>eth1</code>.

Note that <code>ip2nets</code> will use the IP address definition to match the host, not the interface. The <code>ip2nets</code> definition will not verify or otherwise qualify that the IP address matched is associated with the physical network interface in the specification. This means that a pattern can match the IP address of an interface that will not actually be used for LNet communications. From the above example, if a host has an interface <code>eth3</code> with IP address <code>134.32.1.4</code>, then that would be considered a match good enough to trigger the creation of the NID on <code>tcp0</code>(<code>eth2</code>).

Also, if the device is not specified in an <code>ip2nets</code> definition, LNet will pick up the first available device rather than the device that matches the IP address pattern. For example, if the IP address pattern matches the IP address on <code>eth1</code>, but no device is mentioned in the <code>ip2nets</code> definition, then <code>eth0</code> will get an LNet configuration. As an illustration, consider a host with a <code>10.70/16</code> IP address on <code>eth0</code>, and a <code>192.168/24</code> address on <code>eth1</code>. The following <code>ip2nets</code> definition will create a NID on <code>eth0</code>, even though the pattern matches the <code>eth1</code> device:

<pre style="overflow-x:auto;">
options lnet ip2nets="tcp0 192.168.*.*"
</pre>

If, instead, the device is included in the spec, then the configuration will be applied to <code>eth1</code>:

<pre style="overflow-x:auto;">
options lnet ip2nets="tcp0(eth1) 192.168.*.*"
</pre>

The definition is interpreted as follows: configure the first socklnd NID that is found on the host where there is an IP Address matching <code>192.168.*.*</code>. In this respect, it's consistent with the behavior of the much simpler networks syntax in the following example:

<pre style="overflow-x:auto;">
options lnet networks="tcp0"
</pre>

This example creates a NID on the first network device detected by the operating system, because no device was specified. In common with the <code>ip2nets</code> parameter, the lack of definition of a specific network interface means that LNet will configure the first interface that was detected by the host operating system.

If an interface is explicitly specified as well as a pattern, the interface matched using the IP pattern will be compared against the explicitly defined interface. For example, if the <code>ip2nets</code> definition is “<code>tcp(eth0) 192.168.*.3</code>” and there exists in the system a device <code>eth0</code> with IP address <code>192.0.19.3</code> and a device <code>eth1</code> with IP address <code>192.168.3.3</code>, then configuration will fail, because the pattern contradicts the interface specified. A clear warning will be displayed if inconsistent configuration is encountered.

If the LNet number for a NID is 0 (zero), for example, <code>tcp0</code>, or <code>o2ib0</code>, the number will sometimes be omitted from command output, and can usually be omitted from configuration files as well (although it is not recommended – for reasons of clarity alone, it is recommended to supply as much information as is reasonable when creating configuration information).

[[Category:Lustre Networking (LNet)]]

LNet Configuration Edge Case Behaviors and Side-Effects - Revision history

Malcolm at 05:47, 31 August 2017

Malcolm: Created page with "If no explicit configuration is supplied to LNet, either through a modprobe options file or a YAML description for lnetctl, it will attempt to create a valid TCP/..."

Malcolm: Created page with "If no explicit configuration is supplied to LNet, either through a modprobe options file or a YAML description for `lnetctl`, it will attempt to create a valid TCP/..."