Lustre Networking (LNET) Overview - Revision history

Adilger: clarify usage of TCP port 988 is only for socklnd

2021-08-21T08:00:34Z

clarify usage of TCP port 988 is only for socklnd

Malcolm at 05:45, 31 August 2017

2017-08-31T05:45:33Z

← Older revision		Revision as of 21:45, 30 August 2017
Line 46:		Line 46:
	[[Category:Lustre File System Components]]		[[Category:Lustre File System Components]]
	[[Category:Lustre Networking (LNet)]]		[[Category:Lustre Networking (LNet)]]
			[[Category:Lustre Systems Administration]]

Malcolm at 21:44, 9 August 2017

2017-08-09T21:44:07Z

← Older revision		Revision as of 13:44, 9 August 2017
Line 45:		Line 45:

	[[Category:Lustre File System Components]]		[[Category:Lustre File System Components]]
			[[Category:Lustre Networking (LNet)]]

Malcolm at 05:19, 3 August 2017

2017-08-03T05:19:48Z

← Older revision		Revision as of 21:19, 2 August 2017
Line 42:		Line 42:
	</pre>		</pre>

	Note ~~that even~~ though the <code>o2ib</code> LNet driver uses RDMA for communications, TCP is used to establish the initial connection with a peer using the fabric’s IP upper level protocol. After the initial connection, the <code>o2ib</code> LND uses RDMA for all further communications. The LNet NID of an <code>o2ib</code> device also makes use of an IPv4 address to identify the NID, just as for the TCP socket LND~~. By default, LNet uses TCP port 988 to create connections, and this must not be blocked by any firewalls~~.		'''Note:''' Even though the <code>o2ib</code> LNet driver uses RDMA for communications, TCP is used to establish the initial connection with a peer using the fabric’s IP upper level protocol. After the initial connection, the <code>o2ib</code> LND uses RDMA for all further communications. By default, LNet uses TCP port 988 to create connections, and this must not be blocked by any firewalls. The LNet NID of an <code>o2ib</code> device also makes use of an IPv4 address to identify the NID, just as for the TCP socket LND.

	[[Category:Lustre File System Components]]		[[Category:Lustre File System Components]]

Malcolm: Created page with "Lustre’s network communication protocol, LNet, was originally derived from a project called Portals, developed by Sandia National Labs. LNet is designed to be lightweight an..."

2017-08-02T04:19:05Z

Created page with "Lustre’s network communication protocol, LNet, was originally derived from a project called Portals, developed by Sandia National Labs. LNet is designed to be lightweight an..."

New page

Lustre’s network communication protocol, LNet, was originally derived from a project called Portals, developed by Sandia National Labs. LNet is designed to be lightweight and efficient and supports message passing for RPC request processing and RDMA for bulk data movement. All network metadata and file data I/O is managed through the LNet protocol and API. LNet is lightweight and versatile, capable of operating over different network fabrics, including Ethernet, InfiniBand and Intel® OPA. In common with the other major components of Lustre, LNet is implemented as a Linux kernel module.

All participants in a Lustre file system, including servers and clients, must have a valid LNet configuration and be connected either directly on a common network fabric, or via a router between networks.

To support the various types of networks, LNet has a low-level device layer called a Lustre Network Driver (LND), implemented as a pluggable driver module. The LND provides an interface abstraction between the upper level LNet protocol and the kernel device driver for the network interface. Each low-level network protocol requires a separate LND and multiple LNDs can be active on a host simultaneously, if the server requires access to more than one type of network. The most commonly used LNet drivers are the <code>ksocklnd.ko</code> module for TCP/IP networks, and the <code>ko2iblnd.ko</code> module for RDMA networks that make use of the OpenFabrics Enterprise Distribution (OFED) network driver. The <code>ksocklnd.ko</code> module is usually abbreviated to <code>socklnd</code> or referred to as the sockets LND. The <code>ko2iblnd.ko</code> module (usually referred to as the <code>o2ib</code> LND or just called <code>o2ib</code>) supports fabrics running InfiniBand, Omni-Path, and RDMA over Converged Ethernet (RoCE). The letter “k” prefix in the LND names is there to emphasize that these are kernel modules, which is true for the majority of the Lustre software stack. It is often omitted from documentation, to improve readability.

LNet also supports the ability to route Lustre communications between different networks. Dedicated computers called LNet routers can direct traffic between multiple LNets.

Network interfaces on computer systems running LNet are addressed with a node identifier (the IPv4 address of a network device on the host) and also with a protocol identifier and network number for that protocol. The format is:

<pre style="overflow-x:auto;">
<IPv4 address>@<LND protocol><lnd#>
</pre>

The complete string is called an LNet Network Identifier (NID) and it uniquely defines an interface for a host on an LNet communications fabric.

The following example is a NID for an Ethernet interface:

<pre style="overflow-x:auto;">
10.70.207.11@tcp0
</pre>

This is a unique ID for a host with an Ethernet NIC on LNet tcp0 using the socket LND (as indicated by the tcp network type). A second configuration could also be added for a different interface:

<pre style="overflow-x:auto;">
192.168.70.11@tcp1
</pre>

The number appended to the LND protocol type must be a non-negative integer and must be the same for all Lustre hosts (client and server) that participate on the same network. For example, the following two NIDs are not on the same LNet:

<pre style="overflow-x:auto;">
10.70.207.11@tcp0
10.70.207.11@tcp1
</pre>

Even though each NID has the same IPv4 address, they belong to two different LNets, because the LND instance numbers are different (<code>tcp0</code> and <code>tcp1</code>, respectively).

RDMA networks have a similar NID format. For example:

<pre style="overflow-x:auto;">
10.20.30.11@o2ib0
</pre>

Note that even though the <code>o2ib</code> LNet driver uses RDMA for communications, TCP is used to establish the initial connection with a peer using the fabric’s IP upper level protocol. After the initial connection, the <code>o2ib</code> LND uses RDMA for all further communications. The LNet NID of an <code>o2ib</code> device also makes use of an IPv4 address to identify the NID, just as for the TCP socket LND. By default, LNet uses TCP port 988 to create connections, and this must not be blocked by any firewalls.

[[Category:Lustre File System Components]]

← Older revision		Revision as of 00:00, 21 August 2021
Line 1:		Line 1:
	Lustre’s network communication protocol, LNet, was originally derived from a project called Portals, developed by Sandia National Labs. LNet is designed to be lightweight and efficient and supports message passing for RPC request processing and RDMA for bulk data movement. All network metadata and file data I/O is managed through the LNet protocol and API. LNet is ~~lightweight and~~ versatile, capable of operating over different network fabrics, including Ethernet, InfiniBand and Intel® OPA. In common with the other major components of Lustre, LNet is implemented as a Linux kernel module.		Lustre’s network communication protocol, LNet, was originally derived from a project called Portals, developed by Sandia National Labs. LNet is designed to be lightweight and efficient and supports message passing for RPC request processing and RDMA for bulk data movement. All network metadata and file data I/O is managed through the LNet protocol and API. LNet is versatile, capable of operating over different network fabrics, including Ethernet, InfiniBand and Intel® OPA. In common with the other major components of Lustre, LNet is implemented as a Linux kernel module.

	All ~~participants~~ in a Lustre file system, including servers and clients, must have a valid LNet configuration and be connected either directly on a common network fabric, or via a router between networks.		All nodes in a Lustre file system, including servers and clients, must have a valid LNet configuration and be connected either directly on a common network fabric, or via a router between networks.

	To support the various types of networks, LNet has a low-level device layer called a Lustre Network Driver (LND), implemented as a pluggable driver module. The LND provides an interface abstraction between the upper level LNet protocol and the kernel device driver for the network interface. Each low-level network protocol requires a separate LND and multiple LNDs can be active on a host simultaneously, if the server requires access to more than one type of network. The most commonly used LNet drivers are the <code>ksocklnd.ko</code> module for TCP/IP networks, and the <code>ko2iblnd.ko</code> module for RDMA networks that make use of the OpenFabrics Enterprise Distribution (OFED) network driver. The <code>ksocklnd.ko</code> module is usually abbreviated to <code>socklnd</code> ~~or referred to as the sockets LND~~. The <code>ko2iblnd.ko</code> module (usually referred to as the <code>o2ib</code> LND or just called <code>o2ib</code>) supports fabrics running InfiniBand, Omni-Path, and RDMA over Converged Ethernet (RoCE). The letter “k” prefix in the LND names is there to emphasize that these are kernel modules, which is true for the majority of the Lustre software stack. It is often omitted from documentation, to improve readability.		To support the various types of networks, LNet has a low-level device layer called a Lustre Network Driver (LND), implemented as a pluggable driver module. The LND provides an interface abstraction between the upper level LNet protocol and the kernel device driver for the network interface. Each low-level network protocol requires a separate LND and multiple LNDs can be active on a host simultaneously, if the server requires access to more than one type of network. The most commonly used LNet drivers are the <code>ksocklnd.ko</code> module for TCP/IP networks, and the <code>ko2iblnd.ko</code> module for RDMA networks that make use of the OpenFabrics Enterprise Distribution (OFED) network driver. The <code>ksocklnd.ko</code> module is usually abbreviated to <code>socklnd</code>. The <code>ko2iblnd.ko</code> module (usually referred to as the <code>o2ib</code> LND or just called <code>o2ib</code>) supports fabrics running InfiniBand, Omni-Path, and RDMA over Converged Ethernet (RoCE). The letter “k” prefix in the LND names is there to emphasize that these are kernel modules, which is true for the majority of the Lustre software stack. It is often omitted from documentation, to improve readability.

	LNet also supports the ability to route Lustre communications between different networks. Dedicated computers called LNet routers can direct traffic between multiple LNets.		LNet also supports the ability to route Lustre communications between different networks. Dedicated computers called LNet routers can direct traffic between multiple LNets.

	Network interfaces on computer systems running LNet are addressed with a node identifier ~~(the IPv4 address of a network device on the host)~~ and also with a protocol identifier and network number for that protocol. The format is:		Network interfaces on computer systems running LNet are addressed with a node identifier and also with a protocol identifier and network number for that protocol. The format is:

	<pre style="overflow-x:auto;">		<pre style="overflow-x:auto;">
	<~~IPv4~~ address>@<LND protocol><lnd#>		<address>@<LND protocol><lnd#>
	</pre>		</pre>

	The complete string is called an LNet Network Identifier (NID) and it uniquely defines an interface for a host on an LNet communications fabric.		For tcp and o2ib LNDs, the <code>address</code> is the IPv4 address of a network device on the host. Other LNDs may use other address formats, such as a simple integer representing the node number. The complete string is called an LNet Network Identifier (NID) and it uniquely defines an interface for a host on an LNet communications fabric.

	The following example is a NID for an Ethernet interface:		The following example is a NID for an Ethernet interface:
Line 42:		Line 42:
	</pre>		</pre>

	'''Note:''' Even though the <code>o2ib</code> LNet driver uses ~~RDMA~~ for communications, ~~TCP~~ is used to ~~establish~~ the initial connection with a peer ~~using the fabric’s IP upper level protocol~~. After the initial connection, the <code>o2ib</code> LND uses RDMA for all further communications. By default, ~~LNet~~ uses TCP port 988 to create connections, and this must not be blocked by any firewalls~~. The LNet NID of an <code>o2ib</code> device also makes use of an IPv4 address to identify the NID, just as for the TCP socket LND~~.		'''Note:''' Even though the <code>o2ib</code> LNet driver uses OFED Verbs for communications, the IP address of the IB interface is used to identify the IB interface for the initial connection with a peer. After the initial connection, the <code>o2ib</code> LND uses only RDMA for all further communications. By default, <code>socklnd</code> uses TCP port 988 to create connections, and this must not be blocked by any firewalls.

	[[Category:Lustre File System Components]]		[[Category:Lustre File System Components]]
	[[Category:Lustre Networking (LNet)]]		[[Category:Lustre Networking (LNet)]]
	[[Category:Lustre Systems Administration]]		[[Category:Lustre Systems Administration]]