Difference between revisions of "LNet Router Config Guide2"

From Lustre Wiki
Jump to navigation Jump to search
(Created page with "This document provides procedures to configure and tune an LNet router. It will also cover detailed instructions set on setting up connectivity of an Infiniband network to Int...")
 
Line 20: Line 20:
 
* Routers on LAN1 and LAN2 at 10.10.0.20, 10.10.0.21 and 10.20.0.29, 10.20.0.30 respectively
 
* Routers on LAN1 and LAN2 at 10.10.0.20, 10.10.0.21 and 10.20.0.29, 10.20.0.30 respectively
  
The network configuration on the nodes can be done either by adding the module parameters in lustre.conf ('''/etc/modprobe.d/lustre.conf ''') or dynamically by using the ''' lnetctl ''' command utility.
+
The network configuration on the nodes can be done either by adding the module parameters in lustre.conf ( '''/etc/modprobe.d/lustre.conf ''' ) or dynamically by using the ''' lnetctl ''' command utility.
 
Also, current configuration can be exported to a YAML format file and then the configuration can be set by importing that YAML file anytime needed.
 
Also, current configuration can be exported to a YAML format file and then the configuration can be set by importing that YAML file anytime needed.
  
 
=== 2.1 Network Configuration by adding module parameters in lustre.conf ===
 
=== 2.1 Network Configuration by adding module parameters in lustre.conf ===
 +
 +
<nowiki>
 +
Servers:
 +
options lnet networks="o2ib1(ib0)" routes="o2ib2 [email protected]"
 +
 +
Routers:
 +
options lnet networks="o2ib1(ib0),o2ib2(ib1)" "forwarding=enabled"
 +
 +
Clients:
 +
options lnet networks="o2ib2(ib0)" routes="o2ib1 [email protected]" </nowiki>

Revision as of 13:36, 12 October 2017

This document provides procedures to configure and tune an LNet router. It will also cover detailed instructions set on setting up connectivity of an Infiniband network to Intel OPA nodes using LNet router.

1. LNet

LNet supports different network types like Ethernet, InfiniBand, Intel Omni-Path and other proprietary network technologies such as the Cray’s Gemini. It routes LNet messages between different LNet networks using LNet routing. LNet’s routing capabilities provide an efficient protocol to enable bridging between different types of networks. LNet is part of the Linux kernel space and allows for full RDMA throughput and zero copy communications when supported by underlying network. Lustre can initiate a multi-OST read or write using a single Remote Procedure Call (RPC), which allows the client to access data using RDMA at near peak bandwidth rates. With Multi-Rail (MR) feature implemented in Lustre 2.10.X, it allows for multiple interfaces of same type on a node to be grouped together under the same LNet (ex tcp0, o2ib0, etc.). These interfaces can then be used simultaneously to carry LNet traffic. MR also has the ability to utilize multiple interfaces configured on different networks. For example, OPA and MLX interfaces can be grouped under their respective LNet and then can be utilized with MR feature to carry LNet traffic simultaneously.


LNet Configuration Example

An LNet router is a specialized Lustre client where Lustre file system is not mounted and only the LNet is running. A single LNet router can serve different file systems.



For the above example:

  • Servers are on LAN1, a Mellanox based InfiniBand network – 10.10.0.0/24
  • Clients are LAN2, an Intel OPA network – 10.20.0.0/24
  • Routers on LAN1 and LAN2 at 10.10.0.20, 10.10.0.21 and 10.20.0.29, 10.20.0.30 respectively

The network configuration on the nodes can be done either by adding the module parameters in lustre.conf ( /etc/modprobe.d/lustre.conf ) or dynamically by using the lnetctl command utility. Also, current configuration can be exported to a YAML format file and then the configuration can be set by importing that YAML file anytime needed.

2.1 Network Configuration by adding module parameters in lustre.conf

Servers:
options lnet networks="o2ib1(ib0)" routes="o2ib2 [email protected]"
 
Routers:
options lnet networks="o2ib1(ib0),o2ib2(ib1)" "forwarding=enabled"
 
Clients:
options lnet networks="o2ib2(ib0)" routes="o2ib1 [email protected]"