Optimizing o2iblnd Performance - Revision history

Paf at 16:23, 29 July 2019

2019-07-29T16:23:33Z

← Older revision		Revision as of 09:23, 29 July 2019
Line 13:		Line 13:

	* <code>peer_credits=128</code> - the number of concurrent sends to a single peer		* <code>peer_credits=128</code> - the number of concurrent sends to a single peer
	* <code>peer_credits_hiw=64</code> - ~~Hold in Wait~~ – when to eagerly return credits		* <code>peer_credits_hiw=64</code> - High Water mark – when to eagerly return credits
	* <code>credits=1024</code> - the number of concurrent sends (to all peers)		* <code>credits=1024</code> - the number of concurrent sends (to all peers)
	* <code>concurrent_sends=256</code> - send work-queue sizing		* <code>concurrent_sends=256</code> - send work-queue sizing

Malcolm at 03:07, 26 September 2018

2018-09-26T03:07:32Z

← Older revision		Revision as of 20:07, 25 September 2018
Line 21:		Line 21:
	* <code>fmr_flush_trigger=512</code> - the dirty FMR pool flush trigger		* <code>fmr_flush_trigger=512</code> - the dirty FMR pool flush trigger
	* <code>fmr_cache=1</code> - enable FMR caching		* <code>fmr_cache=1</code> - enable FMR caching
	* <code>conns_per_peer=4</code> - create multiple queue pairs per peer to allow higher throughput from a single client. This is of most benefit to OPA interfaces, when coupled with the <code>krcvqs</code> parameter of the OPA <code>hfi1</code> kernel driver. The <code>hfi1</code> driver option <code>krcvqs</code> must also be set. It is recommended to set <code>krcvqs=8</code>. If queue-pair memory usage becomes excessive, reduce the <code>ko2iblnd</code> <code>conns_per_peer</code> value to <code>2</code> and ~~reduce~~ <code>krcvqs~~</code> to <code>4~~</code>.		* <code>conns_per_peer=4</code> - create multiple queue pairs per peer to allow higher throughput from a single client. This is of most benefit to OPA interfaces, when coupled with the <code>krcvqs</code> parameter of the OPA <code>hfi1</code> kernel driver. The <code>hfi1</code> driver option <code>krcvqs</code> must also be set. It is recommended to set <code>krcvqs=4</code>. In some cases, setting <code>krcvqs=8</code> will yield improved IO performance, but this can impact other workloads, especially on clients. If queue-pair memory usage becomes excessive, reduce the <code>ko2iblnd</code> <code>conns_per_peer</code> value to <code>2</code> and <code>krcvqs=2</code>.

	The default values used by Lustre if no parameters are given is:		The default values used by Lustre if no parameters are given is:

Malcolm at 22:59, 28 August 2018

2018-08-28T22:59:42Z

← Older revision		Revision as of 15:59, 28 August 2018
Line 41:		Line 41:

	<pre style="overflow-x:auto;">		<pre style="overflow-x:auto;">
	options hfi1 krcvqs=~~8 piothreshold~~=0 sge_copy_mode=2 wss_threshold=70		options hfi1 krcvqs=4 piothreshold=0 sge_copy_mode=2 wss_threshold=70
	</pre>		</pre>

	See also [https://jira.hpdd.intel.com/browse/LU-8943 LU-8943]		Some experimentation with the <code>krcvqs</code> parameter may be required to find the optimal balance of Lustre IO performance against other workloads. Lustre servers may derive additional performance from increasing the value up to 8. For Lustre clients, higher values can improve Lustre performance but might impact application performance.

			See also:

			* Intel® Omni-Path Performance Tuning User Guide from the OPA documentation bundle: [https://www.intel.sg/content/www/xa/en/support/articles/000016242/network-and-i-o/fabric-products.html End User Publications, Release Notes, and EULAs for Intel® Omni-Path Software]
			* [https://jira.hpdd.intel.com/browse/LU-8943 LU-8943]

	=== irqbalance ===		=== irqbalance ===

Malcolm: /* Additional Intel® Omni-Path Optimization */

2018-02-27T21:11:00Z

Additional Intel® Omni-Path Optimization

← Older revision		Revision as of 14:11, 27 February 2018
Line 48:		Line 48:
	=== irqbalance ===		=== irqbalance ===

	The purpose of <code>irqbalance</code> is to distribute hardware interrupts across processors on a multiprocessor system in order to increase performance. According to the Intel Omni-Path tuning guide, setting the irqbalance hint policy to <code>exact</code> can be beneficial to the <code>hfi1</code> receive and send DMA interrupt algorithms in the driver.		The purpose of <code>irqbalance</code> is to distribute hardware interrupts across processors on a multiprocessor system in order to increase performance. According to the Intel Omni-Path tuning guide ([https://www.intel.com/content/www/us/en/support/articles/000016242/network-and-i-o/fabric-products.html download pdf bundle]), setting the irqbalance hint policy to <code>exact</code> can be beneficial to the <code>hfi1</code> receive and send DMA interrupt algorithms in the driver.

	To install the <code>irqbalance</code> package, run the following command:		To install the <code>irqbalance</code> package, run the following command:

Malcolm at 05:18, 9 November 2017

2017-11-09T05:18:05Z

@@ Line 3: / Line 3: @@
 <pre style="overflow-x:auto;">
 alias ko2iblnd-opa ko2iblnd
-options ko2iblnd-opa peer_credits=128 peer_credits_hiw=64 credits=1024 concurrent_sends=256 ntx=2048 map_on_demand=32 fmr_pool_size=2048 fmr_flush_trigger=512 fmr_cache=1
+options ko2iblnd-opa peer_credits=128 peer_credits_hiw=64 credits=1024 concurrent_sends=256 ntx=2048 map_on_demand=32 fmr_pool_size=2048 fmr_flush_trigger=512 fmr_cache=1 conns_per_peer=4
 install ko2iblnd /usr/sbin/ko2iblnd-probe
@@ Line 21: / Line 21: @@
 * <code>fmr_flush_trigger=512</code> - the dirty FMR pool flush trigger
 * <code>fmr_cache=1</code> - enable FMR caching
 The default values used by Lustre if no parameters are given is:
@@ Line 34: / Line 35: @@
 '''Note:''' It is possible to use the <code>socklnd</code> driver on RDMA fabrics if there is an upper-level protocol that supports TCP/IP traffic, such as the IPoIB driver for InfiniBand fabrics. This use of <code>socklnd</code> on InfiniBand, RoCE, and Intel® OPA networks is not recommended because it will compromise the performance of LNet compared to the RDMA-based <code>o2iblnd</code>, and can have a negative impact on the stability of the resulting network connection. Instead, it is strongly recommended that <code>o2iblnd</code> is used wherever possible; it provides the highest performance with the lowest overheads on these fabrics.
 [[Category:Lustre Networking (LNet)]]
 [[Category:Lustre Systems Administration]]

Malcolm at 05:47, 31 August 2017

2017-08-31T05:47:18Z

← Older revision		Revision as of 22:47, 30 August 2017
Line 36:		Line 36:

	[[Category:Lustre Networking (LNet)]]		[[Category:Lustre Networking (LNet)]]
			[[Category:Lustre Systems Administration]]

Malcolm: Created page with "In addition to defining the LNet interfaces, the kernel module files can be used to supply parameters to other kernel modules used by Lustre. This is commonly used to supply t..."

2017-08-10T03:14:27Z

Created page with "In addition to defining the LNet interfaces, the kernel module files can be used to supply parameters to other kernel modules used by Lustre. This is commonly used to supply t..."

New page

In addition to defining the LNet interfaces, the kernel module files can be used to supply parameters to other kernel modules used by Lustre. This is commonly used to supply tuning optimizations to the LNet drivers, to maximize performance of the network interface. An example of this optimization can be seen in Lustre version 2.8.0 and later, in the file <code>/etc/modprobe.d/ko2iblnd.conf</code>, which includes the following:

<pre style="overflow-x:auto;">
alias ko2iblnd-opa ko2iblnd
options ko2iblnd-opa peer_credits=128 peer_credits_hiw=64 credits=1024 concurrent_sends=256 ntx=2048 map_on_demand=32 fmr_pool_size=2048 fmr_flush_trigger=512 fmr_cache=1

install ko2iblnd /usr/sbin/ko2iblnd-probe
</pre>

This configuration is automatically applied to the LNet kernel module when an Intel® Omni-Path interface is installed, but not when a different network interface is present. Please note that when a host is connected to more than one fabric sharing the same Lustre network driver, options set by <code>modprobe</code> will be applied to all interfaces using the same driver. To set individual, per-device tuning parameters, use the Dynamic LNet configuration utility, <code>lnetctl</code>, to configure the interfaces instead.

The following set of options has been defined to optimize the performance of Intel® Omni-Path Architecture. A detailed description is beyond the scope of this exercise, but the following summary provides an overview:

* <code>peer_credits=128</code> - the number of concurrent sends to a single peer
* <code>peer_credits_hiw=64</code> - Hold in Wait – when to eagerly return credits
* <code>credits=1024</code> - the number of concurrent sends (to all peers)
* <code>concurrent_sends=256</code> - send work-queue sizing
* <code>ntx=2048</code> - the number of message descriptors that are pre-allocated when the <code>ko2iblnd</code> module is loaded in the kernel
* <code>map_on_demand=32</code> - the number of noncontiguous memory regions that will be mapped into a virtual contiguous region
* <code>fmr_pool_size=2048</code> - the size of the Fast Memory registration (FMR) pool (must be <code>>= ntx/4</code>)
* <code>fmr_flush_trigger=512</code> - the dirty FMR pool flush trigger
* <code>fmr_cache=1</code> - enable FMR caching

The default values used by Lustre if no parameters are given is:

* <code>peer_credits=8</code>
* <code>peer_credits_hiw=8</code>
* <code>concurrent_sends=8</code>
* <code>credits=64</code>

Optimizations are applied automatically on detection of an Intel® high performance network interface. Some of the parameters, such as FMR, are incompatible with other devices, such as Mellanox InfiniBand products using the MLX5 driver. It can be disabled by setting <code>map_on_demand=0</code> (the default). The configuration file can be modified or deleted to meet the specific requirements of a given installation.

In general, the default <code>ko2iblnd</code> settings work well with Mellanox InfiniBand HCAs and no tuning is normally required. Architecture differences between Intel® fabrics and Mellanox mean that setting universal defaults is very difficult. Intel® OPA and Intel® True Scale Fabric have an architecture that favors lightweight, high-frequency message-passing communications, compared to Mellanox, which has historically placed an emphasis on throughput-oriented workloads. Because Mellanox InfiniBand has historically been the dominant high-speed fabric, LNet driver development has naturally tended in the past to align with this technology, aided by interfaces that are intended to support storage-like workloads. What the above settings do is tune the LNet driver for communications on Intel® fabrics, if present.

'''Note:''' It is possible to use the <code>socklnd</code> driver on RDMA fabrics if there is an upper-level protocol that supports TCP/IP traffic, such as the IPoIB driver for InfiniBand fabrics. This use of <code>socklnd</code> on InfiniBand, RoCE, and Intel® OPA networks is not recommended because it will compromise the performance of LNet compared to the RDMA-based <code>o2iblnd</code>, and can have a negative impact on the stability of the resulting network connection. Instead, it is strongly recommended that <code>o2iblnd</code> is used wherever possible; it provides the highest performance with the lowest overheads on these fabrics.

[[Category:Lustre Networking (LNet)]]