LNET Selftest - Revision history

Elliswilson at 16:26, 2 December 2024

2024-12-02T16:26:11Z

← Older revision		Revision as of 08:26, 2 December 2024
Line 239:		Line 239:

	<pre style="overflow-x:auto;">		<pre style="overflow-x:auto;">
	#!/bin/sh		#!/bin/bash
	#		#
	# Simple wrapper script for LNET Selftest		# Simple wrapper script for LNET Selftest

Elliswilson at 15:58, 2 December 2024

2024-12-02T15:58:51Z

← Older revision		Revision as of 07:58, 2 December 2024
Line 234:		Line 234:

	== References ==		== References ==
	* Lustre Manual https://wiki.hpdd.intel.com/display/PUB/Documentation		* [https://doc.lustre.org/lustre_manual.xhtml#lnetselftest Testing Lustre Network Performance (LNet Self-Test)]
	** [https://~~build~~.~~hpdd~~.~~intel.com/job/lustre-manual/lastSuccessfulBuild/artifact~~/lustre_manual.xhtml#lnetselftest LNet-~~selftest chapter~~]

	== Appendix: LNET Selftest Wrapper ==		== Appendix: LNET Selftest Wrapper ==

Malcolm at 02:01, 20 October 2017

2017-10-20T02:01:38Z

← Older revision		Revision as of 18:01, 19 October 2017
Line 122:		Line 122:

	Run the script several times, changing the concurrency setting with at the start of every new run. Use the sequence 1, 2, 4, 8, 16, 32, 64, 128. Modify the output filename for each run so that it is clear what results have been captured into each file.		Run the script several times, changing the concurrency setting with at the start of every new run. Use the sequence 1, 2, 4, 8, 16, 32, 64, 128. Modify the output filename for each run so that it is clear what results have been captured into each file.

			== LNET Selftest (1:1 Matrix) ==

			The following command invocation will automatically run a series of benchmarks, testing the connections, in pairs, between all of the nodes mentioned in the list. The <code>awk</code> script generates a list of all unique permutations of host pairs, and then launches the <code>lst-bench.sh</code> script for each pair.

			<pre style="overflow-x:auto;">
			echo "<NID 1> <NID 2> ... <NID N>" \| \
			awk '{for (i=1;i<=NF;i++)T[i]=$i} END {for (i=1; i<NF; i++) {for (j=i+1; j<=NF; j++) system("LFROM="T[i]" LTO="T[j]" ./lst-bench.sh")}}'
			</pre>

	== LNET Selftest Read (Many:1) ==		== LNET Selftest Read (Many:1) ==

Malcolm: /* Appendix: LNET Selftest Wrapper */

2017-10-19T22:23:01Z

Appendix: LNET Selftest Wrapper

← Older revision		Revision as of 14:23, 19 October 2017
Line 235:		Line 235:
	# Simple wrapper script for LNET Selftest		# Simple wrapper script for LNET Selftest
	#		#

	#~~Output file~~		# Parameters are supplied as environment variables
	~~ST=lst~~-~~output-$~~(~~date +%Y-%m-%d-%H~~:%M:~~%S)~~		# The defaults are reasonable for quick verification.
			# For in-depth benchmarking, increase the time (TM)
			# variable to e.g. 60 seconds, and iterate over
			# concurrency to find optimal values.
			#
			# Reference: http://wiki.lustre.org/LNET_Selftest

	# Concurrency		# Concurrency
	CN=64		CN=${CN:-32}
	#Size		#Size
	SZ=1M		SZ=${SZ:-1M}
	# Length of time to run test (secs)		# Length of time to run test (secs)
	TM=30		TM=${TM:-10}
	# Which BRW test to run (read or write)		# Which BRW test to run (read or write)
	BRW=read		BRW=${BRW:-"read"}
	# Checksum calculation (simple or full)		# Checksum calculation (simple or full)
	CKSUM=simple		CKSUM=${CKSUM:-"simple"}

	# The LST "from" list -- e.g. Lustre clients. Space separated list of NIDs.		# The LST "from" list -- e.g. Lustre clients. Space separated list of NIDs.
	LFROM="10.10.2.21@tcp"		# LFROM="10.10.2.21@tcp"
			LFROM=${LFROM:?ERROR: the LFROM variable is not set}
	# The LST "to" list -- e.g. Lustre servers. Space separated list of NIDs.		# The LST "to" list -- e.g. Lustre servers. Space separated list of NIDs.
	LTO="10.10.2.22@tcp"		# LTO="10.10.2.22@tcp"
			LTO=${LTO:?ERROR: the LTO variable is not set}

	### End of customisation.		### End of customisation.

	export LST_SESSION=$$		export LST_SESSION=$$
	echo LST_SESSION = ${LST_SESSION}		echo LST_SESSION = ${LST_SESSION}
Line 263:		Line 270:
	lst add_group lto ${LTO}		lst add_group lto ${LTO}
	lst add_batch bulk_${BRW}		lst add_batch bulk_${BRW}
	lst add_test --batch bulk_${BRW} --from lfrom --to lto brw ${BRW} --concurrency=${CN} check=${CKSUM} size=${SZ}		lst add_test --batch bulk_${BRW} --from lfrom --to lto brw ${BRW} \
			--concurrency=${CN} check=${CKSUM} size=${SZ}
	lst run bulk_${BRW}		lst run bulk_${BRW}
	echo -n "Capturing statistics for ${TM} secs "		echo -n "Capturing statistics for ${TM} secs "
	lst stat lfrom lto ~~2>&1 > ${ST}~~ &		lst stat lfrom lto &
	LSTPID=$!		LSTPID=$!
	# Delay loop with interval markers displayed every 5 secs. Test time is rounded		# Delay loop with interval markers displayed every 5 secs.
	# up to the nearest 5 seconds.		# Test time is rounded up to the nearest 5 seconds.
	i=1		i=1
	j=$((${TM}/5))		j=$((${TM}/5))
Line 275:		Line 283:
	while [ $i -le $j ]; do		while [ $i -le $j ]; do
	sleep 5		sleep 5
	~~echo -n "."~~
	let i++		let i++
	done		done
			kill ${LSTPID} && wait ${LISTPID} >/dev/null 2>&1
	echo		echo
	~~kill ${LSTPID}~~
	lst show_error lfrom lto		lst show_error lfrom lto
	lst stop bulk_${BRW}		lst stop bulk_${BRW}

Malcolm: /* Preparation */

2017-06-26T03:01:02Z

Preparation

← Older revision		Revision as of 19:01, 25 June 2017
Line 23:		Line 23:
	</pre>		</pre>

	~~Dependencies~~ are automatically resolved and loaded by modprobe. This will make sure all the necessary modules are loaded: <code>libcfs</code>, <code>lnet</code>, <code>lnet_selftest</code> and the kernel lustre network driver (LND) appropriate to the fabric or fabrics upon which LNet operates (e.g. ksocklnd for TCP/IP, ko2iblnd for RDMA fabrics).		Kernel module dependencies are automatically resolved and loaded by modprobe. This will make sure all the necessary modules are loaded: <code>libcfs</code>, <code>lnet</code>, <code>lnet_selftest</code> and the kernel lustre network driver (LND) appropriate to the fabric or fabrics upon which LNet operates (e.g. ksocklnd for TCP/IP, ko2iblnd for RDMA fabrics).

	Identify a "console" node from which to conduct the tests. This is the single system from which all LNET selftest commands will be executed. The console node owns the LNET selftest session and there should be only one active session on the network for a given set of nodes. Make sure that each session is isolated from network activity that might distort the results, and do not run any workloads except for LNet-selftest on machines participating in the session.		Identify a "console" node from which to conduct the tests. This is the single system from which all LNET selftest commands will be executed. The console node owns the LNET selftest session and there should be only one active session on the network for a given set of nodes. Make sure that each session is isolated from network activity that might distort the results, and do not run any workloads except for LNet-selftest on machines participating in the session.

Malcolm at 06:56, 23 June 2017

2017-06-23T06:56:27Z

← Older revision		Revision as of 22:56, 22 June 2017
Line 224:		Line 224:
	This is the entire test case code, except for the <code>lst</code> kernel module load. Just change the NIDs of each <code>group</code> to run.		This is the entire test case code, except for the <code>lst</code> kernel module load. Just change the NIDs of each <code>group</code> to run.

			== References ==
			* Lustre Manual https://wiki.hpdd.intel.com/display/PUB/Documentation
			** [https://build.hpdd.intel.com/job/lustre-manual/lastSuccessfulBuild/artifact/lustre_manual.xhtml#lnetselftest LNet-selftest chapter]

	== Appendix: LNET Selftest Wrapper ==		== Appendix: LNET Selftest Wrapper ==

Line 280:		Line 284:
	lst end_session		lst end_session
	</pre>		</pre>

			[[Category:Benchmarking]]

Malcolm: Created page with "== Description == LNET Selftest is a network verification application for Lustre that performs 2 principal functions: * Confirm the correctness of the Lustre Networking confi..."

2017-06-23T06:50:52Z

Created page with "== Description == LNET Selftest is a network verification application for Lustre that performs 2 principal functions: * Confirm the correctness of the Lustre Networking confi..."

New page

== Description ==

LNET Selftest is a network verification application for Lustre that performs 2 principal functions:
* Confirm the correctness of the Lustre Networking configuration of a given machine
* Measure the performance of the connection between a set of machines

LNET Selftest requires that the Lustre software is installed and the LNET kernel module is configured on each of the machines that will participate in the Lustre network. A Lustre file system is not required in order to verify LNet functionality with the LNet-selftest benchmark.

== Purpose ==

LNET-selftest is used to verify that the Lustre networking (LNET) module is properly installed and configured, as well as to establish the performance of the underlying hardware that supports Lustre on a computer system. LNET-selftest is very useful in determining the performance of the networking layer, which is essential when conducting a survey of a Lustre file system installation, either for benchmarking purposes or for identifying bottlenecks or limitations in the data pipeline from client to server.

There are two test types that the software supports: "<code>bulk read-write</code>" (BRW) and "<code>ping</code>". From the [https://wiki.hpdd.intel.com/display/PUB/Documentation Lustre manual]:
* <code>ping</code> - a <code>ping</code> generates a short request message, which results in a short response. Pings are useful to determine latency and small message overhead and to simulate Lustre metadata traffic.
* <code>brw</code> - in a <code>brw</code> ('bulk read write') test, data is transferred from the target machine to the source machine (<code>brwread</code>) or data is transferred from the source to the target (<code>brwwrite</code>). The size of the bulk transfer is set using the <code>size</code> parameter. A BRW test is useful to determine network bandwidth and to simulate Lustre I/O traffic.

== Preparation ==

The LNET Selftest kernel module must be installed and loaded on all machines in the test before the application is started. Identify the set of all systems that will participate in a session and ensure that the kernel module has been loaded. To load the kernel module run this command:

<pre style="overflow-x:auto;">
modprobe lnet_selftest
</pre>

Dependencies are automatically resolved and loaded by modprobe. This will make sure all the necessary modules are loaded: <code>libcfs</code>, <code>lnet</code>, <code>lnet_selftest</code> and the kernel lustre network driver (LND) appropriate to the fabric or fabrics upon which LNet operates (e.g. ksocklnd for TCP/IP, ko2iblnd for RDMA fabrics).

Identify a "console" node from which to conduct the tests. This is the single system from which all LNET selftest commands will be executed. The console node owns the LNET selftest session and there should be only one active session on the network for a given set of nodes. Make sure that each session is isolated from network activity that might distort the results, and do not run any workloads except for LNet-selftest on machines participating in the session.

It is strongly recommended that a survey of overall network health, and analysis of raw network performance between the target machines are carried out prior to running the LNET-selftest benchmark. Fabric health is critically important to preserving performance and reliability in a distributed system. Baseline benchmarking of the fabric will help to identify and measure any performance overhead introduced by LNET and provides a point of comparison between the low-level network capability and the results from the LNet benchmarks. If results are not within expectations, it is important to establish a measurement that is independent of Lustre or LNet, in order to isolate root causes quickly and effectively.

== Using the Wrapper Script ==

Use the LNET Selftest wrapper in the [[#Appendix:_LNET_Selftest_Wrapper|appendix]] to execute the test cases referenced in this document. The header of the script has some variables that need to be set in accordance with the target environment. Without changes, the script is very unlikely to operate correctly, if at all. Here is a listing of the header:
<pre style="overflow-x:auto;">
#Output file
ST=lst-output-$(date +%Y-%m-%d-%H:%M:%S)
# Concurrency
CN=64
#Size
SZ=1M
# Length of time to run test (secs)
TM=30
# Which BRW test to run (read or write)
BRW=read
# Checksum calculation (simple or full)
CKSUM=simple
# The LST "from" list -- e.g. Lustre clients. Space separated list of NIDs.
LFROM="10.73.2.21@tcp"
# The LST "to" list -- e.g. Lustre servers. Space separated list of NIDs.
LTO="10.73.2.22@tcp"
</pre>

=== Notes ===

* '''<code>CN</code>''': the concurrency setting simulates the number of threads performing communication. The LNET Selftest default is 1, which is not enough to properly exercise the connection. Set to at least 16, but experiment with higher values (32 or 64 being reasonable choices).
* '''<code>SZ</code>''': the size setting determines the size of the IO transaction. For bandwidth (throughput) measurements, use 1M.
* '''<code>TM</code>''': test time in seconds– how long to run the benchmark for. Set to a reasonable number in order to ensure collection of sufficient data to extrapolate a meaningful average (at least 60 seconds).
* '''<code>BRW</code>''': The Bulk Read/Write test to use. There are only two choices "read" or "write".
* '''<code>CKSUM</code>''': The checksum checking method. Choose either "simple" or "full".
* '''<code>LFROM</code>''': a space-separated list of NIDs that represent the "from" list (or source) in LNET Selftest. This is often a set of clients.
* '''<code>LTO</code>''': a space-separated list of NIDs that represent the "to" list (or destination) in LNET Selftest. This is often a set of servers.

'''Note:''' The [[#Appendix:_LNET_Selftest_Wrapper|wrapper]] script does not validate input parameters or changes to variables at this time. There is no pre-parsing of variables and will not clean up the environment on error. Use with caution and as a reference for how to operate the LNET Selftest application.

In the sections following this one, the parameters required for the [[#Appendix:_LNET_Selftest_Wrapper|LNET selftest wrapper]] will be displayed first, followed by a description of the manual command-line process.

== Single Client Throughput – LNET Selftest Read (2 Nodes, 1:1) ==

Used to establish point to point unidirectional read performance between two nodes.

Set the wrapper up as follows:

<pre style="overflow-x:auto;">
#Output file
ST=lst-output-$(date +%Y-%m-%d-%H:%M:%S)
# Concurrency
CN=32
#Size
SZ=1M
# Length of time to run test (secs)
TM=60
# Which BRW test to run (read or write)
BRW=read
# Checksum calculation (simple or full)
CKSUM=simple
# The LST "from" list -- e.g. Lustre clients. Space separated list of NIDs.
LFROM="10.73.2.21@tcp"
# The LST "to" list -- e.g. Lustre servers. Space separated list of NIDs.
LTO="10.73.2.22@tcp"
</pre>

Change the <code>LFROM</code> and <code>LTO</code> lists as required.

Run the script several times, changing the concurrency setting with at the start of every new run. Use the sequence 1, 2, 4, 8, 16, 32, 64, 128. Modify the output filename for each run so that it is clear what results have been captured into each file.

== Single Client Throughput – LNET Selftest Write (2 Nodes, 1:1) ==

Used to establish point to point unidirectional write performance between two nodes.

Set the wrapper up as follows:

<pre style="overflow-x:auto;">
#Output file
ST=lst-output-$(date +%Y-%m-%d-%H:%M:%S)
# Concurrency
CN=32
#Size
SZ=1M
# Length of time to run test (secs)
TM=60
# Which BRW test to run (read or write)
BRW=write
# Checksum calculation (simple or full)
CKSUM=simple
# The LST "from" list -- e.g. Lustre clients. Space separated list of NIDs.
LFROM="10.73.2.21@tcp"
# The LST "to" list -- e.g. Lustre servers. Space separated list of NIDs.
LTO="10.73.2.22@tcp"
</pre>

Change the <code>LFROM</code> and <code>LTO</code> lists as required.

Run the script several times, changing the concurrency setting with at the start of every new run. Use the sequence 1, 2, 4, 8, 16, 32, 64, 128. Modify the output filename for each run so that it is clear what results have been captured into each file.

== LNET Selftest Read (Many:1) ==
Many clients, one server read test. Find saturation point for one server. To test, edit the wrapper variables:
* Set <code>BRW=read</code>
* Add a space-separated list of NIDS to the <code>LFROM</code> variable, using quotes to encapsulate the entire string.

== LNET Selftest Write (Many:1) ==
Many clients, one server write test. Find saturation point for one server. Set BRW=write, and add a space-separated list of NIDS to the LFROM variable, using quotes to encapsulate the entire string.

== LNET Selftest Read (Many:Many) ==
Many clients, many servers read test. From initial saturation point for many clients to a single server, grow number of servers and re-test. Grow number of clients to meet saturation point of server pool.

== LNET Selftest Write (Many:Many) ==
Many clients, many servers write test. From initial saturation point for many clients to a single server, grow number of servers and re-test. Grow number of clients to meet saturation point of server pool.

== Running the Benchmark ==
The test cases referenced in this document use a small wrapper script, the listing for which is provided as an [[#Appendix:_LNET_Selftest_Wrapper|appendix]] to this document. The script is intended to simplify and to a certain extent automate the setup of an LNET selftest session, using a small number of well-defined variables to isolate the parameters of interest to benchmarking requirements.

In some circumstances, it may be necessary or preferable to run the LNET selftest benchmark without recourse to the wrapper. The application is very flexible and can be used to create more complex scenarios than those afforded by the wrapper, so the following section describes how to run LNET selftest entirely from the command line.

<ol>
<li>Load the <code>lnet_selftest</code> kernel module on all nodes that are participating in the benchmark.</li>
<li>On the LNET Selftest console node, create a new session. For example:
<pre style="overflow-x:auto;">
export LST_SESSION=$$
lst new_session twonoderead
</pre>
This creates a session called <code>twonoderead</code>. The name of the session is arbitrary but should reflect the type of test being created. The <code>LST_SESSION</code> variable is mandatory – most <code>lst</code> commands will fail to run if it is not set. It is common practice to use the PID of the command shell (<code>$$</code>) as the session identifier in the environment variable but one could also use some other unique identifier, e.g. <code>`date +%s`</code>.
</li>
<li>Define the LNET host groups. Each group is comprised of one or more hosts referenced by their NID. Multiple NIDs can be specified as a space-separated list and/or using pattern matching rules. Group names are arbitrary but should be relevant to the members of that group. In the following example, there is a group called <code>client</code> and a group called <code>server</code>, each with a single host/NID:
<pre style="overflow-x:auto;">
lst add_group client 192.168.0.21@o2ib
lst add_group server 192.168.0.11@o2ib
</pre>
</li>
<li>Create a new batch (a batch is a container for grouping a set of tests). Name the group after the type of test that will be run. In this example, the batch is called "<code>bulk_read</code>":
<pre style="overflow-x:auto;">
lst add_batch bulk_read
</pre>
</li>
<li>Add a test to the batch:
<pre style="overflow-x:auto;">
lst add_test --batch bulk_read --from client --to server brw read check=full size=1M --concurrency=16
</pre>
Multiple tests can be added to a single batch. Tests will be run in parallel.
* The command adds a "<code>brw read</code>" test to the batch "<code>bulk_read</code>".
* This test runs from the nodes in the <code>client</code> group to the nodes in the <code>server</code> group.
* "<code>check=full</code>" forces the test to validate the data (using a checksum).
* "<code>size=1M</code>" sets the size of each IO transaction to 1MB.
* Setting the concurrency simulates running multiple threads. One can also set distribution for running one to many tests (refer to the [https://wiki.hpdd.intel.com/display/PUB/Documentation Lustre manual]).
There are 3 test types: <code>brw read</code>, <code>brw write</code> and <code>ping</code>. Results will be affected by the amount of concurrency defined in the test.
</li>
<li>Execute the tests in the "bulk_read" batch:
<pre style="overflow-x:auto;">
lst run bulk_read
</pre>
"<code>lst run</code>" is launched as a background process and returns to the command prompt immediately. The test will continue to run until either "<code>lst stop</code>" or "<code>lst end_session</code>" is run. No output is returned; use "<code>lst stat</code>" to monitor.
</li>
<li>To monitor the execution of a batch, us the "lst stat" command:
<pre style="overflow-x:auto;">
lst stat <group name> [<group name> ...]
</pre>
<code>lst stat</code> will continue to display output until the process is killed (<code>Control-C</code>). There must be at least one group listed on the command line. Multiple groups can be added as a space-separated list.
</li>
<li>Capture 30 seconds worth of output for groups "client" and "server" to a file:
<pre style="overflow-x:auto;">
lst stat client server > /tmp/lst-out-`date +s` &
LSTPID=$!
sleep 30
kill $LSTPID
</pre>
</li>
<li>Stop the batch:
<pre style="overflow-x:auto;">
lst stop bulk_read
</pre>
This step is optional if there are no more tests to run for the active session, since ending a session will end any active tests.
</li>
<li>Clean up the session when complete:
<pre style="overflow-x:auto;">
lst end_session
</pre>
</li>
<li>Remove the <code>lnet_selftest</code> kernel module from all of the machines.</li>
</ol>

A complete example in one code block:
<pre style="overflow-x:auto;">
export LST_SESSION=$$
lst new_session twonoderead
lst add_group client 192.168.0.21@o2ib
lst add_group server 192.168.0.11@o2ib
lst add_batch bulk_read
lst add_test --batch bulk_read --from client --to server brw read check=full size=1M
lst run bulk_read
lst stat client server & sleep 30; kill $!
lst stop bulk_read
lst end_session
</pre>

This is the entire test case code, except for the <code>lst</code> kernel module load. Just change the NIDs of each <code>group</code> to run.

== Appendix: LNET Selftest Wrapper ==

<pre style="overflow-x:auto;">
#!/bin/sh
#
# Simple wrapper script for LNET Selftest
#

#Output file
ST=lst-output-$(date +%Y-%m-%d-%H:%M:%S)

# Concurrency
CN=64
#Size
SZ=1M
# Length of time to run test (secs)
TM=30
# Which BRW test to run (read or write)
BRW=read
# Checksum calculation (simple or full)
CKSUM=simple

# The LST "from" list -- e.g. Lustre clients. Space separated list of NIDs.
LFROM="10.10.2.21@tcp"
# The LST "to" list -- e.g. Lustre servers. Space separated list of NIDs.
LTO="10.10.2.22@tcp"

### End of customisation.

export LST_SESSION=$$
echo LST_SESSION = ${LST_SESSION}
lst new_session lst${BRW}
lst add_group lfrom ${LFROM}
lst add_group lto ${LTO}
lst add_batch bulk_${BRW}
lst add_test --batch bulk_${BRW} --from lfrom --to lto brw ${BRW} --concurrency=${CN} check=${CKSUM} size=${SZ}
lst run bulk_${BRW}
echo -n "Capturing statistics for ${TM} secs "
lst stat lfrom lto 2>&1 > ${ST} &
LSTPID=$!
# Delay loop with interval markers displayed every 5 secs. Test time is rounded
# up to the nearest 5 seconds.
i=1
j=$((${TM}/5))
if [ $((${TM}%5)) -ne 0 ]; then let j++; fi
while [ $i -le $j ]; do
sleep 5
echo -n "."
let i++
done
echo
kill ${LSTPID}
lst show_error lfrom lto
lst stop bulk_${BRW}
lst end_session
</pre>