Testing HOWTO - Revision history

Elliswilson at 15:27, 14 September 2026

2026-09-14T15:27:56Z

← Older revision		Revision as of 08:27, 14 September 2026
Line 1:		Line 1:
	This HOWTO is intended to demonstrate the basics of configuring and running tests on a multi-node configuration. [[Test Descriptions \| There are many tests available in the Lustre Test Suites]], and the principles demonstrated in configuring your system and running the sanity test suite will apply to them all. We will also demonstrate adding additional configuration for feature specific test suites.		This HOWTO is intended to demonstrate the basics of configuring and running tests on a multi-node configuration. [[Test Descriptions \| There are many tests available in the Lustre Test Suites]], and the principles demonstrated in configuring your system and running the sanity test suite will apply to them all. We will also demonstrate adding additional configuration for feature specific test suites.

			{\| class='wikitable'
			\|-
			!Note: This page was written in 2018 using CentOS 7.1, which has reached End of Life. Current Lustre testing typically uses RHEL 8/9, Rocky Linux 8/9, or similar EL-based distributions. The general testing principles and configuration approach remain valid, but specific package URLs and OS-level commands may need adjustment for your platform. The <code>distro=el6</code> and <code>distro=el7</code> toolkit build links below are unlikely to still resolve; substitute the build for your distribution.
			\|}

	While these examples do use virtual machines, they are merely examples and the specifics should be easy to apply to real hardware with the prerequisites setup.		While these examples do use virtual machines, they are merely examples and the specifics should be easy to apply to real hardware with the prerequisites setup.

	==== System Configuration ====		==== System Configuration ====

	This HOWTO uses a cluster of six virtual machines running CentOS 7.1 with a recent lustre-master build to run the Lustre tests. Two clients, two MDS, two OSS. This enables testing of a wide variety of Lustre features.		This HOWTO uses a cluster of six virtual machines running CentOS 7.1 with a recent lustre-master build to run the Lustre tests. Two clients, two MDS, two OSS. This enables testing of a wide variety of Lustre features.
Line 26:		Line 31:
	* Disable SELINUX		* Disable SELINUX
	** Set SELINUX=disabled in /etc/sysconfig/selinux		** Set SELINUX=disabled in /etc/sysconfig/selinux
			** Note: disable SELinux for general testing, but the sanity-selinux suite requires SELinux to be ''enabled'' and will skip without it.
	* Disable the firewall		* Disable the firewall
	** service firewalld stop && systemctl disable firewalld.service		** service firewalld stop && systemctl disable firewalld.service
Line 36:		Line 42:
	* Add user 'runas' with UID 500 and GID 500 to all the nodes		* Add user 'runas' with UID 500 and GID 500 to all the nodes
	* Create an NFS share that is mounted on all the nodes		* Create an NFS share that is mounted on all the nodes
	** A small number of tests will make use of a shared storage location		** A small number of tests will make use of a shared storage location
	* Configure hostnames and populate /etc/hosts		* Configure hostnames and populate /etc/hosts
	<pre>		<pre>
	127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4		127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
	::1 localhost localhost.localdomain localhost6 localhost6.localdomain6		::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
	192.168.56.201 node01		192.168.56.201 node01
	192.168.56.202 node02		192.168.56.202 node02
	192.168.56.203 node03		192.168.56.203 node03
	192.168.56.204 node04		192.168.56.204 node04
	192.168.56.205 node05		192.168.56.205 node05
	192.168.56.206 node06		192.168.56.206 node06
	</pre>		</pre>

Line 57:		Line 63:
	* Install Parallel IO Simulator (PIOS) - optional, used to simulate shared file and file per process IO		* Install Parallel IO Simulator (PIOS) - optional, used to simulate shared file and file per process IO
	** Available from the EL6 builds of the toolkit - https://build.whamcloud.com/job/toolkit/arch=x86_64,distro=el6/lastSuccessfulBuild/artifact/_topdir/RPMS/x86_64/		** Available from the EL6 builds of the toolkit - https://build.whamcloud.com/job/toolkit/arch=x86_64,distro=el6/lastSuccessfulBuild/artifact/_topdir/RPMS/x86_64/
	* Install ~~POSIX package~~ - optional, used by the ~~posix.sh test~~ suite		* Install <code>pjdfstest</code> and <code>prove</code> - optional, used by the pjdfstest suite for POSIX compliance testing
	** ~~Available from the builds of the toolkit - https:~~/~~/build~~.~~whamcloud~~.~~com/job/toolkit/arch=x86_64,distro=el6/lastSuccessfulBuild/artifact/_topdir/RPMS/x86_64~~/		** The pjdfstest suite also needs <code>mkfs.ext4</code> and loopback mount support, since it compares Lustre results against a local ext4 filesystem. See [[POSIX Compliance Testing]].
			** The older <code>posix.sh</code> suite this HOWTO originally referenced has been removed from the Lustre source tree and replaced by pjdfstest.

	==== Test Configuration ====		==== Test Configuration ====
Line 135:		Line 142:
	</pre>		</pre>

	You will see a lot of output while the tests format the targets, start the filesystem, and mount it on the clients.		You will see a lot of output while the tests format the targets, start the filesystem, and mount it on the clients.

	You will see this output if the test successfully runs:		You will see this output if the test successfully runs:
Line 177:		Line 184:
	Add this machine to our hosts file:		Add this machine to our hosts file:
	<pre>		<pre>
	192.168.56.207 node07		192.168.56.207 node07
	</pre>		</pre>

Line 189:		Line 196:

	<pre>		<pre>
	AGTDEV1=/archive		AGTDEV1=/archive
	agt1_HOST=node07		agt1_HOST=node07
	HSMTOOL_VERBOSE="-v -v -v -v -v -v"		HSMTOOL_VERBOSE="-v -v -v -v -v -v"
Line 201:		Line 208:
	[root@node05 tests]# ./auster -f multinode -rsv -d /opt/results/ sanity-hsm		[root@node05 tests]# ./auster -f multinode -rsv -d /opt/results/ sanity-hsm
	</pre>		</pre>

			==== See Also ====

			* [[Test Descriptions]] — Description and status of every Lustre test suite
			* [[TestingLustreCode]] — Running the Lustre test suites
			* [[Test Groups for Patch Testing]] — Which suites run against each patch
			* [[POSIX Compliance Testing]] — Running the pjdfstest compliance suite

	[[Category: Testing]]		[[Category: Testing]]

Adilger at 17:38, 22 June 2018

2018-06-22T17:38:52Z

Justinmiller at 14:00, 24 March 2016

2016-03-24T14:00:36Z

← Older revision		Revision as of 07:00, 24 March 2016
Line 158:		Line 158:
	==== Test Output ====		==== Test Output ====

	By default, auster will write output to <pre> /tmp/test_logs/<time> </pre>		By default, auster will write output to <pre> /tmp/test_logs/<date>/<time>/ </pre>

	The -d flag directed auster to store the results in the base directory /opt/results/. auster will create <pre> /opt/results/<date>/<time> </pre> directories. Inside the directory corresponding to your test run, you'll find the test output.		The -d flag directed auster to store the results in the base directory /opt/results/. auster will create <pre> /opt/results/<date>/<time>/ </pre> directories. Inside the directory corresponding to your test run, you'll find the test output.

	==== Adding Additional Configuration for sanity-hsm ====		==== Adding Additional Configuration for sanity-hsm ====

KenRawlings: adding categories

2016-03-04T19:34:02Z

adding categories

← Older revision		Revision as of 12:34, 4 March 2016
Line 201:		Line 201:
	[root@node05 tests]# ./auster -f multinode -rsv -d /opt/results/ sanity-hsm		[root@node05 tests]# ./auster -f multinode -rsv -d /opt/results/ sanity-hsm
	</pre>		</pre>

			[[Category: Testing]]

Justinmiller: /* Adding Additional Configuration for sanity-hsm */

2015-09-22T09:41:39Z

Adding Additional Configuration for sanity-hsm

← Older revision		Revision as of 02:41, 22 September 2015
Line 186:		Line 186:
	</pre>		</pre>

	~~Now we need to add some~~ additional environment variables to our multinode.sh configuration file:		Add additional environment variables to our multinode.sh configuration file for sanity-hsm:

	<pre>		<pre>

Justinmiller: /* Adding Additional Configuration for sanity-hsm */

2015-09-22T09:40:40Z

Adding Additional Configuration for sanity-hsm

← Older revision		Revision as of 02:40, 22 September 2015
Line 178:		Line 178:
	<pre>		<pre>
	192.168.56.207 node07		192.168.56.207 node07
			</pre>

			Expand the RCLIENTS definition to include node07:

			<pre>
			RCLIENTS="node05 node06 node07"
	</pre>		</pre>

Justinmiller: /* System Configuration */

2015-09-22T09:39:19Z

System Configuration

← Older revision		Revision as of 02:39, 22 September 2015
Line 8:		Line 8:

	* node01 - MGS and MDS - 192.168.56.201		* node01 - MGS and MDS - 192.168.56.201
	** 1GB MDT - /dev/sdb		** 1GB MGT/MDT - /dev/sdb
	* node02 - MDS - 192.168.56.202		* node02 - MDS - 192.168.56.202
	** 1GB MDT - /dev/sdb		** 1GB MDT - /dev/sdb

Justinmiller at 09:19, 22 September 2015

2015-09-22T09:19:04Z

@@ Line 1: / Line 1: @@
-This HOWTO is intended to demonstrate the basics of configuring and running tests on a multi-node configuration. [[Test Descriptions | There are many tests available in the Lustre Test Suites]], and the principles demonstrated in configuring your system and running the sanity test suite will apply to them all.
+This HOWTO is intended to demonstrate the basics of configuring and running tests on a multi-node configuration. [[Test Descriptions | There are many tests available in the Lustre Test Suites]], and the principles demonstrated in configuring your system and running the sanity test suite will apply to them all. We will also demonstrate adding additional configuration for feature specific test suites.
 While these examples do use virtual machines, they are merely examples and the specifics should be easy to apply to real hardware with the prerequisites setup.
@@ Line 161: / Line 161: @@
 The -d flag directed auster to store the results in the base directory /opt/results/. auster will create <pre> /opt/results/<date>/<time> </pre> directories. Inside the directory corresponding to your test run, you'll find the test output.

Justinmiller at 09:06, 22 September 2015

2015-09-22T09:06:49Z

Show changes

Justinmiller at 17:18, 3 September 2015

2015-09-03T17:18:07Z

← Older revision		Revision as of 10:18, 3 September 2015
Line 4:		Line 4:

	While these examples do use virtual machines, they are merely examples and the specifics should be easy to apply to real hardware with the prerequisites setup.		While these examples do use virtual machines, they are merely examples and the specifics should be easy to apply to real hardware with the prerequisites setup.

	~~This HOWTO uses CentOS 7.1 with Lustre 2.8-RC.~~

	==== System Configuration ====		==== System Configuration ====

	This HOWTO uses a cluster of six virtual machines to run the Lustre tests. Two clients, two MDS, two OSS. This enables testing of a wide variety of Lustre features.		This HOWTO uses a cluster of six virtual machines running CentOS 7.1 with a recent lustre-master to run the Lustre tests. Two clients, two MDS, two OSS. This enables testing of a wide variety of Lustre features.

	* node01 - MGS and MDS - 192.168.56.201		* node01 - MGS and MDS - 192.168.56.201
Line 26:		Line 24:
	==== System Setup ====		==== System Setup ====

	* Lustre ~~installed~~ on ~~all nodes~~, same version		* Install the Lustre clients on two machines, and the Lustre server on the other four, using the same version of Lustre
	** Follow this guide to install Lustre RPMs - [https://wiki.hpdd.intel.com/display/PUB/Walk-thru-+Deploying+Lustre+pre-built+RPMs Walk-thru - Deploying Lustre pre-built RPMs]		** Follow this guide to install Lustre RPMs - [https://wiki.hpdd.intel.com/display/PUB/Walk-thru-+Deploying+Lustre+pre-built+RPMs Walk-thru - Deploying Lustre pre-built RPMs]
	* Disable SELINUX		* Disable SELINUX
Line 32:		Line 30:
	* Disable the firewall		* Disable the firewall
	** service firewalld stop && systemctl disable firewalld.service		** service firewalld stop && systemctl disable firewalld.service
	* Generate passwordless ssh keys for hosts and exchange identities across all nodes, ~~while~~ also ~~accepting~~ the host keys		* Generate passwordless ssh keys for hosts and exchange identities across all nodes, and also accept the host keys
			* Install PDSH and ensure you can execute commands across the cluster
			** https://build.hpdd.intel.com/job/toolkit/arch=x86_64,distro=el7/lastSuccessfulBuild/artifact/_topdir/RPMS/x86_64/
	* Install the epel-release package to enable the EPEL repo		* Install the epel-release package to enable the EPEL repo
	* Install the net-tools package for netstat		* Install the net-tools package for netstat
	* Add user 'runas' with UID 500 and GID 500		* Add user 'runas' with UID 500 and GID 500 to all the nodes
			* Create an NFS share that is mounted on all the nodes
			** A small number of tests will make use of a shared storage location
	* Configure hostnames and populate /etc/hosts		* Configure hostnames and populate /etc/hosts
	<pre>		<pre>
Line 50:		Line 52:
	==== Installing Additional Applications for Testing ====		==== Installing Additional Applications for Testing ====

	* Install PDSH - mandatory
	** https://build.hpdd.intel.com/job/toolkit/arch=x86_64,distro=el7/lastSuccessfulBuild/artifact/_topdir/RPMS/x86_64/
	* Install [https://dbench.samba.org/ dbench] - optional, used to gather baseline metrics of disks used in testing		* Install [https://dbench.samba.org/ dbench] - optional, used to gather baseline metrics of disks used in testing
	** Available from the EPEL repo, or from the Intel HPDD toolkit builds https://build.hpdd.intel.com/job/toolkit/arch=x86_64,distro=el7/lastSuccessfulBuild/artifact/_topdir/RPMS/x86_64/		** Available from the EPEL repo, or from the Intel HPDD toolkit builds https://build.hpdd.intel.com/job/toolkit/arch=x86_64,distro=el7/lastSuccessfulBuild/artifact/_topdir/RPMS/x86_64/
Line 256:		Line 256:
	[root@node05 tests]# ./auster -f multinode -rsv -d /opt/results/ sanity		[root@node05 tests]# ./auster -f multinode -rsv -d /opt/results/ sanity
	</pre>		</pre>

			Running sanity with the -s flag takes about 1h32m in the virtual machine cluster on my computer.

			==== Test Output ====

	The -d flag directed auster to store the results in the base directory /opt/results/. auster will create <pre> /opt/results/<date>/<time> </pre> directories. Inside the directory corresponding to your test run, you'll find the test output.		The -d flag directed auster to store the results in the base directory /opt/results/. auster will create <pre> /opt/results/<date>/<time> </pre> directories. Inside the directory corresponding to your test run, you'll find the test output.

← Older revision		Revision as of 10:38, 22 June 2018
Line 22:		Line 22:

	* Install the Lustre client packages on two machines, and the Lustre server packages on the other four, using the same version of Lustre		* Install the Lustre client packages on two machines, and the Lustre server packages on the other four, using the same version of Lustre
	** Follow this guide to install Lustre RPMs - [https://wiki.~~hpdd.intel~~.com/display/PUB/Walk-thru-+Deploying+Lustre+pre-built+RPMs Walk-thru - Deploying Lustre pre-built RPMs]		** Follow this guide to install Lustre RPMs - [https://wiki.whamcloud.com/display/PUB/Walk-thru-+Deploying+Lustre+pre-built+RPMs Walk-thru - Deploying Lustre pre-built RPMs]
	** Also be sure to install the latest e2fsprogs on the servers - [https://build.~~hpdd.intel~~.com/job/e2fsprogs-master/ e2fsprogs-master builds]		** Also be sure to install the latest e2fsprogs on the servers - [https://build.whamcloud.com/job/e2fsprogs-master/ e2fsprogs-master builds]
	* Disable SELINUX		* Disable SELINUX
	** Set SELINUX=disabled in /etc/sysconfig/selinux		** Set SELINUX=disabled in /etc/sysconfig/selinux
Line 31:		Line 31:
	** The goal is to be able to pdsh using ssh from all machines without requiring any user input		** The goal is to be able to pdsh using ssh from all machines without requiring any user input
	* Install PDSH and ensure you can execute commands across the cluster		* Install PDSH and ensure you can execute commands across the cluster
	** https://build.~~hpdd.intel~~.com/job/toolkit/arch=x86_64,distro=el7/lastSuccessfulBuild/artifact/_topdir/RPMS/x86_64/		** https://build.whamcloud.com/job/toolkit/arch=x86_64,distro=el7/lastSuccessfulBuild/artifact/_topdir/RPMS/x86_64/
	* Install the epel-release package to enable the EPEL repo		* Install the epel-release package to enable the EPEL repo
	* Install the net-tools package for netstat		* Install the net-tools package for netstat
Line 52:		Line 52:

	* Install [https://dbench.samba.org/ dbench] - optional, used to gather baseline metrics of disks used in testing		* Install [https://dbench.samba.org/ dbench] - optional, used to gather baseline metrics of disks used in testing
	** Available from the EPEL repo, or from the ~~Intel HPDD~~ toolkit builds https://build.~~hpdd.intel~~.com/job/toolkit/arch=x86_64,distro=el7/lastSuccessfulBuild/artifact/_topdir/RPMS/x86_64/		** Available from the EPEL repo, or from the Whamcloud toolkit builds https://build.whamcloud.com/job/toolkit/arch=x86_64,distro=el7/lastSuccessfulBuild/artifact/_topdir/RPMS/x86_64/
	* Install [http://www.iozone.org/ iozone] - optional, used to gather baseline metrics of filesystem performance		* Install [http://www.iozone.org/ iozone] - optional, used to gather baseline metrics of filesystem performance
	** Available from ~~Intel HPDD~~ EL7 builds of the toolkit - https://build.~~hpdd.intel~~.com/job/toolkit/arch=x86_64,distro=el7/lastSuccessfulBuild/artifact/_topdir/RPMS/x86_64/		** Available from EL7 builds of the toolkit - https://build.whamcloud.com/job/toolkit/arch=x86_64,distro=el7/lastSuccessfulBuild/artifact/_topdir/RPMS/x86_64/
	* Install Parallel IO Simulator (PIOS) - optional, used to simulate shared file and file per process IO		* Install Parallel IO Simulator (PIOS) - optional, used to simulate shared file and file per process IO
	** Available from the ~~Intel HPDD~~ EL6 builds of the toolkit - https://build.~~hpdd.intel~~.com/job/toolkit/arch=x86_64,distro=el6/lastSuccessfulBuild/artifact/_topdir/RPMS/x86_64/		** Available from the EL6 builds of the toolkit - https://build.whamcloud.com/job/toolkit/arch=x86_64,distro=el6/lastSuccessfulBuild/artifact/_topdir/RPMS/x86_64/
	* Install POSIX package - optional, used by the posix.sh test suite		* Install POSIX package - optional, used by the posix.sh test suite
	** Available from the ~~Intel HPPD~~ builds of the toolkit - https://build.~~hpdd.intel~~.com/job/toolkit/arch=x86_64,distro=el6/lastSuccessfulBuild/artifact/_topdir/RPMS/x86_64/		** Available from the builds of the toolkit - https://build.whamcloud.com/job/toolkit/arch=x86_64,distro=el6/lastSuccessfulBuild/artifact/_topdir/RPMS/x86_64/

	==== Test Configuration ====		==== Test Configuration ====