TestingLustreCode - Revision history

Adilger: remove old wiki comment

2024-06-03T17:14:08Z

remove old wiki comment

← Older revision		Revision as of 10:14, 3 June 2024
Line 1:		Line 1:
	~~<small>''(Copied from old wiki June 2015 - last update Dec 2009)''</small>~~
	__TOC__		__TOC__
	We recommend a "test early, test often" approach to testing.		We recommend a "test early, test often" approach to testing.

KenRawlings: adding categories

2016-03-04T19:33:29Z

adding categories

← Older revision		Revision as of 12:33, 4 March 2016
Line 131:		Line 131:
	* OBD_FAIL_TIMEOUT()		* OBD_FAIL_TIMEOUT()
	Or you can make runtime changes using ''lctl set_param fail_loc=...''.		Or you can make runtime changes using ''lctl set_param fail_loc=...''.

			[[Category: Testing]]

Adilger: /* Creating a Test Case */ link to more comprehensive "Test Descriptions" page

2015-08-20T21:20:51Z

Creating a Test Case: link to more comprehensive "Test Descriptions" page

← Older revision		Revision as of 14:20, 20 August 2015
Line 25:		Line 25:
	The first step in fixing a bug is to create or find a test case that causes the bug to be reproduced, then fix the bug, and finally verify that the code containing the bug passes the test.		The first step in fixing a bug is to create or find a test case that causes the bug to be reproduced, then fix the bug, and finally verify that the code containing the bug passes the test.

	Often if a defect is found in Lustre, it is because a test script in the Lustre acceptance test suite is not testing the failing code. Before starting work on a bug, first check if a test script reproduces the bug. If not, you will need to create a new test and add it to the test suite as a ~~scripted sub-test~~ of ~~one of these~~ tests:		Often if a defect is found in Lustre, it is because a test script in the Lustre acceptance test suite is not testing the failing code. Before starting work on a bug, first check if a test script reproduces the bug. If not, you will need to create a new test and add a sub-test to the test suite (normally) in one of the existing test scripts. See [[Test Descriptions]] for a full list of tests. The list below contains only the most commonly run tests.

	{\| class="wikitable"		{\| class="wikitable"
Line 37:		Line 37:
	\|-		\|-
	\| '''lnet-selftest.sh''' \|\| Test basic LNet functionality and the LNet Self Test tools.		\| '''lnet-selftest.sh''' \|\| Test basic LNet functionality and the LNet Self Test tools.
	\|-
	~~\| '''lustre-rsync-test.sh''' \|\| Test the lustre_rsync tool and ChangeLogs.~~
	\|-
	~~\| '''mmp.sh''' \|\| Test ldiskfs multi-mount protection functionality.~~
	\|-
	~~\| '''ost-pools.sh''' \|\| Tests OST pools (named groupings of OSTs).~~
	\|-		\|-
	\| '''recovery-small.sh''' \|\| Tests to verify RPC replay after communications failure (message loss).		\| '''recovery-small.sh''' \|\| Tests to verify RPC replay after communications failure (message loss).
Line 51:		Line 45:
	\|-		\|-
	\| '''replay-ost-single.sh''' \|\| Tests to verify recovery after OSS failure.		\| '''replay-ost-single.sh''' \|\| Tests to verify recovery after OSS failure.
	\|-
	~~\| '''sanity-gss.sh''' \|\| Test GSSAPI used by Kerberos.~~
	\|-
	~~\| '''sanity-krb5.sh''' \|\| Test Kerberos authentication and encryption.~~
	\|-		\|-
	\| '''sanity-hsm.sh''' \|\| Test Hierarchical Storage Management and ChangeLog functionality.		\| '''sanity-hsm.sh''' \|\| Test Hierarchical Storage Management and ChangeLog functionality.
Line 63:		Line 53:
	\|-		\|-
	\| '''sanity-quota.sh''' \|\| Test quota accounting and enforcement.		\| '''sanity-quota.sh''' \|\| Test quota accounting and enforcement.
	\|-
	~~\| '''sanity-sec.sh''' \|\| Tests security and authentication (root squash, UID/GID mapping, etc).~~
	\|-
	~~\| '''insanity.sh''' \|\| Tests multiple concurrent failure conditions (clients and servers).~~
	\|-
	\|}		\|}

Adilger: Fix URL

2015-08-20T20:58:41Z

Fix URL

← Older revision		Revision as of 13:58, 20 August 2015
Line 20:		Line 20:
	The ''acceptance-small'' test suite is run using the script ''acceptance-small.sh'', which is located in the lustre/tests directory of a compiled Lustre tree. For more details, see [[Acceptance Small (acc-sm) Testing on Lustre]].		The ''acceptance-small'' test suite is run using the script ''acceptance-small.sh'', which is located in the lustre/tests directory of a compiled Lustre tree. For more details, see [[Acceptance Small (acc-sm) Testing on Lustre]].

	'''''Note:''''' When [[Submitting ~~Patches~~]], in [https://review.whamcloud.com/ Gerrit] it will automatically launch a series of tests for that patch with different configurations (ldiskfs vs. ZFS, one vs. multiple MDS nodes) and will mark the patch with '''Verified: +1''' to indicate that it passed.		'''''Note:''''' When [[Submitting Changes]], in [https://review.whamcloud.com/ Gerrit] it will automatically launch a series of tests for that patch with different configurations (ldiskfs vs. ZFS, one vs. multiple MDS nodes) and will mark the patch with '''Verified: +1''' to indicate that it passed.

	== Creating a Test Case ==		== Creating a Test Case ==

Adilger: /* Using the Lustre Testing Framework */ replace old bugzilla patch test process with new Gerrit + autotest process

2015-08-20T20:56:31Z

Using the Lustre Testing Framework: replace old bugzilla patch test process with new Gerrit + autotest process

← Older revision		Revision as of 13:56, 20 August 2015
Line 20:		Line 20:
	The ''acceptance-small'' test suite is run using the script ''acceptance-small.sh'', which is located in the lustre/tests directory of a compiled Lustre tree. For more details, see [[Acceptance Small (acc-sm) Testing on Lustre]].		The ''acceptance-small'' test suite is run using the script ''acceptance-small.sh'', which is located in the lustre/tests directory of a compiled Lustre tree. For more details, see [[Acceptance Small (acc-sm) Testing on Lustre]].

	'''''Note:''''' When [[Submitting Patches]], in [https://~~bugzilla~~.~~lustre~~.~~org~~/ ~~Bugzilla~~] ~~set~~ the ''~~acc-sm passed~~'' ~~flag on the attachment for each individual branch that was tested~~ to indicate that it passed.		'''''Note:''''' When [[Submitting Patches]], in [https://review.whamcloud.com/ Gerrit] it will automatically launch a series of tests for that patch with different configurations (ldiskfs vs. ZFS, one vs. multiple MDS nodes) and will mark the patch with '''Verified: +1''' to indicate that it passed.

	== Creating a Test Case ==		== Creating a Test Case ==

Adilger: /* Creating a Test Case */ add test script descriptions to cover all tests run during regular Intel Gerrit patch revie

2015-08-20T20:53:17Z

Creating a Test Case: add test script descriptions to cover all tests run during regular Intel Gerrit patch revie

← Older revision		Revision as of 13:53, 20 August 2015
Line 28:		Line 28:

	{\| class="wikitable"		{\| class="wikitable"
	\| '''~~sanity.sh~~''' \|\| ~~Tests to verify operation under normal operating conditions~~		\| '''runtests''' \|\| Very basic test of mounting, copying files, remounting, verifying files are still correct, remounting, and unlinking files.
	\|-		\|-
	\| '''~~sanityN~~.sh''' \|\| Tests to verify ~~operations from two clients~~ under normal operating conditions		\| '''sanity.sh''' \|\| Tests to verify basic functionality under normal operating conditions.
	\|-		\|-
	\| '''~~liblustre/tests/sanity~~''' \|\| ~~Runs a test linked~~ to ~~a liblustre client library~~		\| '''sanityn.sh''' \|\| Tests to verify operations from two clients under normal operating conditions.
	\|-		\|-
	\| '''~~recovery~~-~~small~~.sh''' \|\| Tests ~~to verify RPC replay after communications failure~~ (~~message loss~~)		\| '''conf-sanity.sh''' \|\| Tests configuration issues (formatting, mounting, different tunables).
	\|-		\|-
	\| '''~~replay~~-~~single~~.sh''' \|\| ~~Tests to verify recovery after MDS failure~~		\| '''lnet-selftest.sh''' \|\| Test basic LNet functionality and the LNet Self Test tools.
	\|-		\|-
	\| '''~~replay~~-~~dual~~.sh''' \|\| ~~Tests to verify recovery from two clients after server failure~~		\| '''lustre-rsync-test.sh''' \|\| Test the lustre_rsync tool and ChangeLogs.
	\|-		\|-
	\| '''~~replay-ost-single~~.sh''' \|\| ~~Tests to verify recovery after OST failure~~		\| '''mmp.sh''' \|\| Test ldiskfs multi-mount protection functionality.
	\|-		\|-
	\| '''~~lfscktest~~.sh''' \|\| Tests ~~e2fsck and lfsck to detect and fix filesystem corruption~~		\| '''ost-pools.sh''' \|\| Tests OST pools (named groupings of OSTs).
	\|-		\|-
	\| '''insanity.sh''' \|\| Tests multiple concurrent failure conditions		\| '''recovery-small.sh''' \|\| Tests to verify RPC replay after communications failure (message loss).
			\|-
			\| '''replay-single.sh''' \|\| Tests to verify recovery after MDS failure.
			\|-
			\| '''replay-dual.sh''' \|\| Tests to verify recovery from two clients after server failure.
			\|-
			\| '''replay-ost-single.sh''' \|\| Tests to verify recovery after OSS failure.
			\|-
			\| '''sanity-gss.sh''' \|\| Test GSSAPI used by Kerberos.
			\|-
			\| '''sanity-krb5.sh''' \|\| Test Kerberos authentication and encryption.
			\|-
			\| '''sanity-hsm.sh''' \|\| Test Hierarchical Storage Management and ChangeLog functionality.
			\|-
			\| '''sanity-lfsck.sh''' \|\| Tests LFSCK to detect and fix Lustre-level inter-server consistency issues.
			\|-
			\| '''sanity-scrub.sh''' \|\| Tests LFSCK to detect and fix OSD-level filesystem consistency/corruption issues.
			\|-
			\| '''sanity-quota.sh''' \|\| Test quota accounting and enforcement.
			\|-
			\| '''sanity-sec.sh''' \|\| Tests security and authentication (root squash, UID/GID mapping, etc).
			\|-
			\| '''insanity.sh''' \|\| Tests multiple concurrent failure conditions (clients and servers).
	\|-		\|-
	\|}		\|}

Adilger: /* Test Framework Options */ improve description of non-default config file usage

2015-08-20T20:34:28Z

Test Framework Options: improve description of non-default config file usage

← Older revision		Revision as of 13:34, 20 August 2015
Line 73:		Line 73:
	== Test Framework Options ==		== Test Framework Options ==
	The examples below show how to run a full test or sub-tests from the ''acceptance-small'' suite.		The examples below show how to run a full test or sub-tests from the ''acceptance-small'' suite.
	* Run all tests including the standard tests (''sanity*~~'', ''liblustre~~'') with the default (''local.sh'') setup.		* Run all tests including the standard tests (''sanity*'') with the default (''local.sh'') setup.

	<pre>		<pre>
Line 86:		Line 86:
	</pre>		</pre>

	* Run ''acceptance-small'' with a different configuration.		* Run ''acceptance-small'' with a different configuration. Looks for configuration file '''myth.sh''' under '''lustre/tests/cfg'''.

	<pre>		<pre>

John S: Created page with "''(Copied from old wiki June 2015 - last update Dec 2009)'' TOC We recommend a "test early, test often" approach to testing. * If you are developing a new..."

2015-06-25T18:30:48Z

Created page with "<small>''(Copied from old wiki June 2015 - last update Dec 2009)''</small> __TOC__ We recommend a "test early, test often" approach to testing. * If you are developing a new..."

New page

<small>''(Copied from old wiki June 2015 - last update Dec 2009)''</small>
__TOC__
We recommend a "test early, test often" approach to testing.
* If you are developing a new feature for Lustre™, designing tests to exercise the new feature early in the development process will allow you to test your code as you develop it.

* If you are fixing a bug in Lustre, creating a regression test up front will ensure that you can reproduce the reported problem and then verify that it has been fixed. And it will save you the effort of testing the fix manually and then creating a separate regression test later to submit with your bug fix.

We provide several tools to help with testing Lustre code:
* ''Acceptance test suite''. This Lustre testing framework from which a suite of acceptance tests called ''acceptance-small'' can be run is described in the next section [[Testing Lustre Code#Using the Lustre Testing Framework|Using the Lustre Testing Framework]].
* The ''simul parallel file system test tool'' from LLNL exercises file system operations simultaneously from many nodes and processes. For more information, see [[Simul Parallel File System Test Tool]].
* The ''Load Generator (loadgen) test program'' is designed to simulate large numbers of Lustre clients connecting and writing to an OST. For information about using loadgen, see [http://wiki.lustre.org/manual/LustreManual20_HTML/SystemConfigurationUtilities_HTML.html#50438219_pgfId-1294862 Section 36.19.3: ''Testing / Debugging Utilities''] in the [http://wiki.lustre.org/manual/LustreManual20_HTML/index.html ''Lustre Operations Manual'']
* The ''POSIX compliance test suite'' for testing the Lustre file system is described in [[POSIX Compliance Testing]].

To find out more about testing for upcoming Lustre releases, see [[Lustre Test Plans]].

== Using the Lustre Testing Framework ==

Before you submit code, it must pass the ''acceptance-small'' acceptance test suite. We recommend you run the test suite often so that you can find out as soon as possible if your code changes result in a regression.

The ''acceptance-small'' test suite is run using the script ''acceptance-small.sh'', which is located in the lustre/tests directory of a compiled Lustre tree. For more details, see [[Acceptance Small (acc-sm) Testing on Lustre]].

'''''Note:''''' When [[Submitting Patches]], in [https://bugzilla.lustre.org/ Bugzilla] set the ''acc-sm passed'' flag on the attachment for each individual branch that was tested to indicate that it passed.

== Creating a Test Case ==
The first step in fixing a bug is to create or find a test case that causes the bug to be reproduced, then fix the bug, and finally verify that the code containing the bug passes the test.

Often if a defect is found in Lustre, it is because a test script in the Lustre acceptance test suite is not testing the failing code. Before starting work on a bug, first check if a test script reproduces the bug. If not, you will need to create a new test and add it to the test suite as a scripted sub-test of one of these tests:

{| class="wikitable"
| '''sanity.sh''' || Tests to verify operation under normal operating conditions
|-
| '''sanityN.sh''' || Tests to verify operations from two clients under normal operating conditions
|-
| '''liblustre/tests/sanity''' || Runs a test linked to a liblustre client library
|-
| '''recovery-small.sh''' || Tests to verify RPC replay after communications failure (message loss)
|-
| '''replay-single.sh''' || Tests to verify recovery after MDS failure
|-
| '''replay-dual.sh''' || Tests to verify recovery from two clients after server failure
|-
| '''replay-ost-single.sh''' || Tests to verify recovery after OST failure
|-
| '''lfscktest.sh''' || Tests e2fsck and lfsck to detect and fix filesystem corruption
|-
| '''insanity.sh''' || Tests multiple concurrent failure conditions
|-
|}

== Bypassing Failures ==
If one or more tests in ''acceptance-small'' are regularly failing due to an issue not related to the bug you are fixing, you may want to bypass these tests so that you can test your bug fix. Complete these steps:

1. Check to see if a bug has been logged for the failure and file a new bug if one has not yet been opened.

2. Set environment variables to skip these specific tests until you or someone else fixes them. For example, to skip ''sanity.sh'' subtest 36g and 65, ''replay-single.sh'' subtest 42, and all of ''insanity.sh'', set in your environment:

<pre><nowiki>
export SANITY_EXCEPT="36g 65"
export REPLAY_SINGLE_EXCEPT=42
export INSANITY=no
</nowiki></pre>

:You can also skip these tests using a command. For example, when running acceptance-small, enter:

<pre><nowiki>
SANITY_EXCEPT="36g 65" REPLAY_SINGLE_EXCEPT=42 INSANITY=no ./acceptance-small.sh
</nowiki></pre>

The test framework is very flexible and provides an easy "hands-off" way of running tests while you are doing other things like coding. By running the entire test suite regularly, you will soon know whether your code has introduced a new bug or revived an old one.

'''''Note:''''' Questions or problems with the test framework should be emailed to the [mailto:lustre-discuss@lists.lustre.org lustre-discuss] mailing list so that all Lustre users can benefit from the solution.

== Test Framework Options ==
The examples below show how to run a full test or sub-tests from the ''acceptance-small'' suite.
* Run all tests including the standard tests (''sanity*'', ''liblustre'') with the default (''local.sh'') setup.

<pre>
$ cd lustre/tests
$ sh acceptance-small.sh
</pre>

* Run only the ''recovery-small.sh'', ''replay-single.sh'', and ''conf-sanity.sh'' tests.

<pre>
$ ACC_SM_ONLY="recovery-small replay-single conf-sanity" sh acceptance-small.sh
</pre>

* Run ''acceptance-small'' with a different configuration.

<pre>
$ NAME="myth" sh acceptance-small.sh
</pre>

* Run only tests 1, 3, 4, 6, 9 in ''sanity.sh'' with a custom configuration ''example1.sh''.

<pre>
$ ONLY="1 3 4 6 9" NAME=example1 sh sanity.sh
</pre>

* Skip tests 1 ... 30 and run remaining tests in ''sanity.sh''.

<pre>
$ EXCEPT="`seq 1 30`" sh sanity.sh
</pre>

* Clean up after an ''example1.sh'' test failure (normally the system is left mounted for debugging after a failure).

<pre>
$ NAME=example1 sh llmountcleanup.sh
</pre>

* Clean up ''replay-single.sh'' after a test failure (normally the system is left mounted for debugging after a failure).

<pre>
$ ONLY=cleanup sh replay-single.sh
</pre>

'''''Note:''''' The ''acceptance-small'' suite includes two configuration files: ''lustre/tests/cfg/local.sh'' is used as the default configuration file and ''lustre/tests/cfg/ncli.sh'' is used for environments with multiple clients.

== Adding New Tests ==
You can easily add a test to one of the above scripts. Failures can be injected into the Lustre kernel code and monitored using:
* OBD_FAIL_CHECK()
* OBD_FAIL_RACE()
* OBD_FAIL_TIMEOUT()
Or you can make runtime changes using ''lctl set_param fail_loc=...''.