View source for POSIX Compliance Testing

{| class='wikitable'
|- 
!Note: This page originated on the old Lustre wiki. It was identified as likely having value and was migrated to the new wiki. It is in the process of being reviewed/updated and may currently have content that is out of date.
|}

<small>''(Updated: Sep 2009)''</small>
__TOC__
This page provides instructions for installing and running the POSIX compliance suite of file system tests.

'''''Note:''''' The Lustre™ code is not completely POSIX compliant, so test results may show some errors. If you have questions about the test results, submit a request for assistance through [[Reporting Bugs|Bugzilla]].

== Quick Start ==

Several versions of the POSIX compliance suite available to  [http://downloads.lustre.org/public/tools/benchmarks/posix/ download]. Each version is gcc and architecture specific. You will need to determine which version of gcc you are running locally (''{{{gcc -v}}}'') and then download the appropriate tarball.

If a package isn't available for your particular combination of gcc+architecture, see [[POSIX Compliance Testing#Building and Running a POSIX Compliance Test Suite on Lustre|Building and Running a POSIX Compliance Test Suite on Lustre]].

The following quick start versions are provided:
* [http://downloads.lustre.org/public/tools/benchmarks/posix/one-step-gcc2.96-i686.tgz one-step-gcc2.96-i686.tgz]
* [http://downloads.lustre.org/public/tools/benchmarks/posix//one-step-gcc2.96-ia64.tgz one-step-gcc2.96-ia64.tgz]
* [http://downloads.lustre.org/public/tools/benchmarks/posix/one-step-gcc3.04-i686.tgz one-step-gcc3.04-i686.tgz]
* [http://downloads.lustre.org/public/tools/benchmarks/posix/one-step-gcc3.2-i686.tgz one-step-gcc3.2-i686.tgz]

=== Installing Using the Quick Start Script ===
1. Download the following scripts into ''/usr/src/posix'':
* Test script: ''one-step-gcc<gcc version>-<arch>.tgz''
* Quick start script: ''one-step-setup.sh''
Both are available at [http://downloads.lustre.org/public/tools/benchmarks/posix/ downloads.lustre.org/public/tools/benchmarks/posix/].

2. Run the setup script by entering:
<pre>
cd /usr/src/posix
sh one-step-setup.sh
</pre>

4. Edit the configuration file ''/mnt/lustre/TESTROOT/tetexec.cfg'' to enter appropriate values for your system.

5. Save the TESTROOT for running lustre tests by entering:
<pre>
cd /mnt/lustre
tar zcvf /usr/src/posix/TESTROOT.tgz TESTROOT
</pre> 

'''''Note:''''' The quick start procedure only works with the paths ''/home/tet'' and ''/mnt/lustre''. If you want to change the paths, follow the [[POSIX Compliance Testing#Building from Scratch|Building from Scratch]] instructions below and create a new tarball.

=== Running the Test Suite===
To run the test suite, enter:
<pre>
su - vxs0
. ../profile
tcc -e -a /mnt/lustre/TESTROOT -s scen.exec -p
</pre>

== Building and Running a POSIX Compliance Test Suite on Lustre ==
This section describes how to build and run a POSIX compliance test suite for a compiler and architecture for which we don't provide a quick start package.

=== Building from Scratch ===
This section describes how to build a POSIX compliance suite for testing a Lustre file system:

1. Download all the POSIX files in [http://downloads.lustre.org/public/tools/benchmarks/posix/ downloads.lustre.org/public/tools/benchmarks/posix/]

*''tet_vsxgen_3.02.tgz''
*''lts_vsx-pcts2.0beta2.tgz''
*''install.sh''
*''myscen.bld''
*''myscen.exec''

'''''Note:''''' We now use the latest release of the [http://www.opengroup.org/testing/linux-test/lsb-vsx.html LSB-VSX POSIX test suite] (''lts_vsx-pcts2.0beta2.tgz'') and the generic TET/VSXgen framework (''tet_vsxgen_3.02.tgz''). In this release, the issue of "''getgroups() did not return NGROUPS_MAX''" has been fixed.

2. ''DO NOT'' configure or mount a Lustre filesystem yet.

3. Run the ''{{{install.sh}}}'' script and select ''/home/tet'' for the root directory for the test suite installation.  Say 'y' to installing the users and groups. Accept the defaults for the packages to install.

4. Create a temporary directory for holding the POSIX tests while they are being built:
<pre>
mkdir -p /mnt/lustre/TESTROOT;chown vsx0.vsxg0 !$
</pre>

5. Log in as the test user: 
<pre>
su - vsx0
</pre>

6. To build the test suite, run
<pre>
../setup.sh
</pre>
Most of the default answers are correct, except the root directory from which to run the testsets. For this you should specify ''/mnt/lustre/TESTROOT''. For "Install pseudolanguages?", answer 'n'.

7. When the script asks you "Install scripts into TESTROOT/BIN..?", do not stop the script from running (this doesn't work). Instead, use another terminal to replace existing files with files you downloaded by entering:
<pre>
cp .../myscen.bld  /home/tet/test_sets/scen.bld
cp .../myscen.exec /home/tet/test_sets/scen.exec
</pre>
This will confine the tests that are run to those relevant for file systems, avoiding ''hours'' of running other tests on sockets, math, stdio, libc, shell, etc.

8. Continue with the installation at this point. Answer 'y' to the "Build testsets" question. 

The script proceeds to build and install all the file system tests and then runs them all. Even though the script is running them on a local file system, this is a valuable baseline for comparison with the behavior of Lustre. 

The results are put into ''/home/tet/test_sets/results/0002e/journal''. It is suggested that you rename or symlink this directory to ''/home/tet/test_sets/results/ext3/journal'' (or the name of the local file system that the test was run on).

Running the full test should only take 5 minutes or so. 

9. Answer 'n' to re-running just the failed tests, and you are done.  The results (in a very lengthy table) are in ''/home/tet/test_sets/results/report''.

10. At this point you need to ''save'' the test suite to use for running further tests on a Lustre file system.  Tar up the tests to avoid rebuilding each time by entering:
<pre>
tar cvzf TESTROOT.tgz -C /mnt/lustre TESTROOT
</pre>

You will may also want to remove the installed tests to save space, and more importantly to avoid confusion if you forget to mount your Lustre file system before running the test.

=== Running the POSIX Test Suite against Lustre and checking your results ===
1. As root, set up your Lustre filesystem mounted on ''/mnt/lustre'' (e.g. ''sh llmount.sh'') and untar the POSIX tests back to their home by entering:
<pre>
tar --same-owner -xzpvf /path/to/tarball/TESTROOT.tgz -C /mnt/lustre
</pre>

2. As the vsx0 user, you can re-run the tests as many times as you want. If you are newly su'd or logged in as the vsx0 user, you will need to source the environment with '. profile' so that your path and other environment is set up correctly. Run the tests by entering:
<pre>
. /home/tet/profile
tcc -e -s scen.exec -a /mnt/lustre/TESTROOT -p
</pre>

3. Each new result is put in a new directory under ''/home/tet/test_sets/results'' and is given a directory name similar to ''0004e'' (an increasing number and ending with ''e'' for test ''e''xecution, or ''b'' for ''b''uilding the tests). 
  
To look at a formatted report, enter:
<pre>
vrpt results/0004e/journal | less
</pre>
  
Some tests are "Unsupported", "Untested", or "Not In Use", which does not necessarily indicate a problem. 

To compare two test results, use the command:
<pre>
vrptm results/ext3/journal results/0004e/journal | less
</pre>
  
This is more interesting than looking at the result of a single test since it helps to find test failures that are specific to the file system instead of the Linux VFS or kernel. Up to 6 test results can be compared at once. 

It is often useful to rename the results directory to have more meaningful names (such as ''before_unlink_fix'').

=== Isolating and Debugging Failures ===

When failures occur, you'll need to gather more information about what is happening at runtime. For example, some tests may cause kernel panics depending on your configuration.
* The POSIX compliance suite doesn't have debugging enabled by default, so it is useful to turn on the debugging options of VSX. Two debug options of note reside in the config file ''tetexec.cfg'', under your TESTROOT directory:

:*''VSX_DBUG_FILE=output_file'' - If you're running the test under UML with ''hostfs'' support, you should use a file on the ''hostfs'' as the debug output file. In the case of a crash, the debug output will then be safely written to the debug file. 
::'''''Note:''''' the default value for this option puts the debug log under your test directory in ''/mnt/lustre/TESTROOT'', which may not be useful if you experience a kernel panic and lustre (or your machine) crashes.

:*''VSX_DBUG_FLAGS=xxxxx'' - For detailed information about debug flags, refer to the documentation included with the POSIX suite. The following example will make VSX output ''all'' debug messages:
:<pre>VSX_DBUG_FLAGS=t:d:n:f:F:L:l,2:p:P
</pre>
* VSX is based on the TET framework which provides common libraries for VSX. You can have TET print out verbose debug messages by inserting the option -T when running the tests:
<pre>
tcc -Tall5 -e -s scen.exec -a /mnt/lustre/TESTROOT -p 2>&1 | tee /tmp/POSIX-command-line-output.log
 </pre>
* VSX will print out detailed messages in the report for failed tests. This will include the test strategy, what kind of operations are done by the test suite, and what's going wrong. 

Each subtest (e.g. 'access', 'create') usually contains many single tests. The report will show exactly which single testing fails. In this case, you can find more information directly from the VSX source code. For example, if the 5th single test of subtest ''chmod'' failed, you could look at the source:
<pre>
/home/tet/test_sets/tset/POSIX.os/files/chmod/chmod.c
</pre>
...which contains a single test array:
<pre>
     public  struct tet_testlist tet_testlist[] = {
         test1, 1,
         test2, 2,
         test3, 3,
         test4, 4,
         test5, 5,
         test6, 6,
         test7, 7,
         test8, 8,
         test9, 9,
         test10, 10,
         test11, 11,
         test12, 12,
         test13, 13,
         test14, 14,
         test15, 15,
         test16, 16,
         test17, 17,
         test18, 18,
         test19, 19,
         test20, 20,
         test21, 21,
         test22, 22,
         test23, 23,
        NULL, 0
     };
</pre>

If this single test is causing problems, as in the case of a kernel panic, or if you are trying to isolate a single failure, it may be useful to edit the ''tet_testlist'' array down to the single test in question and then recompile the test suite. You can then create a new tarball of the resulting TESTROOT directory, named appropriately (e.g. ''TESTROOT-chmod-5-only.tgz'') and re-run the POSIX suite using the steps above. 

It may also be helpful to edit the ''scen.exec'' file to only run the test set in question:
<pre>
   all
        "total tests in POSIX.os 1"
        /tset/POSIX.os/files/chmod/T.chmod
</pre>

'''''Note:''''' Rebuilding individual POSIX tests is not straightforward due to the reliance on tcc. One option is to substitute edited source files into the source tree while following the manual installation procedure outlined above and let the existing POSIX install scripts do the work. The installation scripts (specifically ''/home/tet/test_sets/run_testsets.sh'') contain relevant commands for building the test suite -- something akin to ''tcc -p -b -s $HOME/scen.bld $*'' -- but these commands may not work outside the scripts. Please let us know if you manage to get better mileage rebuilding these tests.

[[Category: NeedsReview]]