IOTests

The category of IOTests consists in systematically comparing a number of reference files (typically generated in previous stable versions) against the ones generated by the tested revision. These are very important as it is the only way to make absolutely sure that no border effects have been introduced. The drawback then being that they are overly sensitive and can become time consuming to update as the code evolve. Also, these tests are very inelegant if performed naively by hardcoding the reference content in the python code itself.

This is why MadGraph5 proposes an efficient way of handling these tests by proposing a managing system fully incorporated in the test_manager.py script. Also, a structure is proposed to ease new implementation of such tests by making them inherit from the generic class IOTest.

General information about the IOTests

An IOTest is defined by a FolderName which represents the class of tests it belongs to, a TestName which identifies the test and a list of files that it generates and which will be checked against the trusted reference ones from earlier stable versions. The 'run()' function of the IOTest takes care of the generation of these files and can be overwritten in daughter class to exploit IOTests for any purpose.

The reference trusted files generated with previous stable version of the code are stored in the folder 'tests/input_files/IOTestsComparison' which is synchronized to the tarball 'tests/input_files/IOTestsComparison.tar.bz2' which belongs to the bzr revisioned files. For each test, the path of the reference files in 'IOTestsComparison' is 'FolderName/TestName/FileName(s)'.

Notice that when running the unit test all the short IOTests present in the directory 'unit_tests' will be run as well (they are grouped in one unit_test). When running the acceptance tests, the longer IOTests present in the directory 'acceptance_tests' are run, again grouped together in one acceptance_test.

What to do if they fail? Managing the existing IOTests

To access the functionalities for the management of the IOTests, you must run the './tests/test_manager.py' with the '-i' option followed by either '-U' or '-R':

./tests/test_manager.py -i -R <optional_test_specifier>

=> Run all the IOTests (in all test directories). This will monitor the possible differences with respect to the reference files and printout a summary without applying any modification.

./tests/test_manager.py -i -U <optional_test_specifier>

=> Same as with '-R' except that whenever a difference is found, it is printed out and you are asked wether you want to update the reference file with this modification or not.

=> You can add '-f' after '-i' in the command above to bypass the question and automatically update everything. You will get a list of the modification at the end and have one last chance of ignoring them. Please use this with care, as blindly updating the reference files ruins the purpose of these tests.

If <optional_test_specifier> is absent, the test_manager will span all IOTest. You can however specify it to target only certain tests or files. The syntax of the specifier is:

<optional_test_specifier> == FolderNames/TestNames/FileNames

The 'FolderNames' and 'TestNames' entry are simply all the FolderNames and TestNames you want to consider separated by a '&'. The FileNames entry can also be specified like this by explicitly writing all the files you want to consider or you can also use regular expressions for this by putting it in between squared brackets. For any of the three entries above, you can use the keyword 'ALL' to include everything (which means that "ALL/ALL/ALL" is effectively the same as not having a test specifier). Notice that the file name is rather the relative path with respect to the what is defined as the root path of the IOTest. In all IOTests implemented so far, this root path is under the P0_<proc_name> directory in 'SubProcesses'. Here are some examples of test specifier:

<optional_test_specifier> == short_ML_SMQCD_default/ALL/[.+\.f]

=> Runs the two IOTest defined with the FolderName "short_ML_SMQCD_default" and only consider the comparison for files matching the regular expression "[.+\.f]" which means any file with the extension '.f'.

<optional_test_specifier> == ALL/gg_ttx/[../../Source/MODEL/.+\.inc]

=> Runs the two IOTest defined with the name "gg_ttx" (placed in the optimized and default folder names) and only consider the comparison for files in the MODEL directory and matching the regular expression "[.+\.inc]" which means any file with the extension '.inc'.

<optional_test_specifier> == short_ML_SMQCD_optimized&short_ML_SMQCD_default/ALL/ALL

=> Consider all the files referenced in all the tests placed in the folder named short_ML_SMQCD_optimized and short_ML_SMQCD_default.

Notice that if a file is written out based on the content of unordered dictionaries, then this comparison is bound to fail and the file should not be referenced. You can modify what are the referenced files in the the IOTest class in 'tests/IOTests' or at the time of the IOTest creation in the different test classes inheriting from IOTestManager (See below for more information).

Unit tests

Parallel tests

Acceptance tests