IOTests
The category of IOTests consists in systematically comparing a number of reference files (typically generated within previous stable versions) against the ones generated by the tested revision. These are very important as it is the only way to make absolutely sure that no border effects have been introduced. The drawback then being that they are overly sensitive and can become time consuming to update as the code evolve. Also, these tests are very inelegant if performed naively by hardcoding the reference content in the python code itself.
This is why MadGraph5 proposes an efficient way of handling these tests by proposing a management system fully incorporated in the test_manager.py script. A structure is proposed to ease new implementation of such tests by making them inherit from the generic class IOTest.
General information about the IOTests
An IOTest is defined by a FolderName which represents the class of tests it belongs to, a TestName which identifies the test and a list of files that it generates and which will be checked against the trusted reference ones from earlier stable versions.
The reference trusted files generated with previous stable version of the code are stored in the folder 'tests/input_files/IOTestsComparison'. For each test, the path of the reference files in 'IOTestsComparison' is 'FolderName/TestName/FileName(s)'.
What to do if they fail? Managing the existing IOTests
The 'tests/test_manager.py' script shall be used for managing the IOTest. Find more information on this by typing:
./tests/test_manager.py help
To manage the IOTests, you must run the './tests/test_manager.py' followed by either '-U' or '-R'.
./tests/test_manager.py -R <optional_test_specifier>
=> Run all the IOTests (in all test directories). This will monitor the possible differences with respect to the reference files and printout a summary without applying any modification.
./tests/test_manager.py -U <optional_test_specifier>
=> Same as with '-R' except that whenever a difference is found, it is printed out and you are asked wether you want to update the reference file with this modification or not.
=> You can add '-F' after the command above to bypass the question and automatically update everything. You will get a list of the modification at the end and have one last chance of ignoring them. You can also use the less inclusive '-f' which will bypass the review of any filename which has been used already. Please use this with care, as blindly updating the reference files ruins the purpose of these tests.
If <optional_test_specifier> is absent, the test_manager will span all IOTests. You can however specify it to target only certain tests or files. The syntax of the specifier is:
<optional_test_specifier> == FolderNames/TestNames/FileNames
The 'FolderNames' and 'TestNames' entry are simply all the FolderNames and TestNames you want to consider separated by a '&'. The FileNames entry can also be specified like this by explicitly writing all the files you want to consider or you can also use regular expressions for this by putting it in between squared brackets. For any of the three entries above, you can use the keyword 'ALL' to include everything (which means that "ALL/ALL/ALL" is effectively the same as not having a test specifier). Notice that the file name is rather the relative path with respect to the what is defined as the root path of the IOTest. In all IOTests implemented so far, this root path is under the P0_<proc_name> directory in 'SubProcesses'. Here are some examples of test specifier:
<optional_test_specifier> == short_ML_SMQCD_default/ALL/[.+\.f]
=> Runs the two IOTest defined with the FolderName "short_ML_SMQCD_default" and only consider the comparison for files matching the regular expression "[.+\.f]" which means any file with the extension '.f'.
<optional_test_specifier> == ALL/gg_ttx/[../../Source/MODEL/.+\.inc]
=> Runs the two IOTest defined with the name "gg_ttx" (placed in the optimized and default folder names) and only consider the comparison for files in the MODEL directory and matching the regular expression "[.+\.inc]" which means any file with the extension '.inc'.
<optional_test_specifier> == short_ML_SMQCD_optimized&short_ML_SMQCD_default/ALL/ALL
=> Consider all the files referenced in all the tests placed in the folder named short_ML_SMQCD_optimized and short_ML_SMQCD_default.
Finally, notice that if a file is written out based on the content of unordered dictionaries, then this comparison is bound to fail and the file should not be referenced.
Creating new IOTests
The simplest way to learn how to create a new IOTest or cast an existing one into an IOTest, is to look at the examples provided in the test module tests/unit_tests/core/test_IOTest_examples.py.
Here is the recipe:
=> The class in which the test is coded must inherit from IOTests.IOTestManager (import IOTests with 'import tests.IOTests as IOTests').
=> The name of the test function *must* start with 'testIO'.
=> You must place a the following decorator to the function you created:
@IOTests.createIOTest(groupName='<optional_group_name>',testName='<optional_test_name>')
which will carry out all the IOTest registering duties for you behind the scene. If the group and test names are not given, they will be inferred from the class and function name respectively.
=> The body of the IOTest function must contain the code generating the files to be tested. You should write these files under the directory which path 'self.IOpath' which is a temporary folder automatically generated and cleaned up for you by the IOTest manager and made readily available at the time of running the IOTest function.
=> The documentation of the IOTest function is semantic! It is used to defined what are the target files that the test must generate and that must be referenced and compared. The syntax is:
""" target: aFile.txt """
Notice that you should only specify here the path relative to the self.IOPath directory. Several paths can be specified on several documentation lines. Also you can specify directories as well as regular expressions that you should embed in squared brackets '[<my_reg_exp]'. For convenience you can specify a path starting with an hyphen ('-') to indicated that you want to veto it if it were to be selected by another regular expression. Here is an example of a more complicated file specifier:
""" target: testScratch/[Folder(A|C)\/.+\.(f|inc)]
target: FileOut.txt target: -testScratch/[Folder(A|C)\/File(X|Y).f]
"""
To conclude, here is an example of a minimal, yet fully functional IOTest:
class IOTest_SimpleExamples(IOTests.IOTestManager):
@IOTests.createIOTest()
""" target: aFile.txt """ contentToCheck = "This is the content to check" open(pjoin(self.IOpath,'aFile.txt'),'w').write(contentToCheck)
