Changes between Version 6 and Version 7 of DevelopmentPage/CodeTesting


Ignore:
Timestamp:
Oct 20, 2014, 9:00:32 PM (10 years ago)
Author:
Valentin Hirschi
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • DevelopmentPage/CodeTesting

    v6 v7  
    33 The category of IOTests consists in systematically comparing a number of reference files (typically generated within previous stable versions) against the ones generated by the tested revision. These are very important as it is the only way to make absolutely sure that no border effects have been introduced. The drawback then being that they are overly sensitive and can become time consuming to update as the code evolve. Also, these tests are very inelegant if performed naively by hardcoding the reference content in the python code itself.
    44
    5 This is why MadGraph5 proposes an efficient way of handling these tests by proposing a managing system fully incorporated in the test_manager.py script. A structure is proposed to ease new implementation of such tests by making them inherit from the generic class IOTest.
     5This is why MadGraph5 proposes an efficient way of handling these tests by proposing a management system fully incorporated in the test_manager.py script. A structure is proposed to ease new implementation of such tests by making them inherit from the generic class IOTest.
    66== General information about the IOTests ==
    77
    8 An IOTest is defined by a FolderName which represents the class of tests it belongs to, a TestName which identifies the test and a list of files that it generates and which will be checked against the trusted reference ones from earlier stable versions. The 'run()' function of the IOTest takes care of the generation of these files and can be overwritten in daughter class to exploit IOTests for any purpose.
     8An IOTest is defined by a FolderName which represents the class of tests it belongs to, a TestName which identifies the test and a list of files that it generates and which will be checked against the trusted reference ones from earlier stable versions.
    99
    10 The reference trusted files generated with previous stable version of the code are stored in the folder 'tests/input_files/IOTestsComparison' which is synchronized to the tarball 'tests/input_files/IOTestsComparison.tar.bz2' which belongs to the bzr revisioned files. For each test, the path of the reference files in 'IOTestsComparison' is 'FolderName/TestName/FileName(s)'.
    11 
    12 Notice that when running the unit test all the short IOTests present in the directory 'unit_tests' will be run as well (they are grouped in one unit_test).
    13 When running the acceptance tests, the longer IOTests present in the directory 'acceptance_tests' are run, again grouped together in one acceptance_test.
     10The reference trusted files generated with previous stable version of the code are stored in the folder 'tests/input_files/IOTestsComparison'. For each test, the path of the reference files in 'IOTestsComparison' is 'FolderName/TestName/FileName(s)'.
    1411
    1512== What to do if they fail? Managing the existing IOTests ==
    … …  
    1916'''./tests/test_manager.py help'''
    2017
    21 To manage the IOTests, you must run the './tests/test_manager.py' with the '-i' option followed by either '-U' or '-R'.
     18To manage the IOTests, you must run the './tests/test_manager.py' followed by either '-U' or '-R'.
    2219
    23 '''./tests/test_manager.py -i -R <optional_test_specifier>'''
     20'''./tests/test_manager.py -R <optional_test_specifier>'''
    2421    => Run all the IOTests (in all test directories). This will monitor the possible differences with respect to the reference files and printout a summary without applying any modification.
    2522
    26 '''./tests/test_manager.py -i -U <optional_test_specifier>'''
     23'''./tests/test_manager.py -U <optional_test_specifier>'''
    2724    => Same as with '-R' except that whenever a difference is found, it is printed out and you are asked wether you want to update the reference file with this modification or not.
    2825   
    29     => You can add '-f' after '-i' in the command above to bypass the question and automatically update everything. You will get a list of the modification at the end and have one last chance of ignoring them.  Please use this with care, as blindly updating the reference files ruins the purpose of these tests.
     26    => You can add '-F' after the command above to bypass the question and automatically update everything. You will get a list of the modification at the end and have one last chance of ignoring them. You can also use the less inclusive '-f' which will bypass the review of any filename which has been used already.  Please use this with care, as blindly updating the reference files ruins the purpose of these tests.
    3027
    3128If <optional_test_specifier> is absent, the test_manager will span all IOTests. You can however specify it to target only certain tests or files. The syntax of the specifier is:
    … …  
    4441  => Consider all the files referenced in all the tests placed in the folder named short_ML_SMQCD_optimized and short_ML_SMQCD_default.
    4542
    46 If you edit the content of the folder 'tests/input_files/IOTestsComparison' by yourself (discouraged, use the '-U' functionality instead as much as possible) and want to synchronized this folder with the revisioned 'tests/input_files/IOTestsComparison.tar.bz2' tarball, you can simply use:
    47 
    48 '''./tests/test_manager.py -s '''
    49 
    50 All changes you bring to the tarball via the test_manager are registered in the 'tests/input_files/IOTestsRefModifs.log' so that one can see what files were modified compared to earlier revisions of the code.
    51 
    52 Finally, notice that if a file is written out based on the content of unordered dictionaries, then this comparison is bound to fail and the file should not be referenced. You can modify what are the referenced files in the the IOTest class in 'tests/IOTests' or at the time of the IOTest creation in the different test classes inheriting from IOTestManager (See below for more information).
     43Finally, notice that if a file is written out based on the content of unordered dictionaries, then this comparison is bound to fail and the file should not be referenced.
    5344
    5445== Creating new IOTests ==
    5546
    56 In order to do so, you simply have to create a new class in the 'unit_tests' or 'parallel_tests' directory which inherits from 'IOTests.IOTestManager' only. Then you should use the 'setUp' function to add your IOTest with the function 'self.addIOTest(folderName,testName,IOTest)' where folderName and testName are two strings identifying the test and specifying where the reference files are placed in tests/input_files/IOTestsComparison. The last argument IOTest must be an instance of the class IOTests.IOTest. This IOTest class is designed for the creation of a loop process and its output whose files are used for comparison. Of course, you are free to define your own IOTest, inheriting from IOTests.IOTest, and overloading its parameters and the function run() which specifies how the files are generated. Notice however that its attribute 'testedFiles' must be specified. It is a list of the path (relative to the path returned by the run() function) of files to be used as a reference. Notice that for each element of this list you can use regular expressions. On top of this you can prepend '-' to an explicit file name to veto it from the list of files considered (could be done with regexps, but it is simpler like this). Here is an example of how one can define the 'testedFiles'
     47The simplest way to learn how to create a new IOTest or cast an existing one into an IOTest, is to look at the examples provided in the test module tests/unit_tests/core/test_IOTest_examples.py.
    5748
    58 '''testedFiles = ['../../Source/MODEL/[.+\.(f|inc)]','-../../Source/MODEL/lha_read.f','-../../Source/MODEL/param_read.inc']'''
    59   => Consider all files with the extension .f or .inc in the folder '../../Source/MODEL' relative to the path output by the run() function, but vetoing the files 'param_read.inc' and 'lha_read.f'. Notice that regular expressions cannot be used in conjunction with the '-' prepend syntax.
     49Here is the recipe:
    6050
    61 You can find a complete example of the implementation of IOTests in the class 'IOExportMadLoopUnitTest' of the module 'tests/unit_tests/loop/test_loop_exporters.py'.
     51=> The class in which the test is coded must inherit from IOTests.IOTestManager (import IOTests with 'import tests.IOTests as IOTests').
     52
     53=> The name of the test function *must* start with 'testIO'.
     54
     55=> You must place a the following decorator to the function you created:
     56  @IOTests.createIOTest(groupName='<optional_group_name>',testName='<optional_test_name>')
     57which will carry out all the IOTest registering duties for you behind the scene. If the group and test names are not given, they will be inferred from the class and function name respectively.
     58
     59=> The body of the IOTest function must contain the code generating the files to be tested. You should write these files under the directory which path 'self.IOpath' which is a temporary folder automatically generated and cleaned up for you by the IOTest manager and made readily available at the time of running the IOTest function.
     60
     61=> The documentation of the IOTest function is semantic! It is used to defined what are the target files that the test must generate and that must be referenced and compared. The syntax is:
     62  """ target: aFile.txt
     63  """
     64Notice that you should only specify here the path relative to the self.IOPath directory. Several paths can be specified on several documentation lines. Also you can specify directories as well as regular expressions that you should embed in squared brackets '[<my_reg_exp]'. For convenience you can specify a path starting with an hyphen ('-') to indicated that you want to veto it if it were to be selected by another regular expression. Here is an example of a more complicated file specifier:
     65
     66""" target: testScratch/[Folder(A|C)\/.+\.(f|inc)]
     67     target: FileOut.txt
     68     target: -testScratch/[Folder(A|C)\/File(X|Y).f]
     69"""
     70
     71To conclude, here is an example of a minimal, yet fully functional IOTest:
     72
     73class IOTest_SimpleExamples(IOTests.IOTestManager):
     74
     75    @IOTests.createIOTest()
     76        """ target: aFile.txt
     77        """
     78        contentToCheck = "This is the content to check"
     79        open(pjoin(self.IOpath,'aFile.txt'),'w').write(contentToCheck)
    6280
    6381= Unit tests =