Changes between Initial Version and Version 1 of TestProgram

Timestamp:

Mar 20, 2012, 4:18:57 PM (12 years ago)

Author:

trac

Comment:

--

TestProgram

@@ +1 @@
+
+
+
+
+== Where is the test code? ==
+
+   1. The code is on cp3wks05 in the directory /Users/madgraph/TestMG 
+   1. He is also available via CVS: ''cvs co !TestMG'' . He is suppose to be place on same directory as MG_ME
+
+== How do you execute the test? ==
+
+you can run two different types of tests: "fast" and "full" (see next section for details).
+The main script that controls the various tests is a Python script called test.py.
+Here are some examples of commands you can type to perform the tests:
+
+   *    $>./test.py fast all         (to do all available checks in the "fast" mode)
+   *    $>./test.py fast MW      (to check only MadWeight in the "fast" mode)
+   *    $>./test.py full all          (to do all available checks in the "full" mode)    
+   *    $>./test.py fast MW      (to check MadWeight MadWeight in the "full" mode)
+   *    $>./test.py MW/NAME  (to launch only the fast test  NAME in MW) 
+
+
+== What is the difference between the "fast" mode and the "full" mode? ==
+
+These are two different levels of test, for each module in MadGraph: 
+   * the "FAST" mode is intended to be rapid. It should not last much more than 10 minutes or so. You can use it to check whether 
+     some modifications in a specific module (e.g. MadOnia) do not interfere with the good-working of other modules (e.g. MadDipol). 
+   * the "FULL" mode is intended to check that ''all the results''  you can produce with a specific updated  module is in agreement with the results from the previous version of this module.
+
+Essentially, if you update a specific module ''mymod'', you can run the test script in "fast" mode to test all the modules, 
+
+ $>./test.py fast all
+
+and in "full" mode to test specifically ''mymod''
+
+ $>./test.py full mymod
+
+NOTE: for the moment only madonia has is own full test.
+
+== Structure of the code ==
+
+All required information to test a specific module is contained in one given
+directory located in   /Users/madgraph/TEST. This directory should be named
+as '''test_????'''. For a specific test in this module, the relevant files are recorded 
+in a sub-directory '''fast_???''' or '''full_???'''. Each of these test directories contains
+
+   1. all input Cards or files required to perform the test,
+   1. the file '''move_files_level1''' 
+       which specifies which input files should be copied at level 1 (i.e. before the execution of any script) 
+       and in which directory they must be copied in, 
+   1. the file '''routine_level1''' specifies which scripts should be executed first (usually ./bin/newprocess) 
+   1. the file '''move_files_level2''' 
+       which specifies which input files should be copied at level 2 (i.e. after the execution of the scripts mentioned in the file routine_level1) 
+       and in which directory they must be copied in, 
+   1. the file '''routine_level2''' specifies which scripts should be executed at level 2 (such as ./bin/generate_events) 
+   1. the file '''dir_test.py''' which specifies how to extract the results to be compared with the expected one (see section [http://cp3wks05.fynu.ucl.ac.be/twiki/bin/view/Software/TestProgram#How_to_write_dir_test_py how define dir_test.py])
+   1. the file '''expected_results.txt''' which contains the reference results. 
+
+Note: the above files can also be placed in the directory '''test_????'''. They are then used in case there is no such file in the '''fast_????''' (or '''full_????''') directory.
+
+== How does it work? ==
+   1. The script test.py copies the directory "Template" into "Auto_test_???" 
+   1. It copies the input files according to the prescription given in '''move_files_level1'''
+   1. It launches executables (such as new_process) according to the prescription given in '''routine_level1'''.
+       The paths are expected to be given from the main directory of the Template copy.
+   1. It copies the input files according to the prescription given in '''move_files_level2'''
+   1. It launches executables (such as generate_events) according to the prescription given in '''routine_level2'''.
+       The paths are expected to be given from the main directory of the Template copy. 
+   1. A python code collects the results.
+   1. These results are compared to the expected one written in '''expected_results.txt'''
+
+
+== How to implement a new test?  ==
+
+
+   1. First create a new directory in TEST for your module: $>mkdir test_MYMOD
+   1. $>cd test_MYMOD
+   1. Create a directory for a specific test $> mkdir fast_NAME or $>mkdir full_NAME
+   1. $> cd fast_NAME 
+   1. Define the six files  '''move_files_level1''',  '''move_files_level2''', '''routine_level1''' ,  '''routine_level2''',  
+ '''dir_test.py''', '''expected_results.txt''' and import a copy of  all input cards (or files) in this directory.
+
+
+=== A practical way to develop all those things... ===
+
+This is only a proposition of the different step used to devellop a new step
+   1. Create your new test directory
+   1. The defines the four files  '''move_files_level1''',  '''move_files_level2''', '''routine_level1''' ,  '''routine_level2''' (this should be easy)
+   1. Copy the default files /Users/madgraph/TEST/Python_lib/dir_test_default.py in your directory '''fast_????''' (respectively '''full_????''')
+   1. Change his name in dir_test.py
+   1. Launch the test: (in TEST)$> ./test.py test_????/fast_????  (respectively for full)
+   1. Edit the value of actif and put this value to zero (avoid the overwriting of the Template copy and the execution of the different test script)
+   1. Implement your python (or external) routine and test this one with  (in TEST)$> ./test.py test_MYMOD/fast_MYTEST 
+   1. When the last step  works correctly, remove the line actif=0 of the file dir_test.py
+   1. Copy the file test_result.dat in 'expected_result.dat': (in fast_???) $> cp test_results.dat expected_results.dat
+   1. add the third column (precision requirment) in the file expected_results.dat
+   1. launch completly your test to be sure that everithing is fine. (in TEST)$> ./test.py test_MYMOD/fast_MYTEST
+   1. store your module on the cvs
+
+=== How to write input files? ===
+==== How to write  "move_files_level1" and  "move_files_level2" ? ====
+
+ This files contains two columns.
+   1. The first one indicates the name of the files to copy (path starting from directory '''fast_????''' (or '''full_????''')
+   1. The second indicates where to copy the files (path starting from the main directory of the Template copy)
+All lines begining with a # not taking into account
+
+==== How to write "routine_level1" and "routine_level2"? ====
+
+Those files are simply batch files. They will be executed from the main directory of the Template copy.
+A good way to code those files will be to redirect all the output.
+
+==== How to write "dir_test.py" ? ====
+
+   1. First copy the file /Users/madgraph/TEST/Python_lib/dir_test_default.py in your fast_???? (or full_????) directory
+   1. Change his name in dir_test.py
+   1. Then follows the instruction inside (All the line starting with '#>>' indicates an action to perform). 
+
+This files indicates how to use an external programs and what you have to do if you choose to implement your python code. 
+
+==== How to write "expected_results.txt" ?  ====
+This files must contains three columns
+   1. the first one is of string symbolizing the result (this string must be unique)
+   1. The second one is the expected value 
+   1. The third one is the difference pourcentage authorize without warning. 
+      a. if you want to compare string put 's'
+      a. if you want to have exact value put '0'
+
+==== How to write "test_results.txt" ?  ====
+If you choose to develop your own code to collect data, you have to write explicitly the file 'test_result.txt' (in the directory '''fast_????''' or '''full_????''')
+This files must contains two columns
+   1. The first one is of string symbolizing the result (this string must be unique)
+   1. The second one is the related value
+
+
+== Discussion/Idea?  ==
+
+
+
+
+-- Main.OlivierMattelaer - 18 Nov 2008