using URL to stream data for Delphes?

[Sergei Chekanov]

Hi,

Is it possible to use URL as a location for an input files, so there is no need to download them? I'm interested in streaming input files, which will be great for debugging.

./DelphesProMC [URL link to ProMC files] output.root

where truths ProMC file is from
​http://atlaswww1.hep.anl.gov/asc/snowmass2013/delphes36/

best, Sergei

[Pavel Demin]

Hi Sergei,

For some formats I can read data from the standard input stream and I can, for example, pipe curl to Delphes to read data directly from a web server:

curl http://cp3.irmp.ucl.ac.be/~demin/z_ee.hep | ./DelphesSTDHEP examples/delphes_card_CMS.tcl delphes_output.root

However, I don't see how to do it with the ProMC library.

ProMCBook takes file name as an argument, ZipFile also takes file name as an argument. So, there is no way to pass a pointer to stdin.

Does the ProMCBook or ZipFile classes recognize URL links and read files directly form a web server?

Looking at the ZIP file structure at

​http://www.pkware.com/documents/casestudies/APPNOTE.TXT
​http://en.wikipedia.org/wiki/File:ZIP-64_Internal_Layout.svg

I'd say that simple streaming of the ZIP file won't work. The directory structure is located at the end of the file and should be read first.

Reading the central directory structure is not that trivial:

​http://stackoverflow.com/questions/8593904/how-to-find-the-position-of-central-directory-in-a-zip-file

At least 3 HTTP requests are needed to read the file structure:

• HEAD request to read the file size
• GET request for ~65 kilobytes from the end of the file to find the size of the central directory structure
• GET request for X bytes from the end of the file to read the central directory structure

If it's not already implemented in the ProMC or Zipios++ libraries, it means that some work is required to add this functionality.

[Pavel Demin]

Another solution could be using httpfs.

Here are the commands to install and use httpfs under Scientific Linux 6:

sudo yum install fuse fuse-devel
sudo chmod +x /bin/fusermount

wget http://sourceforge.net/projects/httpfs/files/httpfs2/httpfs2-0.1.5.tar.gz
tar -zxf httpfs2-0.1.5.tar.gz

cd httpfs2-0.1.5
make

cd ..

mkdir test
./httpfs2-0.1.5/httpfs2 -c /dev/null http://atlaswww1.hep.anl.gov/asc/snowmass2013/delphes36/ttbar/truth/herwigpp_ttbar_truth_1.promc test

./DelphesProMC examples/delphes_card_CMS.tcl test.root test/herwigpp_ttbar_truth_1.promc

killall httpfs2

It seems to work.

[Pavel Demin]

Just found an example of how to read the ProMC files from a web server:

​https://atlaswww.hep.anl.gov/asc/wikidoc/doku.php?id=asc:promc:examples#random_access

I'll check if it's possible to write a ProMC reader for Delphes in Python.

[Sergei Chekanov]

Hello, Pavel

Thanks a lot for the suggestions. I'll try to go over various tools you have suggested to see what can be done. If not, we will add this functionality to the C++ code. I know Java and Python can be used

(for example, one can read a ProMC file using http & java):

java -jar browser_promc.jar http://atlaswww1.hep.anl.gov/asc/RefHepSim/events/pp/14tev/higgs/pythia8/pythia8_higgs_1.promc

best, Sergei

[Pavel Demin]

I've just noticed that the Python code example at

​​https://atlaswww.hep.anl.gov/asc/wikidoc/doku.php?id=asc:promc:examples#random_access

does not actually implement a random remote access.

It downloads all the file to memory and only then starts to read events.

So, it's almost equivalent to

wget http://atlaswww1.hep.anl.gov/asc/snowmass2013/delphes36/ttbar/truth/herwigpp_ttbar_truth_1.promc

./DelphesProMC examples/delphes_card_CMS.tcl test.root herwigpp_ttbar_truth_1.promc

I've also found a class that implements random remote access at

​http://stackoverflow.com/questions/7829311/is-there-a-library-for-retrieving-a-file-from-a-remote-zip

I'll use this HttpFile class in the new ProMC reader for Delphes.

[Pavel Demin]

Hi Sergei,

Here is a preliminary version of the new ProMC reader for Delphes written in Python:

https://cp3.irmp.ucl.ac.be/projects/delphes/browser/trunk/readers/DelphesProMC.py

It can read both local and remote files.

Unfortunately, it's 6 times slower than the C++ ProMC reader because of the particle loop written in pure Python.

I've also noticed an inconsistency between C++ and Python ProMC interfaces:

• in test/promc/src/ProMC.pb.h, there is ProMCEvent_Event::weight() defined;
• in examples/python/modules/ProMC_pb2.py, there is no promc.ProMCEvent.Event.Weight defined.

Is it expected?

Cheers,

Pavel

[Pavel Demin]

DelphesProMC.py is available in the new release 3.0.10.

[Sergei Chekanov]

Thanks a lot, great!

best wishes, Sergei