Fork me on GitHub

Opened 11 years ago

Closed 11 years ago

Last modified 11 years ago

#220 closed Enhancement (fixed)

using URL to stream data for Delphes?

Reported by: Sergei Chekanov Owned by:
Priority: minor Milestone:
Component: Delphes code Version: Delphes 3
Keywords: promc Cc:

Description


Hi,

Is it possible to use URL as a location for an input files, so there is no need to download them? I'm interested in streaming input files, which will be great for debugging.

./DelphesProMC [URL link to ProMC files] output.root

where truths ProMC file is from
http://atlaswww1.hep.anl.gov/asc/snowmass2013/delphes36/

best, Sergei

Change History (8)

comment:1 by Pavel Demin, 11 years ago

Hi Sergei,

For some formats I can read data from the standard input stream and I can, for example, pipe curl to Delphes to read data directly from a web server:

curl http://cp3.irmp.ucl.ac.be/~demin/z_ee.hep | ./DelphesSTDHEP examples/delphes_card_CMS.tcl delphes_output.root

However, I don't see how to do it with the ProMC library.

ProMCBook takes file name as an argument, ZipFile also takes file name as an argument. So, there is no way to pass a pointer to stdin.

Does the ProMCBook or ZipFile classes recognize URL links and read files directly form a web server?

Looking at the ZIP file structure at

http://www.pkware.com/documents/casestudies/APPNOTE.TXT
http://en.wikipedia.org/wiki/File:ZIP-64_Internal_Layout.svg

I'd say that simple streaming of the ZIP file won't work. The directory structure is located at the end of the file and should be read first.

Reading the central directory structure is not that trivial:

http://stackoverflow.com/questions/8593904/how-to-find-the-position-of-central-directory-in-a-zip-file

At least 3 HTTP requests are needed to read the file structure:

  • HEAD request to read the file size
  • GET request for ~65 kilobytes from the end of the file to find the size of the central directory structure
  • GET request for X bytes from the end of the file to read the central directory structure

If it's not already implemented in the ProMC or Zipios++ libraries, it means that some work is required to add this functionality.

Last edited 11 years ago by Pavel Demin (previous) (diff)

comment:2 by Pavel Demin, 11 years ago

Another solution could be using httpfs.

Here are the commands to install and use httpfs under Scientific Linux 6:

sudo yum install fuse fuse-devel
sudo chmod +x /bin/fusermount

wget http://sourceforge.net/projects/httpfs/files/httpfs2/httpfs2-0.1.5.tar.gz
tar -zxf httpfs2-0.1.5.tar.gz

cd httpfs2-0.1.5
make

cd ..

mkdir test
./httpfs2-0.1.5/httpfs2 -c /dev/null http://atlaswww1.hep.anl.gov/asc/snowmass2013/delphes36/ttbar/truth/herwigpp_ttbar_truth_1.promc test

./DelphesProMC examples/delphes_card_CMS.tcl test.root test/herwigpp_ttbar_truth_1.promc

killall httpfs2

It seems to work.

comment:3 by Pavel Demin, 11 years ago

Just found an example of how to read the ProMC files from a web server:

https://atlaswww.hep.anl.gov/asc/wikidoc/doku.php?id=asc:promc:examples#random_access

I'll check if it's possible to write a ProMC reader for Delphes in Python.

comment:4 by Sergei Chekanov, 11 years ago

Hello, Pavel

Thanks a lot for the suggestions. I'll try to go over various tools you have suggested to see what can be done. If not, we will add this functionality to the C++ code. I know Java and Python can be used

(for example, one can read a ProMC file using http & java):

java -jar browser_promc.jar http://atlaswww1.hep.anl.gov/asc/RefHepSim/events/pp/14tev/higgs/pythia8/pythia8_higgs_1.promc

best, Sergei

comment:5 by Pavel Demin, 11 years ago

I've just noticed that the Python code example at

https://atlaswww.hep.anl.gov/asc/wikidoc/doku.php?id=asc:promc:examples#random_access

does not actually implement a random remote access.

It downloads all the file to memory and only then starts to read events.

So, it's almost equivalent to

wget http://atlaswww1.hep.anl.gov/asc/snowmass2013/delphes36/ttbar/truth/herwigpp_ttbar_truth_1.promc

./DelphesProMC examples/delphes_card_CMS.tcl test.root herwigpp_ttbar_truth_1.promc

I've also found a class that implements random remote access at

http://stackoverflow.com/questions/7829311/is-there-a-library-for-retrieving-a-file-from-a-remote-zip

I'll use this HttpFile class in the new ProMC reader for Delphes.

comment:6 by Pavel Demin, 11 years ago

Hi Sergei,

Here is a preliminary version of the new ProMC reader for Delphes written in Python:

https://cp3.irmp.ucl.ac.be/projects/delphes/browser/trunk/readers/DelphesProMC.py

It can read both local and remote files.

Unfortunately, it's 6 times slower than the C++ ProMC reader because of the particle loop written in pure Python.

I've also noticed an inconsistency between C++ and Python ProMC interfaces:

  • in test/promc/src/ProMC.pb.h, there is ProMCEvent_Event::weight() defined;
  • in examples/python/modules/ProMC_pb2.py, there is no promc.ProMCEvent.Event.Weight defined.

Is it expected?

Cheers,

Pavel

comment:7 by Pavel Demin, 11 years ago

Resolution: fixed
Status: newclosed

DelphesProMC.py is available in the new release 3.0.10.

comment:8 by Sergei Chekanov, 11 years ago

Thanks a lot, great!

best wishes, Sergei

Note: See TracTickets for help on using tickets.