#220 closed Enhancement (fixed)
using URL to stream data for Delphes?
Reported by: | Sergei Chekanov | Owned by: | |
---|---|---|---|
Priority: | minor | Milestone: | |
Component: | Delphes code | Version: | Delphes 3 |
Keywords: | promc | Cc: |
Description
Hi,
Is it possible to use URL as a location for an input files, so there is no need to download them? I'm interested in streaming input files, which will be great for debugging.
./DelphesProMC [URL link to ProMC files] output.root
where truths ProMC file is from
http://atlaswww1.hep.anl.gov/asc/snowmass2013/delphes36/
best, Sergei
Change History (8)
comment:2 by , 11 years ago
Another solution could be using httpfs.
Here are the commands to install and use httpfs under Scientific Linux 6:
sudo yum install fuse fuse-devel sudo chmod +x /bin/fusermount wget http://sourceforge.net/projects/httpfs/files/httpfs2/httpfs2-0.1.5.tar.gz tar -zxf httpfs2-0.1.5.tar.gz cd httpfs2-0.1.5 make cd .. mkdir test ./httpfs2-0.1.5/httpfs2 -c /dev/null http://atlaswww1.hep.anl.gov/asc/snowmass2013/delphes36/ttbar/truth/herwigpp_ttbar_truth_1.promc test ./DelphesProMC examples/delphes_card_CMS.tcl test.root test/herwigpp_ttbar_truth_1.promc killall httpfs2
It seems to work.
comment:3 by , 11 years ago
Just found an example of how to read the ProMC files from a web server:
https://atlaswww.hep.anl.gov/asc/wikidoc/doku.php?id=asc:promc:examples#random_access
I'll check if it's possible to write a ProMC reader for Delphes in Python.
comment:4 by , 11 years ago
Hello, Pavel
Thanks a lot for the suggestions. I'll try to go over various tools you have suggested to see what can be done. If not, we will add this functionality to the C++ code. I know Java and Python can be used
(for example, one can read a ProMC file using http & java):
java -jar browser_promc.jar http://atlaswww1.hep.anl.gov/asc/RefHepSim/events/pp/14tev/higgs/pythia8/pythia8_higgs_1.promc
best, Sergei
comment:5 by , 11 years ago
I've just noticed that the Python code example at
https://atlaswww.hep.anl.gov/asc/wikidoc/doku.php?id=asc:promc:examples#random_access
does not actually implement a random remote access.
It downloads all the file to memory and only then starts to read events.
So, it's almost equivalent to
wget http://atlaswww1.hep.anl.gov/asc/snowmass2013/delphes36/ttbar/truth/herwigpp_ttbar_truth_1.promc ./DelphesProMC examples/delphes_card_CMS.tcl test.root herwigpp_ttbar_truth_1.promc
I've also found a class that implements random remote access at
I'll use this HttpFile class in the new ProMC reader for Delphes.
comment:6 by , 11 years ago
Hi Sergei,
Here is a preliminary version of the new ProMC reader for Delphes written in Python:
https://cp3.irmp.ucl.ac.be/projects/delphes/browser/trunk/readers/DelphesProMC.py
It can read both local and remote files.
Unfortunately, it's 6 times slower than the C++ ProMC reader because of the particle loop written in pure Python.
I've also noticed an inconsistency between C++ and Python ProMC interfaces:
- in test/promc/src/ProMC.pb.h, there is ProMCEvent_Event::weight() defined;
- in examples/python/modules/ProMC_pb2.py, there is no promc.ProMCEvent.Event.Weight defined.
Is it expected?
Cheers,
Pavel
comment:7 by , 11 years ago
Resolution: | → fixed |
---|---|
Status: | new → closed |
DelphesProMC.py is available in the new release 3.0.10.
Hi Sergei,
For some formats I can read data from the standard input stream and I can, for example, pipe curl to Delphes to read data directly from a web server:
However, I don't see how to do it with the ProMC library.
ProMCBook takes file name as an argument, ZipFile also takes file name as an argument. So, there is no way to pass a pointer to stdin.
Does the ProMCBook or ZipFile classes recognize URL links and read files directly form a web server?
Looking at the ZIP file structure at
http://www.pkware.com/documents/casestudies/APPNOTE.TXT
http://en.wikipedia.org/wiki/File:ZIP-64_Internal_Layout.svg
I'd say that simple streaming of the ZIP file won't work. The directory structure is located at the end of the file and should be read first.
Reading the central directory structure is not that trivial:
http://stackoverflow.com/questions/8593904/how-to-find-the-position-of-central-directory-in-a-zip-file
At least 3 HTTP requests are needed to read the file structure:
If it's not already implemented in the ProMC or Zipios++ libraries, it means that some work is required to add this functionality.