Reading Delphes files with uproot ("fBits is not always 4 bytes")

[Pietro Vischia]

Dear Delphes support,

I am using Delphes and python on a Mac M1.

I am (trying to) use uproot to read ROOT Delphes output files (all other methods failed, because I could not find a way of reliably make pyroot read the relevant delphes class definitions---not even the Example1.py runs).

However, uproot fails because it expects fBits to have a (standard, as far as I understand) size of 4 bytes, but in Delphes at least some of them (in particular for me it crashes at Particle.fBits) have a different number of bytes.

Concretely, the error I get is:

ValueError: basket 2 in tree/branch /Delphes;1:Particle/Particle.fBits has the wrong number of bytes (61902) for interpretation AsDtype('>u4')

Apparently this is not trivial to fix in uproot, as it would require major rewriting to accept random values for fBits. I was wondering if it's possible to have a workaround, from the users' side, for the issue by telling Delphes to write out regular rootfiles without the nonstandard fBits size?

The details of the issue are outlined here: ​https://github.com/scikit-hep/uproot5/issues/878 .

Alternatively, I would appreciate any pointer for the following (from the Example1.py) to work:

   ROOT.gSystem.Load("/Users/vischia/workarea/generation/delphes/libDelphes.so")
    try:
        ROOT.gInterpreter.Declare('#include "/Users/vischia/workarea/generation/delphes/classes/DelphesClasses.h"')
        ROOT.gInterpreter.Declare('#include "/Users/vischia/workarea/generation/delphes/external/ExRootAnalysis/ExRootAnalysisLinkDef.h"')
        ROOT.gInterpreter.Declare('#include "/Users/vischia/workarea/generation/delphes/external/ExRootAnalysis/ExRootTreeReader.h"')
  

Even just loading libDelphes.so fails with

/opt/homebrew/Cellar/root/HEAD-e364c89/lib/root:/opt/homebrew/Cellar/root/6.30.04/lib/root:/opt/homebrew/Cellar/root/6.28.06/lib/root:/Users/vischia/workarea_temp/equivariant/delphes/
In file included from input_line_36:1:
/Users/vischia/workarea/generation/delphes/classes/DelphesClasses.h:39:10: fatal error: 'classes/SortableObject.h' file not found
#include "classes/SortableObject.h"
         ^~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from input_line_37:1:
/Users/vischia/workarea/generation/delphes/external/ExRootAnalysis/ExRootAnalysisLinkDef.h:10:10: fatal error: 'ExRootAnalysis/ExRootTreeReader.h' file not found
#include "ExRootAnalysis/ExRootTreeReader.h"
         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

followed by a series of "dictionary not available" warnings:

TClass::Init:0: RuntimeWarning: no dictionary for class HepMCEvent is available

Thank you very much in advance!

Cheers,
Pietro

[Pietro Vischia]

P.S. I forgot to mention that my use case is to load the whole rootfile into an awkward array, pandas dataframe, or numpy array (with or without passing through a RDataFrame) to then dump the resulting array into a hdf5 file as a dataset.

Cheers,
Pietro

[Pavel Demin]

I think that the uproot compatibility issue could be resolved by writing a new module similar to TreeWriter that would be compatible with uproot.

Regarding the libDelphes library loading problem, I can recommend at least two solutions:

• run ROOT and gSystem->Load("libDelphes"); from the directory containing the Delphes source code (/Users/vischia/workarea/generation/delphes/)
• try adding the necessary include paths using gInterpreter->AddIncludePath()

I think that the following commands should work:

gInterpreter->AddIncludePath("/Users/vischia/workarea/generation/delphes/external/");
gInterpreter->AddIncludePath("/Users/vischia/workarea/generation/delphes/classes/");

[Pietro Vischia]

I now load pretty much everything:

ROOT.gSystem.Load("/Users/vischia/workarea/generation/delphes/libDelphes.so")
ROOT.gInterpreter.AddIncludePath("Users/vischia/workarea/generation/delphes/");
ROOT.gInterpreter.AddIncludePath("Users/vischia/workarea/generation/delphes/external/");
ROOT.gInterpreter.AddIncludePath("Users/vischia/workarea/generation/delphes/classes/");
ROOT.gInterpreter.Declare('#include "/Users/vischia/workarea/generation/delphes/classes/DelphesClasses.h"')
ROOT.gInterpreter.Declare('#include "/Users/vischia/workarea/generation/delphes/external/ExRootAnalysis/ExRootAnalysisLinkDef.h"')
ROOT.gInterpreter.Declare('#include "/Users/vischia/workarea/generation/delphes/external/ExRootAnalysis/ExRootTreeReader.h"')

but pyroot still fails with:

In file included from input_line_36:1:
/Users/vischia/workarea/generation/delphes/classes/DelphesClasses.h:39:10: fatal error: 'classes/SortableObject.h' file not found
#include "classes/SortableObject.h"
         ^~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from input_line_37:1:
/Users/vischia/workarea/generation/delphes/external/ExRootAnalysis/ExRootAnalysisLinkDef.h:10:10: fatal error: 'ExRootAnalysis/ExRootTreeReader.h' file not found
#include "ExRootAnalysis/ExRootTreeReader.h"

I cannot understand why. With respect to your suggestion, I also added

gInterpreter->AddIncludePath("Users/vischia/workarea/generation/delphes/");
hoping this would take care of making the code recognize includes like classes/SortableObject.h, but it doesn't seem to work

[Pavel Demin]

I see some problems with the order of the commands and with the paths in your code example.

Please try the following python commands:

import ROOT
ROOT.gInterpreter.AddIncludePath("/Users/vischia/workarea/generation/delphes/classes")
ROOT.gInterpreter.AddIncludePath("/Users/vischia/workarea/generation/delphes/external")
ROOT.gSystem.Load("/Users/vischia/workarea/generation/delphes/libDelphes")

I have just tried these commands and they work for me without any error messages.

[Pavel Demin]

I now see that I forgot the slash in my first example. I have fixed it and updated my first comment.