Problem reading ROOT branch containing Strings with UnROOT.jl

GSavvidis · June 14, 2024, 7:18pm

Hello,
I’m using Geant4 to simulate some decays and I’m saving the result in a root file which I then read with UnROOT.jl. I have the following code:

using DrWatson
quickactivate(@__DIR__)

using UnROOT

f = UnROOT.ROOTFile(datadir("sims/root/ss", "ss_U238_p=10bar_Rmin=292mm_Rmax=300mm.root"))

with structure
tree

I can successfully access branches except those containing Strings, such as the one called “Particle”, shown in the picture above. When I do:

UnROOT.LazyBranch(f, "G4Sim/Particle")

I’m getting the following error:

 Failed to show value:

TaskFailedException

nested task error: DimensionMismatch: expected input array of length 16, got length 12

Stacktrace:

[1] dimension_mismatch_fail(::Type{StaticArraysCore.SVector{16, UInt8}}, a::SubArray{UInt8, 1, Vector{UInt8}, Tuple{UnitRange{Int64}}, true})

@ StaticArrays ~/.julia/packages/StaticArrays/85pEu/src/convert.jl:196

[2] convert

@ ~/.julia/packages/StaticArrays/85pEu/src/convert.jl:201 [inlined]

[3] StaticArray

@ ~/.julia/packages/StaticArrays/85pEu/src/convert.jl:174 [inlined]

[4] reinterpret(::Type{UnROOT.FixLenVector{16, UInt8}}, v::SubArray{UInt8, 1, Vector{UInt8}, Tuple{UnitRange{Int64}}, true})

@ UnROOT ~/.julia/packages/UnROOT/xaBnN/src/custom.jl:86

[5] #235

@ ./none:0 [inlined]

[6] iterate

@ ./generator.jl:47 [inlined]

[7] collect_to!(dest::Vector{UnROOT.FixLenVector{16, UInt8}}, itr::Base.Generator{Base.Iterators.PartitionIterator{Vector{UInt8}}, UnROOT.var"#235#236"{UnROOT.FixLenVector{16, UInt8}}}, offs::Int64, st::Int64)

@ Base ./array.jl:892

[8] collect_to_with_first!

@ ./array.jl:870 [inlined]

[9] collect(itr::Base.Generator{Base.Iterators.PartitionIterator{Vector{UInt8}}, UnROOT.var"#235#236"{UnROOT.FixLenVector{16, UInt8}}})

@ Base ./array.jl:844

[10] interped_data(rawdata::Vector{UInt8}, rawoffsets::Vector{Int32}, ::Type{UnROOT.FixLenVector{16, UInt8}}, ::Type{UnROOT.Nojagg})

@ UnROOT ~/.julia/packages/UnROOT/xaBnN/src/custom.jl:90

[11] basketarray(f::UnROOT.ROOTFile, branch::UnROOT.TBranch_13, ithbasket::Int64)

@ UnROOT ~/.julia/packages/UnROOT/xaBnN/src/iteration.jl:64

[12] basketarray(lb::UnROOT.LazyBranch{UnROOT.FixLenVector{16, UInt8}, UnROOT.Nojagg, Vector{UnROOT.FixLenVector{16, UInt8}}}, ithbasket::Int64)

@ UnROOT ~/.julia/packages/UnROOT/xaBnN/src/iteration.jl:135

[13] #214

@ ~/.julia/packages/UnROOT/xaBnN/src/iteration.jl:486 [inlined]

[14] iterate

@ ./generator.jl:47 [inlined]

[15] _collect(c::UnitRange{Int64}, itr::Base.Generator{UnitRange{Int64}, UnROOT.var"#214#217"{UnROOT.LazyBranch{UnROOT.FixLenVector{16, UInt8}, UnROOT.Nojagg, Vector{UnROOT.FixLenVector{16, UInt8}}}}}, ::Base.EltypeUnknown, isz::Base.HasShape{1})

@ Base ./array.jl:854

[16] collect_similar

@ ./array.jl:763 [inlined]

[17] map

@ ./abstractarray.jl:3285 [inlined]

[18] getindex(ba::UnROOT.LazyBranch{UnROOT.FixLenVector{16, UInt8}, UnROOT.Nojagg, Vector{UnROOT.FixLenVector{16, UInt8}}}, range::UnitRange{Int64})

@ UnROOT ~/.julia/packages/UnROOT/xaBnN/src/iteration.jl:486

[19] (::UnROOT.var"#244#246"{UnROOT.LazyBranch{UnROOT.FixLenVector{16, UInt8}, UnROOT.Nojagg, Vector{UnROOT.FixLenVector{16, UInt8}}}})()

@ UnROOT ~/.julia/packages/UnROOT/xaBnN/src/displays.jl:95

Stack trace

Here is what happened, the most recent locations are first:

1. <mark>**wait**</mark> @ *task.jl:352*

2. <mark>**show**(io::IOContext{IOBuffer}, ::MIME{Symbol("text/plain")}, br::UnROOT.LazyBranch{UnROOT.FixLenVector{16, UInt8}, UnROOT.Nojagg, Vector{UnROOT.FixLenVector{16, UInt8}}})</mark> @ *displays.jl:97*

I’m on Julia 1.10.2 working on Pluto and I’m using the v0.10.31 version of UnROOT.

tamasgal · June 17, 2024, 9:27am

Can you upload the file somewhere?

GSavvidis · June 17, 2024, 2:10pm

Sure! I uploaded it here cos it’s a .root file and I couldn’t upload it here (apologies if I’m too ignorant).
Also I kept digging trying to understand more. I will put some information here in case it could be useful:

From python’s uproot, after reading the tree and calling

tree.typenames()

I got

{'fEvent': 'int32_t', 'ParentID': 'int32_t', 'TrackID': 'int32_t', 
'Particle': 'char*', 'MeanLife': 'double', 'Charge': 'double', 
'Process': 'char*', 'Edep': 'double', 'preTime': 'double', 'postTime': 'double', 'localTime': 'double', 'preKE': 'double', 'postKE': 'double', 
'preX': 'double', 'preY': 'double', 'preZ': 'double', 
'postX': 'double', 'postY': 'double', 'postZ': 'double', 
'preVolume': 'char*', 'postVolume': 'char*'}

From inspection of the branch in Julia I got:

br = f["G4Sim/Process"]
println(br.fLeaves)

UnROOT.TObjArray("", 0, Any[UnROOT.TLeafC
  fName: String "Process"
  fTitle: String "Process"
  fLen: Int32 16
  fLenType: Int32 1
  fOffset: Int32 0
  fIsRange: Bool false
  fIsUnsigned: Bool false
  fLeafCount: UInt32 0x00000000
  fMinimum: Int32 0
  fMaximum: Int32 16
])

Hope this can be useful. Please let me know if more information is needed.

I also tried to convert my TTree into RNTuple using PyROOT but as I’m not using pyroot, I failed miserably.

Edit: Including also the part of Geant4 I implemented to save the data

    G4AnalysisManager* analysisManager2 = G4AnalysisManager::Instance();

    analysisManager2->FillNtupleIColumn(0, eventNumber);
    analysisManager2->FillNtupleIColumn(1, parentID);
    analysisManager2->FillNtupleIColumn(2, trackID);
    analysisManager2->FillNtupleSColumn(3, name);
    analysisManager2->FillNtupleDColumn(4, meanLife);
    analysisManager2->FillNtupleDColumn(5, fCharge);
    analysisManager2->FillNtupleSColumn(6, proc);
    analysisManager2->FillNtupleDColumn(7, edepStep);
    analysisManager2->FillNtupleDColumn(8, preTime);
    analysisManager2->FillNtupleDColumn(9, postTime);
    analysisManager2->FillNtupleDColumn(10, localTime);
    analysisManager2->FillNtupleDColumn(11, preKE);
    analysisManager2->FillNtupleDColumn(12, postKE);
    analysisManager2->FillNtupleDColumn(13, prePoint_x);
    analysisManager2->FillNtupleDColumn(14, prePoint_y);
    analysisManager2->FillNtupleDColumn(15, prePoint_z);
    analysisManager2->FillNtupleDColumn(16, postPoint_x);
    analysisManager2->FillNtupleDColumn(17, postPoint_y);
    analysisManager2->FillNtupleDColumn(18, postPoint_z);
    analysisManager2->FillNtupleSColumn(19, preVolume);
    analysisManager2->FillNtupleSColumn(20, postVolume);
    analysisManager2->AddNtupleRow(0);

jling · June 17, 2024, 6:59pm

if I manually get the rawdata and rawoffsets out, I see:

julia> String.(VectorOfVectors(debug[].rawdata, debug[].rawoffsets .+ 1))
846-element Vector{String}:
 "\x0fRadioactivation"
 "\aionIoni"
 "\aionIoni"
 "\x0fRadioactivation"
 "\x05eIoni"
 "\x0eTransportation"
 "\x0eTransportation"
 "\x0fRadioactivation"
 "\x05eIoni"
 "\x0fRadioactivation"
 "\x05eIoni"
 "\aionIoni"
 "\x0fRadioactivation"
 "\x05eIoni"
 "\x05eIoni"
 "\x05eIoni"
 "\x05eIoni"
 "\x05eIoni"
 "\x05eIoni"

Which seems mildly reasonable.

The bug seems to be that we’re interpreting this branch as FixLenVector instead of the normal offset jagged vector.

jling · June 17, 2024, 7:01pm

github.com

JuliaHEP/UnROOT.jl/blob/01969242f7006350ba0f194e615be63fd448df48/src/root.jl#L480


      
                  # since no classname were found, we now try to determine
                  # type based on leaf information
                  _type, _jaggtype = leaf_jaggtype(leaf, _jaggtype)
              end
              return _type, _jaggtype
          end
          
          function leaf_jaggtype(leaf, _jaggtype)
                  _type = primitivetype(leaf)
                  leafLen = leaf.fLen
                  if leafLen > 1 # treat NTuple as Nojagg since size is static
                      _fTitle = replace(leaf.fTitle, "[$(leafLen)]" => "")
                      # looking for more `[var]`
                      m = match(r"\[\D+\]", _fTitle)
                      _type = FixLenVector{Int(leafLen), _type}
                      if isnothing(m)
                          return _type, Nojagg
                      else
                          #FIXME this only handles [var][fix] case
                          return Vector{_type}, Nooffsetjagg
                      end

this is the root cause of the bug – in this case even though leafLen is > 1, somehow it’s not really fixed length?

github.com

root-project/root/blob/86a5a713474e37d96735231da74c000056d86d0c/tree/tree/inc/TLeaf.h#L72


      
          
          protected:
          
             using Counts_t = std::vector<Int_t>;
             struct LeafCountValues {
                Counts_t fValues;
                Long64_t fStartEntry{-1}; ///<! entry number of corresponding to element 0 of the vector.
             };
          
             Int_t            fNdata;           ///<! Number of elements in fAddress data buffer.
             Int_t            fLen;             ///<  Number of fixed length elements in the leaf's data.
             Int_t            fLenType;         ///<  Number of bytes for this data type
             Int_t            fOffset;          ///<  Offset in ClonesArray object (if one)
             bool             fIsRange;         ///<  (=true if leaf has a range, false otherwise).  This is equivalent to being a 'leafcount'.  For a TLeafElement the range information is actually store in the TBranchElement.
             bool             fIsUnsigned;      ///<  (=true if unsigned, false otherwise)
             TLeaf           *fLeafCount;       ///<  Pointer to Leaf count if variable length (we do not own the counter)
             TBranch         *fBranch;          ///<! Pointer to supporting branch (we do not own the branch)
             LeafCountValues *fLeafCountValues; ///<! Cache of collection/array sizes
          
             TLeaf(const TLeaf&);
             TLeaf& operator=(const TLeaf&);

according to ROOT, this should indeed be fixed length

jling · June 17, 2024, 8:38pm

github.com/JuliaHEP/UnROOT.jl

fix reading TLeafC

JuliaHEP:main ← JuliaHEP:TLeafC_fix

opened 08:35PM - 17 Jun 24 UTC

Moelf

+11 -3

fix: https://discourse.julialang.org/t/problem-reading-root-branch-containing-st…rings-with-unroot-jl/115671/5 ```julia-repl julia> a = LazyTree("./ss_U238_p=10bar_Rmin=292mm_Rmax=300mm.root", "G4Sim"); julia> a Row │ postZ postTime preKE preX MeanLife localTime Edep Trac ⋯ │ Float64 Float64 Float64 Float64 Float64 Float64 Float64 Int3 ⋯ ─────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── 1 │ -102.05345950608458 3.0685201317363455e26 0.0 49.73353991915856 2.034191e26 3.0685201317363455e26 0.0 1 ⋯ 2 │ -102.04651578986307 3.0685201317363455e26 4.1979216899649145 49.73353991915856 -1.0 0.0005457291372516042 4.1979216899649145 3 ⋯ 3 │ -102.05346001428659 3.0685201317363455e26 0.07182831002864987 49.73353991915856 3.004037e15 2.3330065525051903e-6 0.07182831002864987 2 ⋯ 4 │ -102.05346001428659 3.0685201317567714e26 0.0 49.733540045379584 3.004037e15 2.0425886986503205e15 0.0 2 ⋯ 5 │ -102.0539008556144 3.0685201317567714e26 0.025275776502425962 49.733540045379584 -1.0 5.298457306211964e-6 0.025275776502425962 6 ⋯ 6 │ -106.84346294108786 3.0685201317567714e26 0.08209409297571678 49.733540045379584 -1.0 0.02146110939511165 0.0 5 ⋯ 7 │ -2000.0 3.0685201317567714e26 0.08209409297571678 47.430551380285756 -1.0 8.503551864822667 0.0 5 ⋯ 8 │ -102.05346001428659 3.0685201317567714e26 1.344596967101097e-7 49.733540045379584 0.0 7.878121781487446e-305 1.344596967101097e-7 4 ⋯ 9 │ -102.05302086535103 3.0685201317567714e26 0.04298589500133121 49.733540045379584 -1.0 1.041804810150761e-5 0.04298589500133121 8 ⋯ 10 │ -102.05346001428659 3.0685201317567714e26 1.050066202878952e-7 49.733540045379584 0.0 7.741051586449725e-306 1.050066202878952e-7 7 ⋯ 11 │ -102.05337759223123 3.0685201317567714e26 0.012766969704394504 49.733540045379584 -1.0 2.2819678030339488e-6 0.012766969704394504 10 ⋯ 12 │ -102.05346003677495 3.0685201317567714e26 3.0297087505459785e-8 49.733540045379584 1.00325e11 0.0002591365343473467 3.0297087505459785e-8 9 ⋯ 13 │ -102.05346003677495 3.0685201317567714e26 0.0 49.73354002365522 1.00325e11 1.414757510502053e10 0.0 9 ⋯ 14 │ -102.0850983567464 3.0685201317567714e26 0.6354402172300221 49.73354002365522 -1.0 0.0002512298153439289 0.09170627685882858 13 ⋯ 15 │ -102.09325779364904 3.0685201317567714e26 0.5437339403711935 49.679954554654216 -1.0 0.00046216017918337645 0.11757367433007723 13 ⋯ 16 │ -102.07580116793557 3.0685201317567714e26 0.42616026604111623 49.63375029065793 -1.0 0.000623815479346648 0.06776978396433882 13 ⋯ 17 │ -102.08504063006555 3.0685201317567714e26 0.35839048207677743 49.59869675442205 -1.0 0.0007576498289719425 0.07490690583766167 13 ⋯ 18 │ -102.07678505845898 3.0685201317567714e26 0.28348357623911574 49.62889262762954 -1.0 0.000861614254051986 0.042674996475252816 13 ⋯ 19 │ -102.09451113711731 3.0685201317567714e26 0.24080857976386294 49.65041254089083 -1.0 0.000948895940544778 0.1282858215721039 13 ⋯ 20 │ -102.09189861553278 3.0685201317567714e26 0.11252275819175905 49.6500709917623 -1.0 0.000985338653087699 0.03837228992128173 13 ⋯ 21 │ -102.08989108494839 3.0685201317567714e26 0.07415046827047733 49.646863130739476 -1.0 0.0010065402178248463 0.047574254567196175 13 ⋯ 22 │ -102.09023049871139 3.0685201317567714e26 0.026576213703281157 49.645250682518146 -1.0 0.0010121881336086595 0.026576213703281157 13 ⋯ 23 │ -103.57478100532907 3.0685201317567714e26 1.6320048207520537 49.73354002365522 -1.0 0.01572578470351275 0.0 12 ⋯ 24 │ -730.8722678518567 3.0685201317567714e26 1.6320048207520537 51.29807043849364 -1.0 6.500054487921658 0.0 12 ⋯ 25 │ -102.0534596801017 3.0685201317567714e26 5.041423719376326e-6 49.73354002365522 1.117712e22 0.0002572019758040831 5.041423719376326e-6 11 ⋯ 26 │ -102.0534596801017 3.0685363408853736e26 0.0 49.73354012060899 1.117712e22 1.6209128602251994e21 0.0 11 ⋯ 27 │ -102.04642108042141 3.0685363408853736e26 4.7745444062238676 49.73354012060899 -1.0 0.000606515507465791 4.7745444062238676 15 ⋯ 28 │ -102.05346024604968 3.0685363408853736e26 0.08312559378100559 49.73354012060899 3.432811e21 2.7993134822468136e-6 0.08312559378100559 14 ⋯ 29 │ -102.05346024604968 3.0685467785417935e26 0.0 49.73354048515049 3.432811e21 1.0437656419799024e21 0.0 14 ⋯ 30 │ -102.0578740887279 3.0685467785417935e26 4.686717724997379 49.73354048515049 -1.0 0.0005971372135061732 4.686717724997379 17 ⋯ 31 │ -102.05345986634265 3.0685467785417935e26 0.08304227498592809 49.73354048515049 7.284479e19 2.8973621866059515e-6 0.08304227498592809 16 ⋯ ⋮ │ ⋮ ⋮ ⋮ ⋮ ⋮ ⋮ ⋮ ⋮ ⋱ 14 columns and 117333 rows omitted julia> a.Process 117364-element LazyBranch{SubArray{Char, 1, Vector{Char}, Tuple{UnitRange{Int64}}, true}, UnROOT.Nooffsetjagg, ArraysOfArrays.VectorOfVectors{Char, Vector{Char}, Vector{Int32}, Vector{Tuple{}}}}: ['\x0f', 'R', 'a', 'd', 'i', 'o', 'a', 'c', 't', 'i', 'v', 'a', 't', 'i', 'o', 'n'] ['\a', 'i', 'o', 'n', 'I', 'o', 'n', 'i'] ['\a', 'i', 'o', 'n', 'I', 'o', 'n', 'i'] ['\x0f', 'R', 'a', 'd', 'i', 'o', 'a', 'c', 't', 'i', 'v', 'a', 't', 'i', 'o', 'n'] ... julia> a.Particle 117364-element LazyBranch{SubArray{Char, 1, Vector{Char}, Tuple{UnitRange{Int64}}, true}, UnROOT.Nooffsetjagg, ArraysOfArrays.VectorOfVectors{Char, Vector{Char}, Vector{Int32}, Vector{Tuple{}}}}: ['\x04', 'U', '2', '3', '8'] ['\x05', 'a', 'l', 'p', 'h', 'a'] ['\x05', 'T', 'h', '2', '3', '4'] ['\x05', 'T', 'h', '2', '3', '4'] ['\x02', 'e', '-'] ['\t', 'a', 'n', 't', 'i', '_', 'n', 'u', '_', 'e'] ... ``` I think somehow this is still wrong, but not sure how to fix without knowing what the data should be.

I have added a PR that fixes this now. Fundamentally we treat this as Vector{Char} since that’s what TLeafC seems to be. Let’s continue the discussion in the PR.

GSavvidis · June 17, 2024, 10:12pm

Thank you!

tamasgal · June 18, 2024, 7:19am

That’s indeed a bummer. Thanks Jerry for the quick PR, looks good so far