Out of curiosity, are there other people already using Julia in the collaboration apart from you?
Yes, but only a handful. I am doing my best to act as a seed 
Edit: actually most of the people who do not work on the DAQ part (C++ and ROOT) use Python. Funnily also the complete monitoring system (both low-level and high-level) is written in Python (almost exclusively by myself) and I plan to convert it to Julia after my PhD defense.
4 Likes
Btw. it’s really annoying me that we are stuck with ROOT for the next 3 decades or so, for such a simple data format: although our neutrino detector is quite complex and lots of things are highly non-trivial, the raw data it spits out is just a vector of hits (time [UInt32, ns], PMT ID [UInt16], time over threshold [UInt8, ns]) and that’s basically it.
No reason whatsoever for using a dataformat which was developed for a complex detector design like LHC with lots of different modules and functions. Again, we only detect single photon hits at a given position in 3D.
Anyways… sorry for ranting, I thought this was the right topic to get it off my chest 
5 Likes
Would be very nice if MINUIT is imported to julia
1 Like
You should be able to ] add Minuit2_jll. I hope it wasn’t misleading, but this is only the Minuit2 library. It doesn’t contain Julia bindings. But you should be able to play around with Cxx.jl.
Hope that works for you.
Thanks.
Hey guys, just wanted to let you know that today I worked (and am still working) on the ROOTIO.jl package, which is a native Julia package to read ROOT files.
It’s in an extremely early prototyping phase but I already managed to create some preliminary structure readout:
julia> using ROOTIO
julia> f = ROOTFile("test/samples/raw.root");
julia> keys(f)
10-element Array{String,1}:
"JTRIGGER::JTriggerParameters"
"META"
"E"
"KM3NET_TIMESLICE"
"KM3NET_TIMESLICE_L0"
"KM3NET_TIMESLICE_L1"
"KM3NET_TIMESLICE_L2"
"KM3NET_TIMESLICE_SN"
"KM3NET_EVENT"
"KM3NET_SUMMARYSLICE"
I’ll work on this package the following weeks and hope that I get something usable very soon. I need it for my own work either, so it’s a bigger part of my working time at the moment.
13 Likes
oh wow! nice, I will take a look at it is possible. I just want to also add decompression (lzma) support to the TODO list since most root files are compressed.
Yeah, I will first add zlib and then all the others, once I get something useful out of the files Currently I am still at the low level structure design…
1 Like
Today I looked at the LZMA stuff and found that there is basically no implementation in Julia. I found LibArchive.jl (GitHub - yuyichao/LibArchive.jl: Wrapper for libarchive (Multi-format archive and compression library. Supports gz, gzip, tar, pax, cpio, zip, xar, zstd, lz4, lha, ar, cab, mtree, rar, ISO)) from @yuyichao but it is not suitable for in-memory IOStreams as it is designed to work with files.
There is a small C library called easylzma but I have no experience in wrapping a C library in a Julia package. The implementation itself doesn’t look “easy” either, so translating it to Julia is a lot of work 
easylzma looks unmaintained to me, but it appears that there is already a Julia wrapper for the official liblzma library in XZ Utils.
- CodecXZ.jl:
liblzmawrappers as part of TranscodingStreams.jl.
(and the corresponding liblzma binaries are built for all Julia platforms via Yggdrassil).
(Compression/decompression of byte streams is one of those areas where there is probably little benefit to a “native” Julia implementation.)
4 Likes
Ah I see! I have very little experience with compression algorithms, didn’t know that XZ covers that. That’s great! I am already using CodecZlib so it’s nice that the interface is the same as in CodecXZ thanks to TranscodingStreams.jl.
Thanks for pointing this out, I should have looked more into the XZ stuff
Just being curious: why LMZA? LZ4 usually gives about 25–40% worse compression (depending on the data), but at 10–20x speed (both for compression/decompression). It has now got to the point that one might just enable it by default in all kinds of settings (eg transparently in a ZFS filesystem, deduplicating backup, large datasets) without any measurable performance impact.
4 Likes
That’s because we (at least @jling and myself) have to deal with ROOT files containing LMZA compressed data.
Most of the ROOT files in KM3NeT however are compressed using Zlib (I have to admit, I don’t know why this was chosen). Anyways, in our case we have a huge amount of data but the read-in speed is not so relevant for the raw files since most of the time is spent in reconstructing the events, which can take up to several seconds per event. As an example, a ~5GB run-file contains a few tens of thousands of data, so the I/O in that case is negligible compared to the CPU part.
But for high-level data where we consolidate reconstruction info and create a multi-TB dataset, the I/O takes most of the time in processing pipelines, so thanks for point this out!
We also use HDF5 for high-level data (parallel to the ROOT files) and there I experimented with Blosc in the past which was awesome (the read-in speed was also magnitudes faster compared to zlib), but we had compatibility issues with some HDF5 libraries (I think it was e.g. h5py in Python which was part of the requirement specs), so we decided to stick to zlib as default.
3 Likes
FWIW, libarchive supports in memory operation.
https://github.com/yuyichao/LibArchive.jl/blob/master/test/runtests.jl#L581-L635
1 Like
I too wish we use more LZ4 and indeed we do, ROOT support multiple compression, LZMA and LZ4 are both supported) though the 30% for 1000% looks good, the problem is that 30% in absolute units (PBs) is too much for storage and network overhead (sometimes ~TB of files get remote copied across continent
4 Likes