I have a long-running calculation, for which want to emit a log that allows recovery of exact numerical values, so that I can restart or debug the calculation. The typical logged item can be thought of as a (possibly nested) NamedTuple (or Dict), with vectors and scalars, eg (a = rand(SVector{3,Flo…

NB: @Oscar_Smith , to understand your post better, this helped.

[image] Henrique_Becker: Is this guaranteed? Yes, it’s guaranteed by modern float-to-string algorithms like Ryu (which is employed by Julia) and Grisu (which Julia used to employ). This “information preservation” property was codified by Steele and White (1990) and I think it’s become widely…

Very good to know. A shame this is not adopted by more languages. I had this problem about 4 years ago, so it seems like C/C++ is not set on adopting it (as there were C/C++ standard updates between 1990 and 2016 and the problem still existed at 2016).

The problem here is that C/C++ farm a lot out the the OS, and some OSes are pretty bad with some of the finicky stuff like this.

[image] stevengj: What’s wrong with the print output (as used in JSON or whatever)? Nothing, that’s fine. I just need to embed it in JSON etc so that it is easier to process the file with a script if necessary. [image] stephenll: We’ve had a lot of success using TOML to store thousands o…

The largest file we’ve created contains about 1 million floats in total. That takes about .4 seconds to write and about .3 seconds to read on my 2019 MacBook Pro.

Not sure, how close it to your question, but I think there is no need in removing @info, you just need to change your sink. For example LoggingFacilities.jl provides conversion of usual logging output to JSON format. With the help of LoggingExtras.jl you can setup your logging to write to HDF/BSON/w…

Tamas, what a nice idea! It’s a logstash-style of program serialization. I remain a cheerleader for HDF, but they do refer to strings as “Other Non-numeric Datatypes” in their user manual, so maybe there’s a little neglect in that area.

One optition is using TensorBoardLogger.jl. It logs to a ProtoBuf file, that can be displayed by the TensorBoard program, but also can be read by a number of libraries (including TensorBoardLogger.jl itself) [image] GitHub - JuliaLogging/TensorBoardLogger.jl: Easy peasy logging to Tensor…

There is also the option of (Google’s) protocol buffers, which have a Julia implementation ( ProtoBuf.jl ). I don’t have any experience with them myself, but they are a binary-only JSON competitor and might be worth looking into for scaling to large sizes.

Exact numerical logging

General Usage

Tamas_Papp April 15, 2021, 3:05pm 22

Thanks. I just found

which could make that a more disciplined approach. Will experiment further and get back here.

Topic		Replies	Views
What is the preferred way to save variables? General Usage jld , hdf5 , jld2	39	18930	August 24, 2021
A future for JLD2? Community jld2	56	10285	July 19, 2020
(De-)Serialize N-dimensional arrays in julia New to Julia question , package , serialization	28	2394	June 4, 2021
Can't read old JLD2 file Tooling	17	3137	February 19, 2019
Human-readable externalization for multi-dimensional arrays General Usage question	23	1745	August 13, 2019

Exact numerical logging

Related topics