ANN: TraceCalls.jl, a debugging and profiling tool for Julia 0.6

[cstjean]

I’m proud to announce TraceCalls.jl, a functional tracing package with support for IJulia, Atom and the REPL.

Like traditional tracing packages, TraceCalls.jl displays a tree function calls. It goes further by returning a fully-explorable tree-like data structure. This enables some interesting workflows, such as debugging by highlighting the differences between two code versions

Profiling

Or tracking down type instability

Check it out and let me know what you think! The current release has been tested on most major packages, but it is still a bit beta, with a few known limitations. It’s also relatively slow to trace large modules. The next version will improve on that.

Best,

Cédric

[Per]

Looks awesome! A neat feature to implement on top of this would be auto-generatrion of unit tests. Basically re-formatting the output to something like:

@testset "@trace proportions([1,2,2,2,3])" begin
  @test proportions([1,2,2,2,3]) ≈ [0.2, 0.6, 0.2]
    @test span([1,2,2,2,3]) == 1:3 
  ...
end

Optionally, the auto-generated test set could be stored directly in a .jl file (with an associated .h5 file for larger objects) and an include line added to the runtests.jl file of the relevant package.

(Of course, the point of this is not to verify that the code does what it already does, but rather to hunt down bugs by generating tests that trigger them.)

[cstjean]

You could do something like:

> trace_prop = @trace StatsBase proportions([1,2,2,2,3])
> 
> for trace in collect(trace_prop)[2:end] # exclude the root
>     io = STDOUT; mime=MIME"text/plain"()
>     print(io, "@test ")
>     TraceCalls.show_func_name(io, mime, trace)
>     write(io, "(")
>     TraceCalls.show_args(io, mime, trace.args)
>     TraceCalls.show_kwargs(io, mime, trace.kwargs)
>     println(io, ") == $(trace.value)")
> end

@test StatsBase.proportions([1, 2, 2, 2, 3]) == [0.2, 0.6, 0.2]
@test StatsBase.span([1, 2, 2, 2, 3]) == 1:3
@test StatsBase.proportions([1, 2, 2, 2, 3], 1:3) == [0.2, 0.6, 0.2]
@test StatsBase.counts([1, 2, 2, 2, 3], 1:3) == [1, 3, 1]
@test StatsBase.addcounts!([1, 3, 1], [1, 2, 2, 2, 3], 1:3) == [1, 3, 1]

It works for simple functional code, but it’ll be tricky to deal with closures, and mutable state. Another option would be to save the trace via JLD. That would require a PR to that package to handle function and module references.

@Per

(Of course, the point of this is not to verify that the code does what it already does, but rather to hunt down bugs by generating tests that trigger them.)

Give compare_past_trace a try! It’s useful to debug regressions.

[Per]

Thanks! This is exactly what I’ve been looking for.

I think, with compare_past_trace and just a few lines of code, I can have a function that takes two git commits and returns a set of tests that will work on one but fail on the other. (I prefer work with runtest.jl instead of trying out code at the REPL, so this fits perfectly into my workflow.)

Regarding mutable arguments: It’d be great if there were an option to recursively copy everything, always, but using a hash table so that multiple copies of the same object are only stored once. (RAM is cheap these days.)

[cstjean]

@Per

Regarding mutable arguments: It’d be great if there were an option to recursively copy everything, always, but using a hash table so that multiple copies of the same object are only stored once. (RAM is cheap these days.)

I don’t think the hash-table thing could work, but you can try playing with TraceCalls.store(x) = deepcopy(x). You’ll probably have problems with functions.

Come to think of it, TraceCalls.store(x) = REPR(x) should work perfectly fine for handling mutable state when building a test set.

To handle closures and other unrepresentable objects, maybe you could try running each test with eval right away within the for trace in collect(...) loop, and if it fails, don’t write it to the file.

With these changes, that looks like a complete solution for auto-generating tests.

@Per

I can have a function that takes two git commits and returns a set of tests that will work on one but fail on the other.

PR welcome! We can add a section on testing.

[krcools]

Are there plans to bring this amazing tool to Julia-1.0?

[cstjean]

Thank you for the kind words. I’d love to bring it back. I may have some time to work on it after the holidays, but it is really not a trivial problem. Any help is appreciated. @pfitzseb showed a promising proof-of-concept on slack, but I haven’t had the time to look into it.