Using_merge.jl

Package Announcements
1

using_merge.jl

The wish for this started in
this thread.
At the time I was very new to Julia and did not think I could do anything myself about the problem. Now two years later, knowing better Julia, I realized I can do something. Here it is.

I introduce the problem with an example.

In my big Gapjm package (a port of some GAP libraries to Julia) I have a function invariants which computes the invariants of a finite reflection group. However when I use BenchmarkTools to debug for performance my package, I have the following problem:

julia> G= # some group...

julia> invariants(G)
WARNING: both Gapjm and BenchmarkTools export "invariants"; uses of it in module Main must be qualified
ERROR: UndefVarError: invariants not defined
Stacktrace:
 [1] top-level scope at REPL[4]:1
 [2] eval(::Module, ::Any) at ./boot.jl:331
 [3] eval_user_input(::Any, ::REPL.REPLBackend) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/REPL/src/REPL.jl:86
 [4] run_backend(::REPL.REPLBackend) at /home/jmichel/.julia/packages/Revise/kqqw8/src/Revise.jl:1163
 [5] top-level scope at none:0

This is annoying! I do not want to have to qualify every call to invariants just because I am timing my code! What can I do? Well, first I could just import the methods I am using in BenchmarkTools:

julia> using BenchmarkTools: @btime

Actually, every exported name from BenchmarkTools, except invariants, does not conflict with my code:

julia> names(BenchmarkTools)
30-element Array{Symbol,1}:
 Symbol("@ballocated")
 Symbol("@belapsed")
 Symbol("@benchmark")
 Symbol("@benchmarkable")
 Symbol("@btime")
 Symbol("@tagged")
 :BenchmarkGroup
 :BenchmarkTools
 :addgroup!
 :allocs
 :gctime
 :improvements
 :invariants
 :isimprovement
 :isinvariant
 :isregression
 :judge
 :leaves
 :loadparams!
 :mean
 :median
 :memory
 :params
 :ratio
 :regressions
 :rmskew
 :rmskew!
 :trim
 :tune!
 :warmup

so I can do:

julia> using BenchmarkTools: @ballocated, @belapsed, @benchmark, @benchmarkable, @btime, @tagged, BenchmarkGroup, BenchmarkTools, addgroup!, allocs, gctime, improvements, isimprovement, isinvariant, isregression, judge, leaves, loadparams!, mean, median, memory, params, ratio, regressions, rmskew, rmskew!, trim, tune!,
 warmup

Still no conflict. Can I go further and do something even for invariants?
Well, I have one method for invariants in my package:

invariants(a::Group, args...)

while BenchmarkTools has four:

invariants(group::BenchmarkGroup)
invariants(x)
invariants(f, group::BenchmarkGroup)
invariants(f, x)

Even though some of these last methods apply to Any, they do not conflict with my method, so I can use them also by just defining:

invariants(group::BenchmarkGroup) = BenchmarkTools.invariants(group)
invariants(x) = BenchmarkTools.invariants(x)
invariants(f, group::BenchmarkGroup) = BenchmarkTools.invariants(f, group)
invariants(f, x) = BenchmarkTools.invariants(f, x)

I call the end result of the above process merging the package BenchmarkTools with my current package. What I announce here is a function using_merge which does all the above automatically. If you include the text of the function which is in using_merge.jl and then call

julia> using_merge(:BenchmarkTools)

The function determines conflicting method and macro names in the package and merges them as above, and uses the non-conflicting ones.

You will find using_merge.jl in the src directory at

This is just a function (not a module or a package) because:

  • a simple function can do the job, and the function uses eval which needs to eval in the current module, so I find it easier to just include the function rather than to find how to eval in the calling module.

  • I wait for feedback (which I hope will come) before thinking how to package the thing in a Julia package (or not), and whether I should provide a macro rather than a function. Also my implementation is perhaps not the best, I kind of parse the output of methods.

  • I have not yet solved all technical problems, see below.

The function has two optional keyword arguments:

julia> using_merge(:BenchmarkTools;reexport=true)

will reexport all non-conflicting names.

julia> using_merge(:BenchmarkTools;debug=1)

will print all evaled statements, and debug=2 will describe even more verbosely the taken actions.

Since I wrote this function, I found that I got the hoped for modularity benefits in my code. For example, I have in my Gapjm.jl package modules

  • Perms Permutations
  • Cycs Cyclotomic numbers (sums of complex roots of unity)
  • Pols Univariate Laurent polynomials
  • Mvps.jl Multivariate Puisuex polynomials
  • Posets.jl Posets
  • FFields.jl Finite fields

that I designed as independent, stand-alone packages which each can be used without importing anything else from my package. To use them together I can now just using_merge each of them instead of writing (unpleasant) glue code.

I do not advocate always replacing the semantics of using with that of using_merge, but I feel that using_merge is a nice tool for using packages together without having to write glue code (and without having to modify any of the used packages). The meaning of “pirating a type” becomes
a little bit wider in this context, as you saw with the method invariants(y) in BenchmarkTools: it is, I would say, polite, if any of your methods which has a possibly conflicting name uses at least one of your own types in its signature.

There are two technical problems left (that I know of – there may be others I am not aware):

  • I do not know how to implement the above scheme for conflicting macros. For example, you can see that the signature of @btime is:
julia> methods(eval(Symbol("@btime")))
# 1 method for macro "@btime":
[1] @btime(__source__::LineNumberNode, __module__::Module, args...)
...

I do not know how to write a macro btime which forwards this declaration to the current module.

  • I do not know how to forward docstrings. With the declaration
invariants(x) = BenchmarkTools.invariants(x)

a docstring of BenchmarkTools.invariants is not attached to the new definition, so ?invariants will not show the docstrings. You can still see them by ?BenchmarkTools.invariants but they were not merged together with the definitions, which partially defeats my goal. If someone tells me
how I could fix the above problems, I would be grateful.

2 Likes
2

While I am very sympathetic to non-pirate function punning in many cases, I don’t think a function with eval is the way to go. Among other things, it might make static compilation, etc. impossible.

@chakravala built https://github.com/chakravala/ForceImport.jl a while ago, which seemed a better architecture (though perhaps it should be revisited given recent advancements).

If a @merge using or something like that became easy to use, it would be fascinating to see if it is used in innocuous situations - or cause trouble. That could help give feedback on whether easier method merging should be considered for Julia 2 (e.g. maybe formalize automatic method merging with warning/linting for dangerous piracy and/or add a keyword/macro in the language itself).

1 Like
3

Hello,
I, of course, looked at ForceImport.jl. The documentation is not sufficient for me to be sure what it does exactly, but:

  • when I use it, it does not do what I need: for example, in my example, it does not add the methods of invariants to my methods, but replaces my methods with those of BenchmarkTools (my methods are only accessible then as Gapjm.invariants). I do not see any code trying to discern conflicts.

  • I looked at what the macro @force does with macroexpand and it does do an eval. I do
    not know if one can do otherwise.

4

I am no metaprogramming expert but I believe due to scoping the eval in a macro is way better than in a function (world age and whatnot). But I could easily be wrong.