Nested Macros Bypass Package Dependencies

Internals & Design
1

Just ran into an issue in the DiffEq-sphere that seems like it might be a more systemic problem. I am wondering if there is a proper way to address it, or if this needs to be addressed in Julia itself. The issue is here:

https://github.com/JuliaDiffEq/StochasticDiffEq.jl/issues/16#issuecomment-286812619

but the problem is summed up as this. To define a type, I am using Parameters.jl. For example:

@with_kw immutable EM{RSType} <: StochasticDiffEqAlgorithm
  rswm::RSType = RSWM(adaptivealg=:RSwM1)
end

the @with_kw makes the constructor have a keyword argument as you’d expect from this definition, and all is well. However, last night a release was put in to Parameters.jl which updated it to use v0.6. Now, users suddenly get an error:

ERROR: LoadError: LoadError: UndefVarError: @compat not defined
 in macro expansion; at ./none:2 [inlined]
 in anonymous at ./<missing>:?
while loading /home/blegat/.julia/v0.5/StochasticDiffEq/src/algorithms.jl, in expression starting on line 5
while loading /home/blegat/.julia/v0.5/StochasticDiffEq/src/StochasticDiffEq.jl, in expression starting on line 21
ERROR: Failed to precompile StochasticDiffEq to /home/blegat/.julia/lib/v0.5/StochasticDiffEq.ji.
 in eval_user_input(::Any, ::Base.REPL.REPLBackend) at ./REPL.jl:64
 in macro expansion at ./REPL.jl:95 [inlined]
 in (::Base.REPL.##3#4{Base.REPL.REPLBackend})() at ./event.jl:68

That means that Compat isn’t in the modules scope. But it was added to Parameters.jl’s module scope in the recent change. The issue is that the requirement for the dependency leaked out:

https://github.com/mauro3/Parameters.jl/compare/v0.6.0...master#diff-ec1155cc07b8ae5fc6787c9cb23a7986R359

The @with_kw macro now puts in an @compat. So when that expands, it ends up with an expression which has @compat in it. So it seems that StochasticDiffEq.jl now needs a requirement and needs to pull @compat into its scope.

Someone probably has a good minimum example for this, I’m mostly frantically trying to fix this right now. I am sure that it’s a leaking of the Compat dependency since I added using Compat to StochasticDiffEq.jl master, but also I can fix release’s “you need Compat” bug by pinning Parameters.jl to v0.6.0.

Is there a way that we should be using nested macros to avoid this problem? Or should Julia be using the scope of where the macro was defined to handle nested macros?

2

I didn’t look at the source but the macro must not escape anything that it wants to be resolved in the callee (macro) module.

2 Likes
3

Yes, this is likely a misplaced esc.

julia> module M
       using Compat
       macro foo(x)
           :(@compat $(esc(x)))
       end
       macro bar(x)
           esc(:(@compat $x))
       end
       end
M

julia> M.@foo 1
1

julia> M.@bar 1
ERROR: UndefVarError: @compat not defined
1 Like
4

A quick search also return other wrong escapes.

https://github.com/mauro3/Parameters.jl/blob/082972d2886165353a9ee6ff845c9cb1e73d872d/src/Parameters.jl#L212

is wrong.

The simplest solution if you want to escape everything would be to splice in the module i.e. $Parameters instead of Main.Parameters. You can do the same with Compat.

5

And yeah, this escape is wrong.

6

Thanks to all for looking into this! I find the escaping is not the easiest to graps… I’ll fix it tomorrow.

7

Thank you so much for explaining the issue with a nice small example of how to fix it. I was scratching my head as to what to do other than add Compat to each package! I’m putting in some version bounds on the DiffEq dependencies of Parameters as a temporary fix, and will help @mauro3 fix those escapes.

As for other dependencies. This is likely causing an issue in CMakeWrapper.jl (@rdeits) and SciKitLearn.jl (@cstjean). You might want to test and add version bounds.

8

Thanks for the help everyone. We’ll work on Parameters.jl. In the meantime, I put in a PR for upper bounds which will fix the released versions of packages.

For DiffEq:

https://github.com/JuliaLang/METADATA.jl/pull/8343

For others (separated so that way the authors can be notified and delay if needed):

https://github.com/JuliaLang/METADATA.jl/pull/8344

9

You mean, change @compat to $Compat.@compat?

Which line are you referring to?

10

Yes.

Line 466, which you can see in the URL… in macro with_kw. Apparently github doesn’t automatically expand enough content when the link is to a line that’s collapsed by default…

11

https://github.com/JuliaLang/METADATA.jl/pull/8348

Merged within 5min by @tkelman!

Thanks to all!

3 Likes
12

For those following along at home: Turns out that the @compat was not needed at all as the inner-constructor syntax I used

julia> type A{T}
       a::T
       (::Type{A{T}}){T}(a::T) = new{T}(a)
       end

is supported on 0.5 anyway. Also, carelessly adding the @compat lead to this issue: https://github.com/mauro3/Parameters.jl/issues/29. Tagged here: https://github.com/JuliaLang/METADATA.jl/pull/8363