Modules, macros and global variables

General Usage
1

Let’s say, I have:

  1. Module A with global variable RULES, macro @add_rule and function apply_all_rules().
  2. Module B which defines some rules using A.@add_rule thus pushing them to A.RULES.
  3. Module C in which we simply import B.

The behavior I want: once we call import B, all rules defined in B are added to A.RULES so that A.apply_all_rules() can see them.

Things that make it more complicated:

  1. Module precompilation time != module load time (although I can use __init__() to control this).
  2. Macro expansion time != run time.
  3. Global variables smell.

I have a concrete implementation of this strategy that doesn’t work, but before diving into it I’d like to know if somebody solved similar problems in a more elegant way.

2

For 1, you have __init__ as you say.
For 2, the macro expansion shouldn’t as a side effect add anything to RULES, it should expand to code that adds the rule. This code is run during __init__.
For 3, there is still only the RULES global variable, no?

Perhaps if you give a MWE it would be easier to see the concrete problem.

2 Likes
3

I was afraid you’d say this :slight_smile: I’ll try to extract relevant parts of actual implementation preserving all the relevant details and come back to you!

4

While trying to make an MWE I actually fixed the issue. Not sure what was the reason but in addition to __init__() I did 2 more important things:

  1. Made sure I actually use latest versions of A and B. In previous attempts I called up A from pkg console of B, but didn’t actually commit changes to A, so at least in some tests I simply used wrong version of the package.
  2. Replaced push!($RULES, rule) with a dedicated function. Previously I had:
module A

const RULES = []

macro add_rule(rule)
    return esc(quote
       # some magic in a caller module for which we need to escape the whole expression
       push!($RULES, rule)
    end)
end

end

I expected that $RULES will be expanded into actual object and pushing to this object will work regardless of a calling module. For whatever reason it didn’t work, but the following did:

module A

const RULES = []

add_rule_func(rule) = push!(RULES, rule)

macro add_rule(rule)
    return esc(quote
       # some magic in a caller module for which we need to escape the whole expression
       $add_rule_func(rule)   # <-- this line changed
    end)
end

end
5

Just for my own understanding: Why do you need a macro at all to add rules to the RULES vector?

I.e., why not directly call the add_rule function everywhere?

6
  1. It gives you convenient syntax. For an example of such rules take a look here.
  2. It gives you better control of execution. For example, as a first line of actual implementation of the mentioned macro I calculate the calling module (B in example above). Using function you would only be able to pass the module explicitly which would be quite verbose.
7

Thanks for clarifying.

8

Does it really work? Unless you are calling @add_rule inside B’s __init__, I don’t think rules defined in B is available inside C (even after you do using B). This is because, IIUC, @add_rule is mutating data in A which is not saved in the precompilation cache of B. Of course, it’s OK if A, B, and C are all submodules of a same package. Otherwise, I think you’d need to define B.RULES and do append!(A.RULES, RULES) in B.__init__. Alternatively, maybe you can define rules as methods.

1 Like
9

Yes, so I indeed call @add_rule in B.__init__(). Thanks for adding the explanation why it’s needed, I think it’ll help other with similar needs!

Generally it’s a great option too, something that Julia is truly optimized for. But in my specific case I do a lot of symbolic manipulation which is hard to express via types and multimethods.

10

I see that you are defining differentiation rules. I think I agree. My impression is that it’s challenging to support (say) broadcasting in the way ChainRules.jl works (i.e., using methods) because callers cannot “see” the definition. But I’ve been wondering if it’s possible to use some kind of more structured “thunks” in ChainRules.jl, maybe based on LazyArrays.jl, so that symbolic manipulations across differentiation rules is possible.

BTW, if you don’t want to use add_rule_func, I think this should work as well:

macro add_rule(rule)
    m = @__MODULE__
    return esc(quote
       # some magic in a caller module for which we need to escape the whole expression
       push!($m.RULES, rule)
    end)
end
1 Like