Mixin and parameter packages: How many macros is too many macros?

[Raf]

I’ve just put together a couple of simple packages for dealing with model parameters, as I deal with a lot of them… but I find myself using them all the time now, so maybe they are generally useful. But Julia lets you do basically anything, and I’m not sure yet if they are totally sane. So feedback would be great before I use them everywhere.

This is a working example of both - a declaration of three interchangable types with parameters for the three modes of an arrhenius type temperature function:

using Parameters, Unitful, Mixers, MetaParameters

@metaparam describe ""
@metaparam paramrange [300.0u"K", 400.0u"K"]

abstract type AbstractTempCorr end

@mix tbase{T} begin
    reftemp::T    = 310.0u"K"   | [273.0u"K", 325.0u"K"]     | "Reference temperature for all rate parameters"
    arrtemp::T    = 2000.0u"K"  | [200.0u"K", 4000.0u"K"]    | "Arrhenius temperature"
end
@mix tlow{T} begin
    lowerbound::T = 280.0u"K"   | [273.0u"K", 325.0u"K"]     | "Lower boundary of tolerance range"
    arrlower::T   = 20000.0u"K" | [2000.0u"K", 40000.0u"K"]  | "Arrhenius temperature for lower boundary"
end
@mix tup{T} begin
    upperbound::T = 315.0u"K"   | [273.0u"K", 325.0u"K"]     | "Upper boundary of tolerance range"
    arrupper::T   = 70000.0u"K" | [7000.0u"K", 140000.0u"K"] | "Arrhenius temperature for upper boundary"
end

@tbase @describe @paramrange @with_kw mutable struct TempCorr{} <: AbstractTempCorr end
@tbase @tlow @describe @paramrange @with_kw mutable struct TempCorrLower{} <: AbstractTempCorr end
@tbase @tlow @tup @describe @paramrange @with_kw mutable struct TempCorrLowerUpper{} <: AbstractTempCorr end

julia> fieldnames(TempCorrLower)                            
4-element Array{Symbol,1}:                                  
 :reftemp                                            
 :arrtemp                                                 
 :lowerbound
 :arrlower                                                
                   
julia> fieldnames(TempCorrLowerUpper)
6-element Array{Symbol,1}:                                  
 :reftemp                                            
 :arrtemp                                                 
 :lowerbound
 :arrlower                                                
 :upperbound
 :arrupper                                                
                                                            
julia> describe(TempCorrLowerUpper(), :arrupper)          
"Arrhenius temperature for upper boundary"                  
                                                            
julia> paramrange(TempCorrLowerUpper(), :arrupper)        
2-element Array{Unitful.Quantity{Float64,Unitful.Dimensions{(Unitful.Dimension{:Temperature}(1//1),)},Unitful.FreeUnits{(Unitful.Unit{:Kelvin,Unitful.Dimensions{(Unitful.Dimension{:Temperature}(1//1),)}}(0, 1//1),),Unitful.Dimensions{(Unitful.Dimension{:Temperature}(1//1),)}}},1}:
   7000.0 K                                                 
 140000.0 K  

It looks crazy with that many macros. But everything has a range, description and default value on the same line without duplication or much boilerplate besides the macros. Mixers.jl could accept macros and take the union as it does parametric types, and even those would be DRY! But maybe that is even crazier than using that many macros in the first place.

Mixers.jl provides the composable mixins. They include both parametric types and fields so you can just chain them together arbitrarily to add fields to a struct.

Mostly this is intended to reduce code duplication in type composition and provide shared data layout decoupled from the type hierarchy. Including parametric types in the mixin makes it much more succinct and general than just a @def macro.

MetaParameters.jl adds the “metaparameters” to struct fields that are retrieved with a function of the same name as the custom macro. I used to build big structs to pass these kind of things around but they are only occasionally needed and are immutable, so why clutter my models with them? It’s not registered yet, I’m not sure if it should be…

[chakravala]

My package Reduce.jl also uses a lot of chained macros. Chaining macros is actually kind of nice, because you can write a method that rewrites the whole chain into a single command. That’s what macroshift is for

https://github.com/chakravala/Reduce.jl/blob/42399557faf6361fa294e8c45c3ac622022f7f03/src/switch.jl#L113-L126

Essentially, it can reduce a chained macro call of REDUCE switch names into a single evaluation. e.g.

julia> using Reduce
Reduce (Free CSL version, revision 4521), 11-Mar-18 ...

julia> @rounded @factor x^3-2x+1
:((x + 1.61803398875) * (x - 1) * (x - 0.61803398875))

is automatically rewritten into

julia> rcall(:(x^3-2x+1),:rounded,:factor)
:((x + 1.61803398875) * (x - 1) * (x - 0.61803398875))

so it’s not actually exectued as a chain, but rewritten into a single command with any number of switch names.

The actual macro definition looks like this,

https://github.com/chakravala/Reduce.jl/blob/42399557faf6361fa294e8c45c3ac622022f7f03/src/switch.jl#L100-L104

Therefore, I believe that chained macros can be quite useful in Julia, if used efficiently.

[Raf]

Ok so those macros do’t necessarily run, just work as flags. That’s interesting.

I was thinking of something similar so you can mash @mix macros together arbitrarily, and add macros after @mix that are only run later on the actual structs.

Do you do any profiling of compile time when your using tons of macros?

[chakravala]

@Raf

Do you do any profiling of compile time when your using tons of macros?

Not sure I completely understand the question. The macro definition is small, so the compile time is not so significant. Most of the compile time goes into the parser and other functionality.

I do think running the macro is a tiny bit slower than directly evaluating the command, but that has nothing to do with compilation.

[Raf]

Sure, I meant first run JIT compliation if you have hundreds of chained macros doing tree walks etc changing the code. But I guess it’s not going to add up to much.

I’ve taken on the “don’t use macros everywhere” dictum but I’m not sure of the real reasons for it, except maybe that we’ll end up with a lisp style custom syntax mess.

But chained macros seem to have endless uses…

[mauro3]

On the plus side: Many macros are useful to write code more quickly by reducing boiler-plate, abbreviate, etc. It may make the code more maintainable, by avoiding repetitions which would need to be updated in sync during a refactor. (Although, I think, that problem can often be solved with more careful code design.)

On the negative side, it makes your code much harder to read, for other people or your future self. This will decrease contributions from others, as they need to learn your meta-language first, and may impede the maintainability of your code by yourself.

So, for long term projects or projects where you expect/want outside contributors, I think limiting macro usage probably makes sense. This conversation from another thread between @osofr and Mike Innes (one of our top macro-gurus) illustrates my point nicely:

@osofr

Mike’s implementation is a thing of beauty, some of his Julia code is just on another level. For example, take a look at the Flux implementation of backprop with Adam optimizer – there is so much Julia magic there it hurts. Sometimes the source code can be very hard to understand (for a newbie), but its a very rewarding experience when you eventually get it.

gets the reply

@MikeInnes

Optimisers are actually the part of Flux’s interface that I’m least happy with right now (which is largely why they are not documented beyond basic usage). I’m not happy that they are magical and want to redesign something that’s simple and powerful.

[Raf]

Yes its a real balance. In this case I’m trying to keep things together that should be written together but not stored together. But the macros are pushing the code into that weird magical territory you’re talking about…

Reading those threads got me thinking that aiming for macro behaviour to feel as much like regular julia is more important than wether they are less magical in reality! I could just make the mixin look exactly like a struct with struct ... end syntax and a capitalised naming convention. Then adding parametric types and the @with_kw macro would look right at home.

[Raf]

I.e this is way more magical, but looks way less magical (except the first line)

@chaingang mixpars @mix @describe @paramrange @with_kw

@mixpars struct TBase{T} 
    reftemp::T    = 310.0u"K"   | [273.0u"K", 325.0u"K"]     | "Reference temperature for all rate parameters"
    arrtemp::T    = 2000.0u"K"  | [200.0u"K", 4000.0u"K"]    | "Arrhenius temperature"
end
@mixpars struct TLow{T}
    lowerbound::T = 280.0u"K"   | [273.0u"K", 325.0u"K"]     | "Lower boundary of tolerance range"
    arrlower::T   = 20000.0u"K" | [2000.0u"K", 40000.0u"K"]  | "Arrhenius temperature for lower boundary"
end
@mixpars struct TUp{T}
    upperbound::T = 315.0u"K"   | [273.0u"K", 325.0u"K"]     | "Upper boundary of tolerance range"
    arrupper::T   = 70000.0u"K" | [7000.0u"K", 140000.0u"K"] | "Arrhenius temperature for upper boundary"
end

@TBase mutable struct TempCorr{} <: AbstractTempCorr end
@Tbase @TLow mutable struct TempCorrLower{} <: AbstractTempCorr end
@Tbase @Tlow @Tup  mutable struct TempCorrLowerUpper{} <: AbstractTempCorr end

Guess I have to implement that now…

[ScottPJones]

@mauro3

On the plus side: Many macros are useful to write code more quickly by reducing boiler-plate, abbreviate, etc. It may make the code more maintainable, by avoiding repetitions which would need to be updated in sync during a refactor.

Agree totally here. Macros (in moderation) can really make life easier (although writing them can be a pain at times, when you are dealing with macros that have to evaluate things in different modules)
Things like @enum, @reexport, @def, traits macros (thanks, @mauro3!) and logging macros are quite useful.
I like them more for simple, low level things, like that, not so much for inventing a whole new language.
I’m adding a macro for defining an API, @api.

@mauro3

(Although, I think, that problem can often be solved with more careful code design.)

In some cases, maybe, but I’ve found frequently it’s the only way to get things to happen at compile-time and be readable (string macros for formatting, regex support, or character set encodings).

@mauro3

On the negative side, it makes your code much harder to read, for other people or your future self. This will decrease contributions from others, as they need to learn your meta-language first, and may impede the maintainability of your code by yourself.

Very true, a handful of macros can go a long way, more than that, yes, you end up learning a new DSL.

[chakravala]

In my case, the package I am working on is literally a parser for another programming language, so making the features of the other language usable in Julia is part of what the package is supposed to do, so yea you are going to end up with some new DSL-like features. Documentation will be increased in future by the time 1.0 is released, it is still a work in progress. The other thing is that this package will actually be used to build DSL’s in other Julia packages too, so it’s all a part of the end goal. However, the details need to be settled / stable before I spend a whole bunch of time writing the docs for it.

[Raf]

I find the @mix macro in Mixers.jl to be more useful than standard @def for working with structs, I wrote it after getting sick of editing the parametric types manually for @def macros. But it is adding one more bit of DSL syntax.

I’m just writing another tiny package now for DRY chained macros so you can use heaps of them, but it looks like you’re still being a minimalist…