Is there a `with` operator or usage pattern?

New to Julia
1

In Visual Basic, the With scope operator lets you work with the attributes of a given object without having to repeat the object’s name. Perhaps something like…

julia> subtotal = (price=12.99, shipping=3.99)
(price = 12.99, shipping = 3.99)

julia> with(subtotal) price+shipping; end
16.98

Basically, it’s something akin to NamedTuple’s merge only that it’d work with current lookup space and delegate lookup to a set of objects.

Something like this is helpful when working with database records, so you don’t have to repeat the structure(s) that you’re working with. I’m curious what the normal pattern for something like this might be in Julia; ideally one that could work in nested way.

Anyway, it probably doesn’t exist, or there’s probably someone who wrote a macro that does it. I’m just curious if there are any usage patterns, besides the obvious one – use short variable names.

2

Generally, a macro cannot do this without enumerating the fields since macros transform source ASTs and do not have type information.

I would recommend the excellent Parameters.jl for an idiomatic solution (@unpack), its README has a list of related packages.

2 Likes
3

Thanks Tamas. This works…

julia> let price=12.99, shipping=3.99
          price + shipping
       end
16.98

So, perhaps it could be extended to work with splat?

julia> subtotal = (price=12.99, shipping=3.99)
julia> let subtotal...
          price + shipping
       end
ERROR: syntax: invalid let syntax around REPL[16]:1
1 Like
4

Sounds like https://github.com/joshday/PropertyUtils.jl!

julia> using PropertyUtils

julia> subtotal = (price=12.99, shipping=3.99)
(price = 12.99, shipping = 3.99)

julia> @with subtotal begin
           price+shipping
       end
16.98
7 Likes
5

Note that this solution uses eval, which breaks a lot of things that would otherwise work, eg type inference:

julia> using PropertyUtils

julia> function f(nt)
       @with nt a + b
       end
f (generic function with 1 method)

julia> @code_warntype f((a = 1, b = 2))
Variables
  #self#::Core.Const(f)
  nt::NamedTuple{(:a, :b), Tuple{Int64, Int64}}

Body::Any
1 ─ %1 = PropertyUtils.replace_props!::Core.Const(PropertyUtils.replace_props!)
│   %2 = $(Expr(:copyast, :($(QuoteNode(:(a + b))))))::Expr
│   %3 = (%1)(nt, %2)::Expr
│   %4 = Main.eval(%3)::Any
└──      return %4
4 Likes
6

I am not sure what you mean here. Do you are saying that the macro has no way to difference between struct fields and other external (or new) bindings? Because it seems to me that a macro could do it by just creating getfield calls, if it could assume every reference to a binding is a reference to a field.

7

The macro per se has no way of knowing what subtotal is (a struct, a NamedTuple, etc).

Yes, it could walk the AST and replace all references to variables with getproperty (which the compiler will transform into getfield when applicable), with a clever fallback like PropertyUtils.@with. Then there is the issue of properties calculated on demand.

I am not saying that these could not be dealt with, but it is not as easy as doing the same in R, where “scope” if kind of a first-class object you can manipulate freely.

8

What is the issue? AFAIK properties computed on demand are implemented by specializing getproperty, so it should be transparent to the macro if they are or not computed on demand.

I have neither the knowledge to dispute this claim, nor any doubt it probably is correct, XD. I only wanted to point out that it is not impossible to assemble a solution with a Julia macro that does not interfere with the type inference (or, more generally, need not to invoke eval)

9

The user may not be aware that repeated uses of a property will involve repeated calculation on demand (since this could be an implementation detail of a given type). Cf UnPack.@unpack, for which this is not an issue — variables in the scope are assigned once, and that’s it.

Possibly, I would not speculate. It is generally hard to establish that something is impossible, and I didn’t say it was.

However, I fully understand that people coming from other languages would miss with if they are used to it. I just don’t think this kind of macro meshes well with how Julia works. YMMV.

10

Oh, I did not think about this. However, this seems to me to be the responsibility of the user, not of the macro. If the user is not aware of this, the macro does nothing special, the user could as well compute the field two times without using the macro. If the user is aware of this, the user will try to mention the field a single time in the @with scope, and this will correctly translate to the computation only happening a single time.

I think it is worth pointing out that DataframesMeta.jl has a @with macro that works with dataframe objects (this is, instead of fields from structs, or named tuples, it considers columns from dataframes). It is a little different, as instead od bindings it replaces Symbols by columns, and have an escape (^) for using Symbol literals inside the block.

11

If we could generate a function f as below, then you could just splat the NamedTuple into the keyword parameters of f

subtotal = (price=12.99, shipping=3.99)
f(; price, shipping) = price + shipping
f(; subtotal ...) # 16.98

I feel like I’m almost there:

julia> subtotal = (price=12.99, shipping=3.99)
(price = 12.99, shipping = 3.99)

julia> function genfunc(nt, body)
           local props = propertynames(nt)
               f = :( (; $(props...) )-> $body )
           g = Meta.eval(f)
           Base.invokelatest(g; nt...)
       end
genfunc (generic function with 1 method)

julia> genfunc(subtotal, quote
           shipping + price
       end)
16.98
12

I got it, and I think this addresses @Tamas_Papp’s type inference issues:

module WithMacro

export @with

macro with(nt,body)
    local b = QuoteNode(body)
    quote
        let f = genfunc($(esc(nt)), $b)
            f(; $(esc(nt))...)
        end
    end
end

function genfunc(nt, body)
    local props = propertynames(nt)
    f = :( (; $(props...) )-> $body )
    g = Meta.eval(f)
    # Base.invokelatest(g; nt...)
end

end

Usage:

julia> using .WithMacro

julia> subtotal = (price=12.99, shipping=3.99)
(price = 12.99, shipping = 3.99)

julia> @with subtotal begin
           price + shipping
       end
16.98

julia> person = (first="Mark", last="Kittisopikul")
(first = "Mark", last = "Kittisopikul")

julia> @with person begin
           "$first $last"
       end
"Mark Kittisopikul"

Analysis:

julia> f = WithMacro.genfunc(subtotal,quote
           price + shipping
       end)
#11 (generic function with 1 method)

julia> @code_warntype f(; subtotal...)
Variables
  #unused#::Core.Compiler.Const(Base.Meta.var"#11#13##kw"(), false)
  @_2::NamedTuple{(:price, :shipping),Tuple{Float64,Float64}}
  @_3::Core.Compiler.Const(Base.Meta.var"#11#13"(), false)
  price::Float64
  shipping::Float64
  @_6::Float64
  @_7::Float64

Body::Float64
1 ─ %1  = Base.haskey(@_2, :price)::Core.Compiler.Const(true, false)
│         %1
│         (@_6 = Base.getindex(@_2, :price))
└──       goto #3
2 ─       Core.Compiler.Const(:(Core.UndefKeywordError(:price)), false)
└──       Core.Compiler.Const(:(@_6 = Core.throw(%5)), false)
3 ┄       (price = @_6)
│   %8  = Base.haskey(@_2, :shipping)::Core.Compiler.Const(true, false)
│         %8
│         (@_7 = Base.getindex(@_2, :shipping))
└──       goto #5
4 ─       Core.Compiler.Const(:(Core.UndefKeywordError(:shipping)), false)
└──       Core.Compiler.Const(:(@_7 = Core.throw(%12)), false)
5 ┄       (shipping = @_7)
│   %15 = (:price, :shipping)::Core.Compiler.Const((:price, :shipping), false)
│   %16 = Core.apply_type(Core.NamedTuple, %15)::Core.Compiler.Const(NamedTuple{(:price, :shipping),T} where T<:Tuple, false)
│   %17 = Base.structdiff(@_2, %16)::Core.Compiler.Const(NamedTuple(), false)
│   %18 = Base.pairs(%17)::Core.Compiler.Const(Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}(), false)
│   %19 = Base.isempty(%18)::Core.Compiler.Const(true, false)
│         %19
└──       goto #7
6 ─       Core.Compiler.Const(:(Base.kwerr(@_2, @_3)), false)
7 ┄ %23 = Base.Meta.:(var"#11#12")(price, shipping, @_3)::Float64
└──       return %23

julia> f = WithMacro.genfunc(person,quote
           "$first $last"
       end)
#14 (generic function with 1 method)

julia> @code_warntype f(;person...)
Variables
  #unused#::Core.Compiler.Const(Base.Meta.var"#14#16##kw"(), false)
  @_2::NamedTuple{(:first, :last),Tuple{String,String}}
  @_3::Core.Compiler.Const(Base.Meta.var"#14#16"(), false)
  first::String
  last::String
  @_6::String
  @_7::String

Body::String
1 ─ %1  = Base.haskey(@_2, :first)::Core.Compiler.Const(true, false)
│         %1
│         (@_6 = Base.getindex(@_2, :first))
└──       goto #3
2 ─       Core.Compiler.Const(:(Core.UndefKeywordError(:first)), false)
└──       Core.Compiler.Const(:(@_6 = Core.throw(%5)), false)
3 ┄       (first = @_6)
│   %8  = Base.haskey(@_2, :last)::Core.Compiler.Const(true, false)
│         %8
│         (@_7 = Base.getindex(@_2, :last))
└──       goto #5
4 ─       Core.Compiler.Const(:(Core.UndefKeywordError(:last)), false)
└──       Core.Compiler.Const(:(@_7 = Core.throw(%12)), false)
5 ┄       (last = @_7)
│   %15 = (:first, :last)::Core.Compiler.Const((:first, :last), false)
│   %16 = Core.apply_type(Core.NamedTuple, %15)::Core.Compiler.Const(NamedTuple{(:first, :last),T} where T<:Tuple, false)
│   %17 = Base.structdiff(@_2, %16)::Core.Compiler.Const(NamedTuple(), false)
│   %18 = Base.pairs(%17)::Core.Compiler.Const(Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}(), false)
│   %19 = Base.isempty(%18)::Core.Compiler.Const(true, false)
│         %19
└──       goto #7
6 ─       Core.Compiler.Const(:(Base.kwerr(@_2, @_3)), false)
7 ┄ %23 = Base.Meta.:(var"#14#15")(first, last, @_3)::String
└──       return %23
13

Note that this doesn’t work at all within a function:

julia> function f(subtotal)
         @with subtotal begin
           price + shipping
         end
       end
f (generic function with 1 method)

julia> f(subtotal)
ERROR: MethodError: no method matching (::Base.Meta.var"#14#16")(; price=12.99, shipping=3.99)
Closest candidates are:
  #14(; price, shipping) at REPL[1]:16
Stacktrace:
 [1] macro expansion at ./REPL[1]:9 [inlined]
 [2] f(::NamedTuple{(:price, :shipping),Tuple{Float64,Float64}}) at ./REPL[8]:2
 [3] top-level scope at REPL[9]:1

and for the same reason it cannot be benchmarked with @btime from BenchmarkTools.

For a much simpler solution which works in local and global scope and imposes no performance penalty at all, see: With struct do - #12 by rdeits

For example:

julia> using MacroTools

julia> using MacroTools: postwalk

julia> macro with(container, expr)
         esc(postwalk(expr) do ex
           if @capture(ex, &field_)
             :(($container).$field)
           else
             ex
           end
         end)
       end
@with (macro with 1 method)

julia> subtotal = (price=12.99, shipping=3.99)
(price = 12.99, shipping = 3.99)

julia> @with subtotal begin
         &price + &shipping
       end
16.98

julia> function f(subtotal)
         @with subtotal begin
           &price + &shipping
         end
       end
f (generic function with 1 method)

julia> f(subtotal)
16.98

julia> using BenchmarkTools

julia> @btime f($subtotal)
  0.029 ns (0 allocations: 0 bytes)
16.98

Note that the suspiciously-fast benchmark time of < 1ns means that the compiler has actually eliminated all of the code in the benchmark and just replaced it with the answer. That’s actually great news for us–it means that the revised @with macro doesn’t make the code any harder for the compiler to analyze and optimize.

4 Likes
14

It’s also worth noting that you could simply write:

let s = subtotal
  s.price + s.shipping
end

which only requires two more characters but avoids needing any metaprogramming at all. Sometimes the simplest solution is the best :slightly_smiling_face:

10 Likes
15

Yep. One could also use subtotal.price + subtotal.shipping for what it’s worth. This is a trivial example to ask the question. What I would hope from something like this is that it’d not limit the variables only to the tuple. Secondly, that it could be used in a nested way so that 3-part hierarchical data could be flattened. Third, that I don’t have to use prefixes when there’s no collisions in the field names. Sometimes using prefixes like this make sense, sometimes they get in the way. It depends on the context.

16

A MWE of what you want to do would be useful.

1 Like
17

Firstly, this was mostly just a question to learn more about Julia – my knowledge is really variable. It’s always interesting to learn more about how macros work, especially with regard to type stability when processing something in a hard loop.

So, my data source, has 3 levels. I’d like the last level of my operations to be written without having to know exactly where the field came from.

using Faker
orders = [(last_name=Faker.last_name(), region=rand(1:3),
           lines=[(amount=rand(10:20), size=rand(1:10),
                   product= Faker.sentence(number_words=6))
              for j in 1:3]) for i in 1:2]

discount = .10
for o in orders
  let region = o.region, last_name = o.last_name
    for l in o.lines
     let amount = l.amount, product = l.product, size = l.size
       println(last_name, " ", product, " ",   
               (amount * discount) + (region*size))
     end
    end
  end
end

I was picturing something like… using let but letting the splat syntax pull variable names from a tuple into my current working namespace. This lets the fields in the data structure update where I only have to worry about changing the leaf nodes without updating the various levels of wrapping…

for order in orders
  let order...
    for line in lines
      let line...
        println(last_name, " ", product, " ",   
               (amount * discount) + (region*size))
      end
    end
  end
end

Anyway. As I noted, I’m not struggling with this. There are plenty of work-arounds. This is a curiosity, since I’ll have to work with lots of situations like this, and writing the code as above seems redundant. I mean, Julia already knows what fields are in my tuple, why should I have to tell it again? Thanks!

18

StaticModules.jl has a performant @with macro as well.

Though @unpack I think is best since its the most explicit.

2 Likes
19

I admit, I also kind of like the VB with statement - mostly because it kind of shows the user that your are doing something “with” object MyObjectWithAFairlyLongName.

Taken from the Microsoft web site (HERE):

VB with pattern

Private Sub AddCustomer()
    Dim theCustomer As New Customer

    With theCustomer
        .Name = "Coho Vineyard"
        .URL = "http://www.cohovineyard.com/"
        .City = "Redmond"
    End With

    With theCustomer.Comments
        .Add("First comment.")
        .Add("Second comment.")
    End With
End Sub

Comparable (simple) Julia Pattern

Since, in Julia, you don’t allocate new memory with variables (you basically create a new reference), you can do something fairly similar without any special macros or anything:

function AddCustomer()
    theCustomer= Customer()

    c = theCustomer
        c.Name = "Coho Vineyard"
        c.URL = "http://www.cohovineyard.com/"
        c.City = "Redmond"
    #Done with c = theCustomer

    c = theCustomer.Comments
        add(c, "First comment.")
        add(c, "Second comment.")
    #Done with c = theCustomer.Comments
end #AddCustomer()

Note that:

  • Assignments c.field=X are only possible with mutable struct-ures.
  • If you perform c=theCustomer on an immutable Customer object (for read-only operations), Julia MIGHT perform a copy - but will probably just point to the original data - because it knows it cannot change.

Conclusion

So I probably wouldn’t go to the lengths of using macros when you can just apply this simple pattern.

1 Like
20

Did not notice @rdeits gave a similar answer as I just did. But his solution is even better because it also limits the scope of the new variable in the same way VB’s with statement does.

3 Likes