Is there a `with` operator or usage pattern?

The user may not be aware that repeated uses of a property will involve repeated calculation on demand (since this could be an implementation detail of a given type). Cf UnPack.@unpack, for which this is not an issue — variables in the scope are assigned once, and that’s it.

Possibly, I would not speculate. It is generally hard to establish that something is impossible, and I didn’t say it was.

However, I fully understand that people coming from other languages would miss with if they are used to it. I just don’t think this kind of macro meshes well with how Julia works. YMMV.

Oh, I did not think about this. However, this seems to me to be the responsibility of the user, not of the macro. If the user is not aware of this, the macro does nothing special, the user could as well compute the field two times without using the macro. If the user is aware of this, the user will try to mention the field a single time in the @with scope, and this will correctly translate to the computation only happening a single time.

I think it is worth pointing out that DataframesMeta.jl has a @with macro that works with dataframe objects (this is, instead of fields from structs, or named tuples, it considers columns from dataframes). It is a little different, as instead od bindings it replaces Symbols by columns, and have an escape (^) for using Symbol literals inside the block.

If we could generate a function f as below, then you could just splat the NamedTuple into the keyword parameters of f

subtotal = (price=12.99, shipping=3.99)
f(; price, shipping) = price + shipping
f(; subtotal ...) # 16.98

I feel like I’m almost there:

julia> subtotal = (price=12.99, shipping=3.99)
(price = 12.99, shipping = 3.99)

julia> function genfunc(nt, body)
           local props = propertynames(nt)
               f = :( (; $(props...) )-> $body )
           g = Meta.eval(f)
           Base.invokelatest(g; nt...)
       end
genfunc (generic function with 1 method)

julia> genfunc(subtotal, quote
           shipping + price
       end)
16.98

I got it, and I think this addresses @Tamas_Papp’s type inference issues:

module WithMacro

export @with

macro with(nt,body)
    local b = QuoteNode(body)
    quote
        let f = genfunc($(esc(nt)), $b)
            f(; $(esc(nt))...)
        end
    end
end

function genfunc(nt, body)
    local props = propertynames(nt)
    f = :( (; $(props...) )-> $body )
    g = Meta.eval(f)
    # Base.invokelatest(g; nt...)
end

end

Usage:

julia> using .WithMacro

julia> subtotal = (price=12.99, shipping=3.99)
(price = 12.99, shipping = 3.99)

julia> @with subtotal begin
           price + shipping
       end
16.98

julia> person = (first="Mark", last="Kittisopikul")
(first = "Mark", last = "Kittisopikul")

julia> @with person begin
           "$first $last"
       end
"Mark Kittisopikul"

Analysis:

julia> f = WithMacro.genfunc(subtotal,quote
           price + shipping
       end)
#11 (generic function with 1 method)

julia> @code_warntype f(; subtotal...)
Variables
  #unused#::Core.Compiler.Const(Base.Meta.var"#11#13##kw"(), false)
  @_2::NamedTuple{(:price, :shipping),Tuple{Float64,Float64}}
  @_3::Core.Compiler.Const(Base.Meta.var"#11#13"(), false)
  price::Float64
  shipping::Float64
  @_6::Float64
  @_7::Float64

Body::Float64
1 ─ %1  = Base.haskey(@_2, :price)::Core.Compiler.Const(true, false)
│         %1
│         (@_6 = Base.getindex(@_2, :price))
└──       goto #3
2 ─       Core.Compiler.Const(:(Core.UndefKeywordError(:price)), false)
└──       Core.Compiler.Const(:(@_6 = Core.throw(%5)), false)
3 ┄       (price = @_6)
│   %8  = Base.haskey(@_2, :shipping)::Core.Compiler.Const(true, false)
│         %8
│         (@_7 = Base.getindex(@_2, :shipping))
└──       goto #5
4 ─       Core.Compiler.Const(:(Core.UndefKeywordError(:shipping)), false)
└──       Core.Compiler.Const(:(@_7 = Core.throw(%12)), false)
5 ┄       (shipping = @_7)
│   %15 = (:price, :shipping)::Core.Compiler.Const((:price, :shipping), false)
│   %16 = Core.apply_type(Core.NamedTuple, %15)::Core.Compiler.Const(NamedTuple{(:price, :shipping),T} where T<:Tuple, false)
│   %17 = Base.structdiff(@_2, %16)::Core.Compiler.Const(NamedTuple(), false)
│   %18 = Base.pairs(%17)::Core.Compiler.Const(Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}(), false)
│   %19 = Base.isempty(%18)::Core.Compiler.Const(true, false)
│         %19
└──       goto #7
6 ─       Core.Compiler.Const(:(Base.kwerr(@_2, @_3)), false)
7 ┄ %23 = Base.Meta.:(var"#11#12")(price, shipping, @_3)::Float64
└──       return %23

julia> f = WithMacro.genfunc(person,quote
           "$first $last"
       end)
#14 (generic function with 1 method)

julia> @code_warntype f(;person...)
Variables
  #unused#::Core.Compiler.Const(Base.Meta.var"#14#16##kw"(), false)
  @_2::NamedTuple{(:first, :last),Tuple{String,String}}
  @_3::Core.Compiler.Const(Base.Meta.var"#14#16"(), false)
  first::String
  last::String
  @_6::String
  @_7::String

Body::String
1 ─ %1  = Base.haskey(@_2, :first)::Core.Compiler.Const(true, false)
│         %1
│         (@_6 = Base.getindex(@_2, :first))
└──       goto #3
2 ─       Core.Compiler.Const(:(Core.UndefKeywordError(:first)), false)
└──       Core.Compiler.Const(:(@_6 = Core.throw(%5)), false)
3 ┄       (first = @_6)
│   %8  = Base.haskey(@_2, :last)::Core.Compiler.Const(true, false)
│         %8
│         (@_7 = Base.getindex(@_2, :last))
└──       goto #5
4 ─       Core.Compiler.Const(:(Core.UndefKeywordError(:last)), false)
└──       Core.Compiler.Const(:(@_7 = Core.throw(%12)), false)
5 ┄       (last = @_7)
│   %15 = (:first, :last)::Core.Compiler.Const((:first, :last), false)
│   %16 = Core.apply_type(Core.NamedTuple, %15)::Core.Compiler.Const(NamedTuple{(:first, :last),T} where T<:Tuple, false)
│   %17 = Base.structdiff(@_2, %16)::Core.Compiler.Const(NamedTuple(), false)
│   %18 = Base.pairs(%17)::Core.Compiler.Const(Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}(), false)
│   %19 = Base.isempty(%18)::Core.Compiler.Const(true, false)
│         %19
└──       goto #7
6 ─       Core.Compiler.Const(:(Base.kwerr(@_2, @_3)), false)
7 ┄ %23 = Base.Meta.:(var"#14#15")(first, last, @_3)::String
└──       return %23

Note that this doesn’t work at all within a function:

julia> function f(subtotal)
         @with subtotal begin
           price + shipping
         end
       end
f (generic function with 1 method)

julia> f(subtotal)
ERROR: MethodError: no method matching (::Base.Meta.var"#14#16")(; price=12.99, shipping=3.99)
Closest candidates are:
  #14(; price, shipping) at REPL[1]:16
Stacktrace:
 [1] macro expansion at ./REPL[1]:9 [inlined]
 [2] f(::NamedTuple{(:price, :shipping),Tuple{Float64,Float64}}) at ./REPL[8]:2
 [3] top-level scope at REPL[9]:1

and for the same reason it cannot be benchmarked with @btime from BenchmarkTools.

For a much simpler solution which works in local and global scope and imposes no performance penalty at all, see: With struct do - #12 by rdeits

For example:

julia> using MacroTools

julia> using MacroTools: postwalk

julia> macro with(container, expr)
         esc(postwalk(expr) do ex
           if @capture(ex, &field_)
             :(($container).$field)
           else
             ex
           end
         end)
       end
@with (macro with 1 method)

julia> subtotal = (price=12.99, shipping=3.99)
(price = 12.99, shipping = 3.99)

julia> @with subtotal begin
         &price + &shipping
       end
16.98

julia> function f(subtotal)
         @with subtotal begin
           &price + &shipping
         end
       end
f (generic function with 1 method)

julia> f(subtotal)
16.98

julia> using BenchmarkTools

julia> @btime f($subtotal)
  0.029 ns (0 allocations: 0 bytes)
16.98

Note that the suspiciously-fast benchmark time of < 1ns means that the compiler has actually eliminated all of the code in the benchmark and just replaced it with the answer. That’s actually great news for us–it means that the revised @with macro doesn’t make the code any harder for the compiler to analyze and optimize.

4 Likes

It’s also worth noting that you could simply write:

let s = subtotal
  s.price + s.shipping
end

which only requires two more characters but avoids needing any metaprogramming at all. Sometimes the simplest solution is the best :slightly_smiling_face:

10 Likes

Yep. One could also use subtotal.price + subtotal.shipping for what it’s worth. This is a trivial example to ask the question. What I would hope from something like this is that it’d not limit the variables only to the tuple. Secondly, that it could be used in a nested way so that 3-part hierarchical data could be flattened. Third, that I don’t have to use prefixes when there’s no collisions in the field names. Sometimes using prefixes like this make sense, sometimes they get in the way. It depends on the context.

A MWE of what you want to do would be useful.

1 Like

Firstly, this was mostly just a question to learn more about Julia – my knowledge is really variable. It’s always interesting to learn more about how macros work, especially with regard to type stability when processing something in a hard loop.

So, my data source, has 3 levels. I’d like the last level of my operations to be written without having to know exactly where the field came from.

using Faker
orders = [(last_name=Faker.last_name(), region=rand(1:3),
           lines=[(amount=rand(10:20), size=rand(1:10),
                   product= Faker.sentence(number_words=6))
              for j in 1:3]) for i in 1:2]

discount = .10
for o in orders
  let region = o.region, last_name = o.last_name
    for l in o.lines
     let amount = l.amount, product = l.product, size = l.size
       println(last_name, " ", product, " ",   
               (amount * discount) + (region*size))
     end
    end
  end
end 

I was picturing something like… using let but letting the splat syntax pull variable names from a tuple into my current working namespace. This lets the fields in the data structure update where I only have to worry about changing the leaf nodes without updating the various levels of wrapping…

for order in orders
  let order...
    for line in lines
      let line...
        println(last_name, " ", product, " ",   
               (amount * discount) + (region*size))
      end
    end
  end
end 

Anyway. As I noted, I’m not struggling with this. There are plenty of work-arounds. This is a curiosity, since I’ll have to work with lots of situations like this, and writing the code as above seems redundant. I mean, Julia already knows what fields are in my tuple, why should I have to tell it again? Thanks!

StaticModules.jl has a performant @with macro as well.

Though @unpack I think is best since its the most explicit.

2 Likes

I admit, I also kind of like the VB with statement - mostly because it kind of shows the user that your are doing something “with” object MyObjectWithAFairlyLongName.

Taken from the Microsoft web site (HERE):

VB with pattern

Private Sub AddCustomer()
    Dim theCustomer As New Customer

    With theCustomer
        .Name = "Coho Vineyard"
        .URL = "http://www.cohovineyard.com/"
        .City = "Redmond"
    End With

    With theCustomer.Comments
        .Add("First comment.")
        .Add("Second comment.")
    End With
End Sub

Comparable (simple) Julia Pattern

Since, in Julia, you don’t allocate new memory with variables (you basically create a new reference), you can do something fairly similar without any special macros or anything:

function AddCustomer()
    theCustomer= Customer()

    c = theCustomer
        c.Name = "Coho Vineyard"
        c.URL = "http://www.cohovineyard.com/"
        c.City = "Redmond"
    #Done with c = theCustomer

    c = theCustomer.Comments
        add(c, "First comment.")
        add(c, "Second comment.")
    #Done with c = theCustomer.Comments
end #AddCustomer()

Note that:

  • Assignments c.field=X are only possible with mutable struct-ures.
  • If you perform c=theCustomer on an immutable Customer object (for read-only operations), Julia MIGHT perform a copy - but will probably just point to the original data - because it knows it cannot change.

Conclusion

So I probably wouldn’t go to the lengths of using macros when you can just apply this simple pattern.

1 Like

Did not notice @rdeits gave a similar answer as I just did. But his solution is even better because it also limits the scope of the new variable in the same way VB’s with statement does.

3 Likes

I think your problem comes from the fact this little claim “Julia already knows what fields are in my tuple” is not well defined. Inside a macro, Julia, in fact, does not know what fields are in your tuple. The macro are executed before the function is compiled, and it has no information about the values (and therefore the types) of the symbols/expressions passed to them, they only know about the symbols themselves.

This is, a macro takes as arguments the text/“source code” you are passing to it, without any values inside the bindings or anything. It has no way to know the binding var you passed has a value of type T that has fields a, b, and c which it could expand to a, b, c = var.a, var.b, var.c. This is the reason @unpack works the way it works, it needs the user to pass to it the names of the variables, so it knows what to extract from the symbol var that may have any value (or even not be defined) in runtime.

A macro can expand to a call of fieldnames (which return a list of the struct fields) to construct a Dict (mapping the name of a field to the field in the correct struct) in runtime and the macro can also expand the reference to bindings in your expression to queries in such Dict (which would also happen in runtime). The result would be a code many times slower than the one done by hand.

4 Likes

First, I fully agree with everyone here that this kind of thing tends to be a footgun and using UnPack leads to far more readable code.

That said, worth pointing out that while you can’t do this with macros since as said above they only operate on expessions, you can do it quite cleanly with generated functions, which also have access to the type of your object, as well as lowered code to get the scoping right. Here’s a MWE:

using MacroTools: postwalk
@generated function with(func, obj::T) where {T}
    method = Base._methods_by_ftype(Tuple{func}, -1, typemax(UInt64))[1][3]
    code = Base.uncompressed_ast(method).code
    ir_to_expr(i) = postwalk(x -> x isa Core.SSAValue ? ir_to_expr(x.id) : x, code[i])
    postwalk(ir_to_expr(length(code))) do x
        x isa GlobalRef && hasfield(T,x.name) ? :(obj.$(x.name)) : x
    end
end

subtotal = (price=12.99, shipping=3.99)
handling = 0.99

with(subtotal) do
    price + shipping + handling
end # 17.97

And if you check out the generated code, its optimal / inferred:

julia> @code_warntype with(subtotal) do
           price + shipping
       end

Body::Float64
1 ─ %1 = Base.getproperty(obj, :price)::Float64
│   %2 = Base.getproperty(obj, :shipping)::Float64
│   %3 = (%1 + %2)::Float64
└──      return %3

This works in a local scope too as long as you don’t use other local variables, although thats just a limitation of the way I wrote it, you could fix that by doing a better job of modifying the IR code than the simple thing I did above. Someone with more expertise can probably do something even cleaner, maybe with Cassette or IRTools or something, but it was fun to work out this proof of concept.

8 Likes

@MA_Laforge If you update your example to be a complete MWE that defines the structure and uses let, I’d be delighted to switch the “solution” for new Julia developers who are looking for the simple, canonical replacement pattern.

@marius311 – that’s amazing. I’d love a performant implementation that works at multiple levels where prefixes aren’t needed. Given how many attempts at @with there are, it’d be great if one were type stable.

1 Like

One easier possibility to implement efficiently would be to write @with macro where you mark the expressions that are fields, e.g.:

@with subtotal begin
    _.price + _.shipping + _.handling
end
3 Likes

@stevengj So, I’m trying to separate two concerns: (a) where the variables are coming from, with (b) how they are used. Sometimes this is helpful (and sometimes it’s not). In @marius311’s example, the handling variable comes from the global context and is not treated any differently as price and shipping.

subtotal = (price=12.99, shipping=3.99)
handling = 0.99

with(subtotal) do
    price + shipping + handling
end # 17.97

@stevengj that’s exactly what I did above: Is there a `with` operator or usage pattern? - #13 by rdeits

2 Likes

Not certain I fully understand what you are looking for in your book, esp. given the complexity of your example here:

A simple solution

but I’ll try my best to provide an example of what I think you are seeking in a “simple, canonical replacement pattern”:

mutable struct Customer
	name::String
	city::String
	country::String
	additionalInfo::Array{String}
end
Customer() = Customer("", "", "", [])

function generateDummyCustomer()
	newCustomer = Customer()
	let c = newCustomer
		c.name = "Alfred Customer"
		c.city = "Boston"
		c.country = "United States"
	end
	let i = newCustomer.additionalInfo
		push!(i, "Good customer")
		push!(i, "Provides useful feedback")
	end
	return newCustomer
end

Other suggestions for temporary variable

Since I don’t often like using “i” for anything but iterators (or Complex(0,1)), other alternatives are:

  • let info = newCustomer.additionalInfo (Use a succinct/clear alias)
  • let x = newCustomer.additionalInfo (x is usually a good “go-to”)
  • let o = newCustomer.additionalInfo (o is often an acceptable variable for “objects”)

In theory, I also like @stevengj’s idea of using “_” - but I found Julia complains when an “all-underscore identifier” is used on the RHS:

julia> _ = 3; a=4
4

julia> b = _ + a
ERROR: syntax: all-underscore identifier used as rvalue
Stacktrace:
 [1] top-level scope at REPL[11]:1
2 Likes

Note that I changed the example a bit because I’m guessing you don’t want to put in a copy Microsoft code in your book.

I also don’t like that (I can only guess) the example they used probably needs to be manipulating global variables somehow.

  • In VB Private Sub AddCustomer() can’t return a value (last I checked).
  • That means the Customer() constructor would have to store itself somewhere accessible through a form of global variable to be useful.

Then again: I don’t think I’ve programmed in VB in over 10 years (Mostly Julia’s fault).