# Memoize Pkg? Bad use of Metaprogramming?

**URL:** <https://discourse.julialang.org/t/memoize-pkg-bad-use-of-metaprogramming/46874>\
**Category:** General Usage\
**Tags:** memoize\
**Created:** [September 18, 2020, 8:48pm UTC](https://discourse.julialang.org/t/memoize-pkg-bad-use-of-metaprogramming/46874 "2020-09-18T20:48:05Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![anon92994695](https://avatars.discourse-cdn.com/v4/letter/a/ce7236/32.png) [@anon92994695](https://discourse.julialang.org/u/anon92994695)\
**Post date:** [September 18, 2020, 8:48pm UTC](https://discourse.julialang.org/t/memoize-pkg-bad-use-of-metaprogramming/46874/1 "2020-09-18T20:48:05Z")

</div>

Been doing some long-running computations. I mean … long. So I am doing the obvious thing, storing results to disk along the way. For reproducibility reasons I’d like to be able to memoize lots of intermediate results along the way. Even better, if I could show the code to generate the results but not have to re-run it or have it isolated from the execution flow. So I came up with a solution, it’s something like I once did with @kolia a while ago, but I’d like some feedback.

```nohighlight
using JLD2
macro memoize(cache_var)
    filename = String(cache_var) * ".jld2"
    if !isfile( filename )
        @eval @save $filename $cache_var
    else
        @eval @load $filename $cache_var
    end
end

macro memoize(cache_var, fn)
    filename = String(cache_var) * ".jld2"
    if !isfile( filename )
        @eval $fn
        @eval @save $filename $cache_var
    else
        @eval @load $filename $cache_var
    end
end

macro free_memo(cache_var)
    filename = String(cache_var) * ".jld2"
    if !isfile( filename )
        @warn "Cache is already empty."
    else
        rm(filename)
    end
end
@memoize a
a

@memoize a begin
    a = 123
    a += 456
    a /= 5
    println("Hey we ran this code!")
end

@free_memo a

```

Main points:

- is there already a package that does this? if so link please?
- Bad use of metaprogramming?
- Too dangerous to bundle up, correct, and share (incorrect use could cost a user a lot of time or electrons)?

Feel free to dartboard anything else bothersome. The word “cache” will likely upset some people here, and I foresee concerns about scope and where files are stored :D.

---

<div class="post-metadata">

**Author:** ![rdeits](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rdeits/32/286_2.png) [@rdeits](https://discourse.julialang.org/u/rdeits)\
**Post date:** [September 18, 2020, 9:11pm UTC](https://discourse.julialang.org/t/memoize-pkg-bad-use-of-metaprogramming/46874/2 "2020-09-18T21:11:16Z")

</div>

> [@anon92994695](#):
>
> Bad use of metaprogramming?

I’m afraid so. A macro body should:

1. Never use `eval()`
2. Never have side effects

Your macro does both. It’s important to understand that the body of the macro is run during the compilation process (when the code is lowered from the parser output). That means that your macro is going to try to look for files, run functions, create files, and delete files all _during compilation_. There’s little to no chance that’s actually what you want.

The good news is: this is easy to fix. You don’t need a macro at all! Just turn your macros into functions and then their effects will happen exactly when you’d expect (when the function is run).

---

<div class="post-metadata">

**Author:** ![rdeits](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rdeits/32/286_2.png) [@rdeits](https://discourse.julialang.org/u/rdeits)\
**Post date:** [September 18, 2020, 9:21pm UTC](https://discourse.julialang.org/t/memoize-pkg-bad-use-of-metaprogramming/46874/3 "2020-09-18T21:21:47Z")

</div>

I should add: if you really want to be able to use the shorthand syntax of something like:

```julia
@memoize a begin...end

```

instead of a more functional form like `a = memoize("a") do .... end` then it’s totally appropriate to use a macro. The key is that you need to make sure that the macro _produces code that does the work_ rather than trying to do the work itself.

For example, here’s a badly written macro that tries to create a variable named `a` by calling a function `f`:

```julia
julia> macro make_a()
         @eval a = f()
       end

```

This is deceptive, because it _kind of_ works in global scope:

```julia
julia> f() = 1
f (generic function with 1 method)

julia> @make_a()
1

julia> a
1

```

But on closer inspection, it falls apart.

First of all, the macro does the wrong thing in a local scope, since `@eval` creates a new _global_ variable named `a`. For example (in a new Julia session):

```julia
julia> macro make_a()
         @eval a = f()
       end
@make_a (macro with 1 method)

julia> f() = 2
f (generic function with 1 method)

julia> function foo()
         @make_a()
         println("a = ", a)
       end
foo (generic function with 1 method)

julia> foo()
a = 2

julia> a
2

```

The `a` variable has “leaked” out of the function `f()`, which is definitely not good.

Furthermore, the macro doesn’t even work depending on which order we define our functions. Try this in a new Julia session:

```julia
julia> macro make_a()
         @eval a = f()
       end
@make_a (macro with 1 method)

julia> function foo()
         @make_a()
         println("a = ", a)
       end
ERROR: LoadError: UndefVarError: f not defined
Stacktrace:
 [1] top-level scope at none:1
 [2] eval(::Module, ::Any) at ./boot.jl:331
 [3] @make_a(::LineNumberNode, ::Module) at ./REPL[1]:2
in expression starting at REPL[2]:2

```

We can’t even _define_ the function `foo()` because the macro body is trying to call `f()` during compilation. Again, not what you want.

The fix is to make the macro produce _code_ that does what you want:

```julia
julia> macro make_a()
         quote
           $(esc(:a)) = f()
         end
       end
@make_a (macro with 1 method)

julia> function foo()
         @make_a()
         println("a = ", a)
       end
foo (generic function with 1 method)

julia> f() = 3
f (generic function with 1 method)

julia> foo()
a = 3

julia> a
ERROR: UndefVarError: a not defined

```

Now we can define `foo()` before `f()` because we’re no longer trying to _call_ `f()` at compilation time, and `a` no longer leaks out into global scope.

---

<div class="post-metadata">

**Author:** ![anon92994695](https://avatars.discourse-cdn.com/v4/letter/a/ce7236/32.png) [@anon92994695](https://discourse.julialang.org/u/anon92994695)\
**Post date:** [September 18, 2020, 9:21pm UTC](https://discourse.julialang.org/t/memoize-pkg-bad-use-of-metaprogramming/46874/4 "2020-09-18T21:21:48Z")

</div>

Yea I figured it wasn’t good, didn’t know about the side effects thing - but it does make a lot of sense. I now see why most of their “good uses” are for code generation. Good information thank you. I tend not to write macros so I am pretty weak with metaprogramming stuff. I can make them functions though no problem :).

I’m still very interested to know if I’ve duplicated functionality available elsewhere though. This seems like something useful for DrWatson or similar.

---

<div class="post-metadata">

**Author:** ![jling](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jling/32/212909_2.png) [@jling](https://discourse.julialang.org/u/jling)\
**Post date:** [September 18, 2020, 9:36pm UTC](https://discourse.julialang.org/t/memoize-pkg-bad-use-of-metaprogramming/46874/5 "2020-09-18T21:36:14Z")

</div>

[GitHub - marius311/Memoization.jl: Easily and efficiently memoize any function, closure, or callable object in Julia.](https://github.com/marius311/Memoization.jl) maybe?

---

<div class="post-metadata">

**Author:** ![anon92994695](https://avatars.discourse-cdn.com/v4/letter/a/ce7236/32.png) [@anon92994695](https://discourse.julialang.org/u/anon92994695)\
**Post date:** [September 18, 2020, 9:42pm UTC](https://discourse.julialang.org/t/memoize-pkg-bad-use-of-metaprogramming/46874/6 "2020-09-18T21:42:43Z")

</div>

@jling this might work, thanks! I wanted to have things kind of “script” like but ready-made solutions are nice. I don’t think you can store the memoized function to disk though? but maybe? I know you can serialize Julia functions(and compositions of functions). Not sure. Something to look that though.

---

<div class="post-metadata">

**Author:** ![tamasgal](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamasgal/32/27946_2.png) [@tamasgal](https://discourse.julialang.org/u/tamasgal)\
**Post date:** [September 18, 2020, 9:44pm UTC](https://discourse.julialang.org/t/memoize-pkg-bad-use-of-metaprogramming/46874/7 "2020-09-18T21:44:02Z")

</div>

> [@anon92994695](#):
>
> is there already a package that does this? if so link please?

You should have a look at `DrWatson.jl`, especially at the function `produce_or_load()`: [Saving Tools · DrWatson](https://juliadynamics.github.io/DrWatson.jl/dev/save/#DrWatson.produce_or_load)

> **[GitHub - JuliaDynamics/DrWatson.jl: The perfect sidekick to your scientific...](https://github.com/JuliaDynamics/DrWatson.jl)**
>
> The perfect sidekick to your scientific inquiries. Contribute to JuliaDynamics/DrWatson.jl development by creating an account on GitHub.

---

<div class="post-metadata">

**Author:** ![Oscar\_Smith](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oscar_smith/32/25343_2.png) [@Oscar\_Smith](https://discourse.julialang.org/u/Oscar_Smith)\
**Post date:** [September 18, 2020, 9:44pm UTC](https://discourse.julialang.org/t/memoize-pkg-bad-use-of-metaprogramming/46874/8 "2020-09-18T21:44:23Z")

</div>

I don’t know of any implementation of this in Julia, but I wrote a python one last winter for a project.

---

<div class="post-metadata">

**Author:** ![anon92994695](https://avatars.discourse-cdn.com/v4/letter/a/ce7236/32.png) [@anon92994695](https://discourse.julialang.org/u/anon92994695)\
**Post date:** [September 18, 2020, 9:52pm UTC](https://discourse.julialang.org/t/memoize-pkg-bad-use-of-metaprogramming/46874/9 "2020-09-18T21:52:44Z")

</div>

DrWatson to the rescue. Yea this does look like it does what I’d need. Basically need to phrase every block as a function call. Which is a-okay. Now ideally you could do this with a block of code too - but I am starting to see why that is a difficult ask. I may mark this as a solution, but I want to do a little brainstorming to see if I can’t sneak my way around to something a little more bespoke.

@oscar_smith - yea the last time I wrote something like this it was also in python. It worked pretty well too.

---

<div class="post-metadata">

**Author:** ![braamvandyk](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/braamvandyk/32/5086_2.png) [@braamvandyk](https://discourse.julialang.org/u/braamvandyk)\
**Post date:** [September 19, 2020, 11:48am UTC](https://discourse.julialang.org/t/memoize-pkg-bad-use-of-metaprogramming/46874/10 "2020-09-19T11:48:15Z")

</div>

Thanks for a very clear and helpful explanation.

---

<div class="post-metadata">

**Author:** ![anon92994695](https://avatars.discourse-cdn.com/v4/letter/a/ce7236/32.png) [@anon92994695](https://discourse.julialang.org/u/anon92994695)\
**Post date:** [September 19, 2020, 2:15pm UTC](https://discourse.julialang.org/t/memoize-pkg-bad-use-of-metaprogramming/46874/11 "2020-09-19T14:15:39Z")

</div>

Edit - fixes on this iteration are two posts below  
@rdeits - does this fix the main issues? Now each function just returns a quoted expression and no evals happen. Now no side effects happen in the macros, but they happen in the scope they are requested (ie in a function call or package space).

```nohighlight
using JLD2
macro memoize(cache_var)
    filename = String(cache_var) * ".jld2"
    if !isfile( filename )
        quote
            @save $filename $cache_var
        end
    else
        quote
            @load $filename $cache_var
        end
    end
end

macro memoize(cache_var, fn)
    filename = String(cache_var) * ".jld2"
    if !isfile( filename )
        quote
            $fn
            @save $filename $cache_var
        end
    else
        quote
            @load $filename $cache_var
        end
    end
end

macro free_memo(cache_var)
    filename = String(cache_var) * ".jld2"
    if !isfile( filename )
        @warn "Cache is already empty."
    else
        rm(filename)
    end
end

@memoize a
a

@memoize a begin
    a = 123
    a += 456
    a /= 5
    println("Hey we ran this code!")
end

```

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [September 19, 2020, 2:23pm UTC](https://discourse.julialang.org/t/memoize-pkg-bad-use-of-metaprogramming/46874/12 "2020-09-19T14:23:11Z")

</div>

The `isfile` check will run when the macro is expanded. It seems that should be part of the expression returned as well.

---

<div class="post-metadata">

**Author:** ![anon92994695](https://avatars.discourse-cdn.com/v4/letter/a/ce7236/32.png) [@anon92994695](https://discourse.julialang.org/u/anon92994695)\
**Post date:** [September 19, 2020, 2:26pm UTC](https://discourse.julialang.org/t/memoize-pkg-bad-use-of-metaprogramming/46874/13 "2020-09-19T14:26:22Z")

</div>

Derrp yes that should be in the returned expression itself. No worries I’ll take another crack at it. Thanks. Sorry no coffee in me yet. I also forgot to quote the remove file in the free\_memo macro - yikes.

Here this should be better

```nohighlight
using JLD2

macro memoize(cache_var)
    filename = String(cache_var) * ".jld2"
    quote
        if !isfile( $filename )
            @save $filename $cache_var
        else 
            @load $filename $cache_var
        end
    end
end

macro memoize(cache_var, fn)
    filename = String(cache_var) * ".jld2"
    quote    
        if !isfile( $filename )
            $fn
            @save $filename $cache_var
        else
            @load $filename $cache_var
        end
    end
end

macro free_memo(cache_var)
    filename = String(cache_var) * ".jld2"
    quote
        if !isfile( $filename )
            @warn "Cache is already empty."
        else
            rm($filename)
        end
    end
end

```

---

<div class="post-metadata">

**Author:** ![rdeits](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rdeits/32/286_2.png) [@rdeits](https://discourse.julialang.org/u/rdeits)\
**Post date:** [September 19, 2020, 3:52pm UTC](https://discourse.julialang.org/t/memoize-pkg-bad-use-of-metaprogramming/46874/14 "2020-09-19T15:52:26Z")

</div>

Yeah, that’s much better. I think the only thing you’re missing is that you need to `escape` the symbols provided to the macro, e.g. `$(esc(var))` instead of `$var`. The section of the manual on macro hygiene has more info about escaping.

---

<div class="post-metadata">

**Author:** ![anon92994695](https://avatars.discourse-cdn.com/v4/letter/a/ce7236/32.png) [@anon92994695](https://discourse.julialang.org/u/anon92994695)\
**Post date:** [September 19, 2020, 5:21pm UTC](https://discourse.julialang.org/t/memoize-pkg-bad-use-of-metaprogramming/46874/15 "2020-09-19T17:21:19Z")

</div>

I really appreciate your input on this thread. I learned a bit, and some of the dangling things from the metaprogramming tutorial on Julia’s documentation have sunk in. Really appreciate it.

The same sentiment goes to Kristoffer and everyone else who contributed.