Memoize and broken compilation

General Usage
1

After I have moved from 0.6.3 to Julia 1.1 I get the

WARNING: eval into closed module Memoize:
Expr(:const, Symbol("##A_memoized_cache") = Expr(:call, :IdDict))
  ** incremental compilation may be broken for this module **

And indeed Julia now crashes on certain execution paths. There were no problems with Julia 0.6.3.

How can I avoid this behaviour?

2

This looks like a known issue with that package: https://github.com/JuliaCollections/Memoize.jl/issues/32 . I suspect the error comes from the use of eval inside the macro body here: https://github.com/JuliaCollections/Memoize.jl/blob/master/src/Memoize.jl#L98

I suspect this is fixable but will require some changes to Memoize.jl.

As a temporary workaround could you try:

  1. Removing using Memoize from your code
  2. Copying the definition of @memoize from https://github.com/JuliaCollections/Memoize.jl/blob/master/src/Memoize.jl into your own module

This isn’t a real solution, but I suspect that it may make the problem you’re seeing go away for now.

2 Likes
3

Thank you!

Googling showed up a lot more similar cases than are mentioned on the issue page. The advice which finally seems to make things work again was to put “precompile(false)” on top of the package.

My trust in Memoize has disappeared nevertheless. So I tried Anamnesis. It worked but execution time went up by a factor of 6 and space by a factor of almost 4.

Let me add that I am using Mathematica and Maple and for my applications memoization is a sine qua non. This is provided by both programs in a very straightforward way. I am a bit disappointed about the current state of this technique with Julia.

4

If you’re disappointed in the state of memoization in Julia, then please contribute with PRs and related discussions! Julia and its ecosystem are heavily reliant on volunteers to step up and fix things that they see as broken or lacking. When you pay for Mathematica or Maple, you’re explicitly paying a company’s development team in the hopes that they’ll do what you want (which they sometimes will do, and sometimes will not). When you yourself contribute to Julia and its ecosystem, you get exactly what you work for, and nothing less (and sometimes more, if other people join in to help out)!

1 Like
5

If you’re disappointed in the state of memoization in Julia,
then please contribute with PRs and related discussions!

Done. See this thread. My research has also shown that the
problem has long been recognized, just not by me.

Julia and its ecosystem are heavily reliant on volunteers to
step up and fix things

My understanding of Julia is not sufficient fix things.

When you … what you … > When you … what you …

Thanks for the enlightenment.

6

I meant that you should contribute to the code itself, and discussions on how to fix it. Complaining about broken things is only useful if someone else if using that to prioritize and keep track of what they want to work on; most people do not work like this, especially in the open source ecosystem. Instead, people fix what they want fixed.

Neither was mine until I started trying to fix things I didn’t like, at which point I started learning how to write and use Julia. One doesn’t suddenly become a Julia developer/contributor out of thin air; it requires a desire to learn and improve, even if you don’t currently have the skills to accomplish what you want to see done.

2 Likes
7

Can anyone comment on how this might be fixable?

I got tired of the warnings & ripped @memoize out of my code, in favour of just explicitly defining a const Dict() for each such function.

2 Likes
8

Cassette will let you memoize functions even after they’ve been defined. Check out this SO post: How to do memoization or memoisation in Julia 1.0 - Stack Overflow

2 Likes
9

Just a guess without running the code, but since the warning indicates eval into closed module Memoize:

Generally the intent appears to be to eval into the current module (at least, that’s what makes sense in my opinion), so the eval should be modified to ensure it only evals into the current module (which is allowed). I’m going to guess that this line doesn’t do what is desired any longer. Honestly, I wonder if that macro really needs to be so complicated, since as you say, you can duplicate this pretty well with just a plain const Dict of the desired type.

10

I agree; in this post-0.6 world, Cassette is probably the best approach for memoization in the common case. You of course run the risk of things not working (kwargs have been a historic issue for Cassette overdubbing, not sure if that’s fully fixed yet), but your linked SO article probably suffices for most needs.

11

I think you may end up writing better code if you create your own cache

Not everything needs some fancy macro to do it

3 Likes