Recompilation triggered when re-running function from module

Hello! I’m relatively new to Julia and have encountered some behavior I cannot explain.

In a MWE, I have a function that takes an ODEProblem and algorithm and just calls DiffEq.solve() with these arguments.

function myfun(prob, alg)
    sol = solve(prob, alg)
end

When I put this function in a script (that I include in my main script), @time reports compilation time on the first run, but not when I paste the relevant line @time myscript(prob, alg) into the REPL or include my main script again. So far, so good.

But when I include the same function in a module (with Project.toml and Manifest.toml) and do using MyModule in my main script, something weird happens.

  • including main.jl in new Julia session: @time reports >99% compilation time, as expected
  • re-including main.jl from the same session: @time reports >99% compilation time, 100% of which was recompilation
  • if instead of re-including main.jl, I just paste the relevant line @time myfun(prob, alg) in the REPL, @time reports no compilation time (as I would expect)

So, why is re-including the main script different from just running the relevant line from the REPL? And why does this happen only when calling a function from a local module, and not when calling a function from a script?
Any help is appreciated!

(main.jl for reference, as I cannot upload files)

using DifferentialEquations
using MyModule
include("script.jl")

function simple_deriv!(du, u, p, t)
    du[:] = -1.0 .* u
    return nothing
end

t_0 = 0.0
t_end = 1.0
u_0 = [1.0]
alg = ImplicitEuler()
args = (; adaptive=false, dt=1 // 20)
prob = ODEProblem(simple_deriv!, u_0, (t_0, t_end); args...)

print("running from module:")
@time myfun(prob, alg)

print("running from script:")
@time myscript(prob, alg)

FTR, Julia features several startup options that make answering questions like this one much easier. I’m guessing they won’t be needed here, just FYI. These include:

--warn-overwrite=yes

--trace-compile=stderr

# only available on nightly
--trace-compile-timing

# only available on nightly
--trace-dispatch=stderr

What does this file contain? Does it contain a redefinition of myfun?

script.jl contains the function myscript(prob, alg), which is identical to myfun

OK, now I realize I don’t really understand what issue are you describing, a more minimal reproducer could help clear things up.

Perhaps it’d help to explain some things about using and include:

  • using has multiple functionalities:
    • loading and precompiling packages: this happens only the first time you call using SomePackage
    • bringing the names of a module (possibly the top-level module of a package) into scope
  • The include function is a lower-level tool. What it does is literally include the contents of the file given as argument into the place where it was called. If you know C or C++, include in Julia is the equivalent of #include C preprocessor directive.

So if you do:

using SomePackage
using SomePackage

… the second using is a no-op.

If you do:

include("some_file")
include("some_file")

you will literally be including the contents of the file some_file into the caller file (or REPL) twice. So, for example, if some_file is just the definition of a function, calling include twice will first define the function, and then redefine it, overwriting the first definition.

Newcomers tend to misuse include a lot. In fact, include is only rarely actually necessary.

Does this help?

I was confused initially, some further investigation showed that defining the function in a module vs. in a script does not make a difference.

Here is a simplified example:

using DifferentialEquations

function simple_deriv!(du, u, p, t)
    du[:] = -1.0 .* u
    return nothing
end

t_0 = 0.0
t_end = 1.0
u_0 = [1.0]
alg = ImplicitEuler()
args = (; adaptive=false, dt=1 // 20)
prob = ODEProblem(simple_deriv!, u_0, (t_0, t_end); args...)

@time solve(prob, alg)
print("done")

My issue:

  • When including this file in a new session, @time reports >99% compilation time
  • running just @time solve(prob, alg) afterwards reports no compilation
  • re-defining simple_deriv! by pasting the same definition in the REPL, and then running @time solve(prob, alg) reports >99% compilation time, 100% of which was recompilation. The same happens when I re-include the file above from the REPL

I don’t understand why this is happening – if I re-define the function with the exact same expression, what changes? Why is recompilation necessary?

Julia doesn’t check that the body of the method is the same as before. Furthermore, I don’t think that condition is sufficient for the compiler to establish that the redefinition doesn’t require recompilation (invalidation) - it’d have to check that the behavior is completely the same, which means it’d have to check that for each mentioned name, called function, etc.

In short, redefining a method invalidates the compiled code, causing future calls to require recompilation.