How to avoid repeatedly calling include()?

New to Julia
1

Hello Julia Community :slightly_smiling_face:,

I’m developing my first Julia package at the moment. In the main module of this package, there are many functions that call functions from other modules. For example:

module main_module

function foo(; kwargs)
   include("Module_One.jl")
   x = Module_One.Function_One(kwargs...)
   return x
end

function bar(; kwargs)
   include("Module_Two.jl")
   y = Module_Two.Function_Two(kwargs...)
   return y
end

function baz(; kwargs...)
   include("Module_Three.jl")
   z = Module_Three.Function_Three(kwargs...)
   return z
end

Unless I’m mistaken :eyes:, if I call one of these functions regularly, e.g. foo(), then each time it will look to include Module_One, which seems wasteful…

To be more efficient, would it be wise/foolish to do the following instead? :face_with_raised_eyebrow:

module main_module

function foo(; kwargs)
   (@isdefined Module_One) ? nothing : include("Module_One.jl")
   x = Module_One.Function_One(kwargs...)
   return x
end

The idea being to skip re-including “Module_One” each time the function is called again…
Or is Julia efficient enough already that doing this is unnecessary?

Having tried to run code this way already, I tend to get a world age error the first time (i.e. The applicable method may be too new: running in world age 12345 ) but it works ok after the first run…

Any advice will be greatly appreciated :pray:t3: :innocent:
Thanks!

1 Like
2

To take a step back.
Why are you doing this?
What is you actual high level goals?
What do you think your code does?

This is incredibly atypical Julia code.
It’s not clear to me if you think you are doing something standard.
If if you are doing something atypical for a reason.
Without knowing that it is hard to answer appropriately.

7 Likes
3

Since include dynamically creates functions in the global scope, those functions have a newer world age than the one for example foo is currently running in. You can make this work the first time by using Base.invokelatest(Module_One.Function_One) instead of Module_One.Function_One(), but I have to agree with @oxinabox that this is typically an antipattern in Julia.

2 Likes
4

Hi @oxinabox, thanks for your help :slightly_smiling_face: :+1:t3:

In the example above, there are just three functions in main_module (foo, bar and baz), but in reality there are > 40. Each function has many lines of code, so putting many functions with lots of lines of code in main_module was cumbersome and messy to work with.
I want people to be able to simply call main_module.foo(; kwargs) or main_module.bar(; kwargs) etc., but have the code for each function stored in a different file (“Module_One.jl”, “Module_Two.jl”).

The nature of these functions means that they will be used in loops (likely >10000 times). So my thinking was that after the first time main_module.foo(; kwargs) was called, the @isdefined if statement would avoid including Module_One.Function_One again and again.

I appreciate that it’s atypical, but I couldn’t think of a simpler way to go about it… :man_shrugging:t3:
Hopefully my goals are clearer now :crossed_fingers:t3:

5

Thanks @simeonschaub :+1:t3:
That’s a very clear explanation.

Hopefully my reply above might elucidate the issue further… :crossed_fingers:t3:

6

My first question is why do you have a new module for each of these functions? Every of these include("Module_One.jl") declare a new module? Why? You know the code inside the files included by include do not need to be wrapped in a module, right?

My second question is why do you not do something like:

# MainModule file (please do not use snake_case for module names)
module MainModule
    include("foo.jl")
    include("bar.jl")
    include("baz.jl")
    ...
end

and then

# file foo.jl
function foo(; kwargs)
    # long body of your function here
end
5 Likes
7

So the usual pattern in Julia is to structure packages something like the following:

# MyModule.jl
module MyModule
export foo_functionality_one, foo_functionality_two, bar_functionality

include("foo_file.jl")
include("bar_file.jl")
end

# foo_file.jl
foo_functionality_one() = ...
foo_functionality_two() = ...
foo_functionality_private() = ...

# bar_file.jl
function bar_functionality()
    something = foo_functionality_private()
    ...
end

Each include literally just copy-pastes the file’s contents in where you call it. (EDIT: not quite accurate, see oxinabox’s comment below.) So you should usually put the includes outside the functions, not inside.

You do need to make sure that you don’t include the same file twice, as else everything gets evaluated twice, and you end up with two versions of the same function floating around.

If there’s dependencies between foo_file.jl and bar_file.jl, as in the example above, this means that you just implicitly assume that they’re present in bar_file.jl, and rely on MyModule.jl including the necessary functionality.

Likewise it’s up to you to get the includes in the right order – if for example bar_file.jl needs something in foo_file.jl to be evaluated first.

With that out of the way, you may like FromFile, which handles the above details for you. This means you can write the above example package as

# MyModule.jl
module MyModule
using FromFile
export foo_functionality_one, foo_functionality_two, bar_functionality

@from "foo_file.jl" import foo_functionality_one, foo_functionality_two
@from "bar_file.jl" import bar_functionality
end

# foo_file.jl
foo_functionality_one() = ...
foo_functionality_two() = ...
foo_functionality_private() = ...

# bar_file.jl
using FromFile
@from "foo_file.jl" import foo_functionality_private

function bar_functionality()
    something = foo_functionality_private()
    ...
end

i.e. making dependencies explicit, handling duplicate calls, not having to worry about include order.

Does that help you get started?

7 Likes
8

Maybe, you could try to use Julia’s multiple dispatch capabilities together with types. This is very simplified example, not sure what exactly you want to achieve.

struct Foo1
    name
end

struct Foo2
    name
end

function run(v::Foo1)
    println("Foo1")
end

function run(v::Foo2)
    println("Foo2")
end

run(Foo1("a"))
run(Foo2("a"))

and results

Foo1
Foo2

You could generalize it with aid of abstract type/types.

9

Also, you may have a look on my modules/packages tutorial for a broader discussion…

2 Likes
10

To be clear it doesn’t actually do this.
Which is why putting it in a function is odd.
Because the contents won’t end up inside the function.

include evaluates the contents of the file at global scope

15 Likes
11

I remember that I thought as well that include just copy-pastes. This misunderstanding does come up repeatedly on this forum. Probably calling it eval_file or something alike would have been clearer.

5 Likes
12

This misunderstanding is, in fact, so common that I made a half-phrase PR just to change the documentation from:

" include behaves as if the contents of the source file were evaluated in its place."

to

" include behaves as if the contents of the source file were evaluated in the global scope of the including module."

14 Likes
13

I’m sure that half of the confusion is people with C/C++ experience mapping Julia ‘include’ to C ‘#include’ in their minds. I know that was a factor for me.

At least in Julia we don’t have the subtlety of angle brackets vs double quotes.

Anyway I wonder, why is include() even valid to write anywhere other than in module-global scope?

3 Likes
14

It’s a fair question. 2 answers

  1. Because it is actually just a normal function (that does some normal things like reading a file, and then ccalling a very unnormal function). Normal functions can occur anywhere in julia code
  2. Occationally it is useful to conditionally define a bunch of things at global scope. See how Requires.jl is used inside __init__ functions
7 Likes
15

Maybe it is even worth reinforcing that if a function includes a file and is executed, and later the included file is modified, the function will execute the new code. That is, the include inside the function is a call to include, not at all a copy and paste. (I think that a note about that would be nice in the manual).

julia> function f()
         include("./file.jl")
       end
f (generic function with 1 method)

julia> @code_lowered f()
CodeInfo(
1 ─ %1 = Main.include("./file.jl")
└──      return %1
)
5 Likes
16

Agreed. I had exactly this thought some time ago …

2 Likes
17

I wander how many packages would be broken if the behavior of include suddenly became an actual copy-paste.

1 Like
18

Almost every package that calls include in it’s __init__, or that depends on one that does.

A lower bound on this is every package with a transitive dependency on Requires.jl
Which is 1304 packages

2 Likes
19

See Including code that defines a module within a function definition does not fail!? · Issue #39421 · JuliaLang/julia · GitHub

2 Likes
20

Thanks for all the comments on this :+1:t3:. It seems that the implementation of include() is commonly misinterpreted :man_shrugging:t3:

Knowing that include() brings contents of files into the global scope, I can see how my original idea is redundant (and somewhat faulty if the same functions are included separately -

).
It might be best just to include all the functions that may be called at the top of main_module, even though 90% of the them likely won’t be used. :face_with_raised_eyebrow:

Just out of curiosity, is there a way to call functions/code from another file locally within a function? :thinking: :thinking:

Thanks again guys! :raised_hands:t3: :raised_hands:t3: