How to avoid repeatedly calling include()?

Since include dynamically creates functions in the global scope, those functions have a newer world age than the one for example foo is currently running in. You can make this work the first time by using Base.invokelatest(Module_One.Function_One) instead of Module_One.Function_One(), but I have to agree with @oxinabox that this is typically an antipattern in Julia.

2 Likes

Hi @oxinabox, thanks for your help :slightly_smiling_face: :+1:t3:

In the example above, there are just three functions in main_module (foo, bar and baz), but in reality there are > 40. Each function has many lines of code, so putting many functions with lots of lines of code in main_module was cumbersome and messy to work with.
I want people to be able to simply call main_module.foo(; kwargs) or main_module.bar(; kwargs) etc., but have the code for each function stored in a different file ("Module_One.jl", “Module_Two.jl”).

The nature of these functions means that they will be used in loops (likely >10000 times). So my thinking was that after the first time main_module.foo(; kwargs) was called, the @isdefined if statement would avoid including Module_One.Function_One again and again.

I appreciate that it’s atypical, but I couldn’t think of a simpler way to go about it… :man_shrugging:t3:
Hopefully my goals are clearer now :crossed_fingers:t3:

Thanks @simeonschaub :+1:t3:
That’s a very clear explanation.

Hopefully my reply above might elucidate the issue further… :crossed_fingers:t3:

My first question is why do you have a new module for each of these functions? Every of these include("Module_One.jl") declare a new module? Why? You know the code inside the files included by include do not need to be wrapped in a module, right?

My second question is why do you not do something like:

# MainModule file (please do not use snake_case for module names)
module MainModule
    include("foo.jl")
    include("bar.jl")
    include("baz.jl")
    ...
end

and then

# file foo.jl
function foo(; kwargs)
    # long body of your function here
end
5 Likes

So the usual pattern in Julia is to structure packages something like the following:

# MyModule.jl
module MyModule
export foo_functionality_one, foo_functionality_two, bar_functionality

include("foo_file.jl")
include("bar_file.jl")
end

# foo_file.jl
foo_functionality_one() = ...
foo_functionality_two() = ...
foo_functionality_private() = ...

# bar_file.jl
function bar_functionality()
    something = foo_functionality_private()
    ...
end

Each include literally just copy-pastes the file’s contents in where you call it. (EDIT: not quite accurate, see oxinabox’s comment below.) So you should usually put the includes outside the functions, not inside.

You do need to make sure that you don’t include the same file twice, as else everything gets evaluated twice, and you end up with two versions of the same function floating around.

If there’s dependencies between foo_file.jl and bar_file.jl, as in the example above, this means that you just implicitly assume that they’re present in bar_file.jl, and rely on MyModule.jl including the necessary functionality.

Likewise it’s up to you to get the includes in the right order – if for example bar_file.jl needs something in foo_file.jl to be evaluated first.

With that out of the way, you may like FromFile, which handles the above details for you. This means you can write the above example package as

# MyModule.jl
module MyModule
using FromFile
export foo_functionality_one, foo_functionality_two, bar_functionality

@from "foo_file.jl" import foo_functionality_one, foo_functionality_two
@from "bar_file.jl" import bar_functionality
end

# foo_file.jl
foo_functionality_one() = ...
foo_functionality_two() = ...
foo_functionality_private() = ...

# bar_file.jl
using FromFile
@from "foo_file.jl" import foo_functionality_private

function bar_functionality()
    something = foo_functionality_private()
    ...
end

i.e. making dependencies explicit, handling duplicate calls, not having to worry about include order.

Does that help you get started?

6 Likes

Maybe, you could try to use Julia’s multiple dispatch capabilities together with types. This is very simplified example, not sure what exactly you want to achieve.

struct Foo1
    name
end

struct Foo2
    name
end

function run(v::Foo1)
    println("Foo1")
end

function run(v::Foo2)
    println("Foo2")
end

run(Foo1("a"))
run(Foo2("a"))

and results

Foo1
Foo2

You could generalize it with aid of abstract type/types.

Also, you may have a look on my modules/packages tutorial for a broader discussion…

https://syl1.gitbook.io/julia-language-a-concise-tutorial/language-core/11-developing-julia-packages

2 Likes

To be clear it doesn’t actually do this.
Which is why putting it in a function is odd.
Because the contents won’t end up inside the function.

include evaluates the contents of the file at global scope

15 Likes

I remember that I thought as well that include just copy-pastes. This misunderstanding does come up repeatedly on this forum. Probably calling it eval_file or something alike would have been clearer.

5 Likes

This misunderstanding is, in fact, so common that I made a half-phrase PR just to change the documentation from:

" include behaves as if the contents of the source file were evaluated in its place."

to

" include behaves as if the contents of the source file were evaluated in the global scope of the including module."

14 Likes

I’m sure that half of the confusion is people with C/C++ experience mapping Julia ‘include’ to C ‘#include’ in their minds. I know that was a factor for me.

At least in Julia we don’t have the subtlety of angle brackets vs double quotes.

Anyway I wonder, why is include() even valid to write anywhere other than in module-global scope?

3 Likes

It’s a fair question. 2 answers

  1. Because it is actually just a normal function (that does some normal things like reading a file, and then ccalling a very unnormal function). Normal functions can occur anywhere in julia code
  2. Occationally it is useful to conditionally define a bunch of things at global scope. See how Requires.jl is used inside __init__ functions
7 Likes

Maybe it is even worth reinforcing that if a function includes a file and is executed, and later the included file is modified, the function will execute the new code. That is, the include inside the function is a call to include, not at all a copy and paste. (I think that a note about that would be nice in the manual).

julia> function f()
         include("./file.jl")
       end
f (generic function with 1 method)

julia> @code_lowered f()
CodeInfo(
1 ─ %1 = Main.include("./file.jl")
└──      return %1
)

5 Likes

Agreed. I had exactly this thought some time ago …

2 Likes

I wander how many packages would be broken if the behavior of include suddenly became an actual copy-paste.

1 Like

Almost every package that calls include in it’s __init__, or that depends on one that does.

A lower bound on this is every package with a transitive dependency on Requires.jl
Which is 1304 packages

2 Likes

See Including code that defines a module within a function definition does not fail!? · Issue #39421 · JuliaLang/julia · GitHub

2 Likes

Thanks for all the comments on this :+1:t3:. It seems that the implementation of include() is commonly misinterpreted :man_shrugging:t3:

Knowing that include() brings contents of files into the global scope, I can see how my original idea is redundant (and somewhat faulty if the same functions are included separately -

).
It might be best just to include all the functions that may be called at the top of main_module, even though 90% of the them likely won’t be used. :face_with_raised_eyebrow:

Just out of curiosity, is there a way to call functions/code from another file locally within a function? :thinking: :thinking:

Thanks again guys! :raised_hands:t3: :raised_hands:t3:

I would think that you would include the file and then you may call the function, or am I misunderstanding something?

It may also be easiser, while you’re getting started with julia to not use modules and just share a namespace - multiple dispatch means you can reuse your own and base methods for new types, and namespace clashes are less important. The NameOfPackage.jl file is often the only module, and usually has all of the include statements in it in one place. Some packages use multiple modules, but often namespace boundaries occur only at the package level.

You will also find with julia, unlike some static languages, that you can call functions defined in later includes from earlier includes, so are freed from that kind of constraint and worrying too much about includes and where you can call functions - although types used in method signatures and inheritance need to be defined previously.

4 Likes