Using or import search current directory?

New to Julia
1

This is really an idea which comes from Python. (Rust uses it too, actually.) Please don’t write it off just for this reason, as there is a practical problem to be solved here.

Consider the following example package.

MyRepoName/
  PackageC/
    src/
      PackageC.jl
      Module1.jl
    test/
      runtests.jl
      test_PackageC.jl

With a few details:

# test_PackageC.jl

using PackageC
using Test

@test PackageC.exampleC("helloworld") == "helloworld"

@test PackageC.Module1.exampleModule1() == 1

# runtests.jl

println("running tests for PackageC")
include("test_PackageC.jl")

# PackageC.jl

module PackageC

using Module1

function exampleC(arg::String)
    return arg
end

end

# Module1.jl
module Module1

function exampleModule1()
    return 1
end

end

Finally, you would need to know something about how the environment has been setup.

The idea is to simply use JULIA_LOAD_PATH to set the installation location of the repository MyRepoName, regardless of what type of system it is installed to.

There are two good example usecases for this.

  1. Developers own machine. The dev clones the repo and exports JULIA_LOAD_PATH to wherever he has cloned it. He can run julia from anywhere in the command line, and by running using PackageC, he can load PackageC. He can then test it, play with it, measure its performance, etc.
  2. Deployment with Dockerfile. This one is simpler. Just write a Dockerfile. Copy MyRepoName to the container, and export JULIA_LOAD_PATH, pointing to the destination location where MyRepoName has been copied to.

Either way, what this means is:

  • There is an entry in LOAD_PATH which points to MyRepoName. So, using PackageX works, because this directory will be searched for PackageX.

Ok that’s all the minimal information required to see what we are trying to do, and to undersand what doesn’t work.

Why does this not work?

  • using Module1 does not work, because Module1 cannot be loaded through the same mechanism by which using PackageC is loaded.
  • The reason for it is Julia does not search the working directory of the source file containing a using or import statement.

Difference with Python

In contrast, this will work in Python, because import will also search the directory containing the file from which a using or import statement is executed.

(Actually, to be fully accurate here, the import statement has to be of the form import PackageName.ModuleName, or import .ModuleName if using a relative import path.)

Why does it matter?

The obvious solution here is to use include. However, this creates a problem.

The use of include explicitly causes code to be loaded from a file.

In contrast, with import X, Python looks to see if it already knows what X is, and if it does, it creates a reference to it. If it does not know what X is, then it will fall back to loading from file, searching the various locations specified in PYTHON_PATH. (Which might include the current working directory, or a directory local to the current working directory of the Python interpreter.)

The reason why we care about this is it would be useful for us to be able to dynamically load and replace code in a similar way to which Python is able to.

With include currently “locked” to loading code from files, and using and import currently “locked” to searching only the locations specified in LOAD_PATH, this currently isn’t possible - at least not in a straightforward way.

Python example

It might be useful to see a short Python example which demonstrates this behavior.

Directory structure:

MyRepoNamePython/
  src/
    PackagePy/
      __init__.py
      __main__.py
      Module1.py

# __init__.py

import PackagePy.Module1

# __main__.py

# Module1.py

def exampleModule1():
    return 1

Run python3 from src:

>>> import PackagePy
>>> PackagePy.Module1.exampleModule1()
1
>>> def exampleModule1b():
      return 2

>>> PackagePy.Module1.exampleModule1 = exampleModule1b
>>> PackagePy.Module1.exampleModule1()
2

Attempts at a Julia solution

Here I show some of my attempts to resolve this, by hacking around with LOAD_PATH. This doesn’t seem to work. I am not sure exactly why.

module PackageC

push!(LOAD_PATH, "$(@__DIR__)")

using Module1

export exampleC

function exampleC(arg::String)
    return arg
end

pop!(LOAD_PATH)

end

However, even with these dynamic changes to LOAD_PATH, Julia still does not seem to be able to find Module1.jl.

Does anyone know why this is the case?

If you don’t find anything presented here compelling, perhaps the more succinct argument is more compelling:

  • Since Julia is a dynamic language, it should be possible to dynamically replace types, functions, modules and packages.

Such a thing isn’t possible in Rust of course because it is statically typed.

2

The standard way to do this in Julia is

include("Module1.jl")
using .Module1

(I tend to think of the top-leve package file as analogous to a Makefile — it is where you put the information about which source files and directories make up the package. Each subdirectory can have its own top-level “Makefile” that gets included at the root level.)

There have been some proposals for an abbreviated syntax that lets you omit the include, e.g. using ./Module1, but this has never seemed to have been a big priority. Requiring one extra line in your “makefile” for submodules, which aren’t very common to begin with, hasn’t been an onerous requirement in practice. See

There have also been several previous discourse threads about this. Just google “Julia import module current directory” — it’s pretty common for people coming from Python to expect files and modules to be interchangeable.

5 Likes
3

In particular, the GitHub issue you linked got pulled in at least two very different directions, which kind of blocked any further progress on the matter.

1 Like
4

This is one of the ugly and annoying issues of Julia… But not everything can be perfect.

1 Like
5

Just wanted to draw attention to this quote from my post.

My post was quite long, so it’s easy to miss this important detail.

Python doesn’t behave the same way. If the module has already been loaded, it won’t go and re-load it again. With include and Julia, the behavior is different. It always re-loads the code from disk, even if it has already been loaded.

6

So what? Why would you include it more than once? e.g. you do that a the top-level of your package (your “makefile”), and thereafter you do using .Module1, or using ..Module1 from submodules of your package.

If you are using the module from completely unrelated code (so that it doesn’t know what has already been included), then it should probably be put into a separate package.

4 Likes
7

I explained in detail in the first post.

If there are parts of it you don’t understand please ask specific questions.

8

You only wrote

The reason why we care about this is it would be useful for us to be able to dynamically load and replace code in a similar way to which Python is able to.

Could you give an example problem that you are trying to solve this way? (e.g. for replacing code during development/debugging, but for that we have Revise.jl.)

Julia is definitely less dynamic than Python in some ways, e.g. you can’t monkey-patch objects with new methods.

2 Likes
9

Yes. See my first post.

10

The problem with the “standard” solution of include + using . is that it’s not 100% equivalent to just using as include effectively creates a copy of the module. Not an issue as long as your dependencies are neatly chained, but in more complicated situations it becomes a hassle.

I just recently stumbled over this. I have a function that is declared in a base module B. It is used in derived module D but needs to be overridden/specialised in support module S. Ideally I would have simply used using B in D and import B in S, but that doesn’t work of course. Setting the load path doesn’t work either since all of this is happening in a package. And include + using . doesn’t work, since then D and S both have their own copy of B and D doesn’t see S’ version of the function. What I ended up doing was to create a package module that include+using .s all the other packages. In D and S I then use ..B to get access to the function. It works, but I think it’s really not a pretty solution.

2 Likes
11

If they don’t belong in the same package, why not just put B in its own package? If they are separate packages, why do you need include, as opposed to just adding the package to your current env?

12

You keep claiming that, but I’m not seeing an actual problem other than “I want it to work like Python modules”. You don’t explain why PackageC.jl can’t just use include, because you don’t say why you need it to be more dynamic, other than re-iterating that this is what you want.

13

So, D is a macro package. The generated code makes use of some functionality the interface functions of which are defined in B. S provides some simple generic implementations of these (some of which most users will probably want to use). I could have put everything into one big module of course but that seems rather messy. On the other hand S isn’t big enough to warrant the overhead of creating a package for it.

14

If it’s easier for you to have it all in a single repo/package, I fail to see the problem with using ..B … this is how it’s supposed to work in a single integrated repo.

15

It’s still not clear from your descriptions why the typical practice of evaluating (include possibly) a module once in one place and importing it everywhere else wouldn’t have worked.

16

It’s unnecessarily complicated and it enforces a very specific import structure (package-global module that imports everything else). I could also imagine that my solution would lead to plenty of knock-on headaches in a bigger/more complicated project. If local using/import was a thing all of that could be avoided.

1 Like
17

It’s so we can dynamically replace modules, at runtime.

18

I can’t use import/using since it is a local module. Changing the load path (which I usually use for non-package code) doesn’t work inside of a package.

19

And it won’t happen in Julia because files do not encapsulate modules, module expressions do. Note that Python doesn’t even have a module expression, it automatically considers each file its own module.

This happens for Julia packages. If separating into a package to add to your environment is overkill, then evaluating the module is the only way you can let Julia know of its existence. Only do it once per session; think of it as a session-wise add and rm, not import.

Watch out, this is risky for correctness and isn’t as easily done in Python like you’re implying. Like you said, after a Python module is loaded and imported, it’s referenced on subsequent imports. That means if you change the file’s contents, subsequent imports will ignore it. You need importlib.reload to actually reload the module, then you’ll have to repeat any from-import statements to update those references. They’re not forcing you to do busy work, it’s genuinely difficult to track down and sometimes impossible to replace an obsolete module and its contents. You have to juggle 2 modules with the same name, and that is awful to work with. You don’t even have to experiment with modules and files to see this, just redefine a superclass and verify that obsolete instances and superclassing remains. Python’s as dynamic as it gets but that has its practical limits.

The same applies to Julia, only more so because compilation bakes in references to modules; it’s one of a few ways it needs to be less dynamic for optimization. Revise is used because it doesn’t evaluate the module expression again, you just modify the original module from file edits.

2 Likes
20

Sorry, there must be some disconnect in language here that is making it hard for us to understand what you’re doing exactly. Evaluating the module once and importing it in other modules is routine, no need to change load paths because the import statement searches through the nested modules. This is what stevengj and I were talking about, to be precise:

julia> module A # possibly `include`d from a file
        export x
        x = "hello world"
       end
Main.A

julia> module B
        using ..A # searches parent module of B for A
        println(x)
       end
hello world
Main.B

julia> module C
        using ..A # searches parent module of C for A
        println(x)
       end
hello world
Main.C

julia> B.A === C.A # same module, didn't "reload" it
true

You’re probably trying to do something different, but it’s not clear what.