Defining multiple modules in a package

[GlenHenshaw]

I am developing two packages. Let’s call them “A” and “B”. Package “A” defines several modules, let’s call them “d”, “e”, and “f”. I have activated environment B from Pkg and made package A a dependency.

A.jl:

module A
    include("d.jl")
    using .d: f
    export f

    include("e.jl")
    include("f.jl")
end

When I try to use functions defined in d.jl from code in package B:

module B
    import A
    import A.d

    A.d.f()
end

I very reliably get the error

ERROR: LoadError: UndefVarError: d not defined

I’ve tried every permutation of include’s, import’s, using’s, and dot notation I can think of. The error never changes.

EDIT:

This was user error. The problem was that there was already a package named “A” installed (which, needless to say, doesn’t define anything named “d”), and "using A’’ found that package instead of the one I was writing.

[lmiq]

This works, perhaps that gives you a direction:

julia> module A
         module A1
            f(x) = x + 1
         end
       end
Main.A

julia> module B
         import ..A.A1 # two dots
         g(x) = A1.f(x)
       end
Main.B

julia> B.g(1)
2

Ps. I don’t think that module names can start with numbers.

(otherwise I think that if A is a package already installed you should be able to use using A . I am not sure if what you are seeing results from including the files containing the modules).

[GlenHenshaw]

I’m not actually naming the modules starting with numbers. I’ve changed the question accordingly.

[MA_Laforge]

Step 1: No need for using here

Ok, so first thing is: you don’t need to do using .d:f. That’s just if you wish to have the f “symbol” directly available from the A module.

Julia doesn’t do information hiding, so as long as your code is “sourced”/loaded, you can access the structures/functions (actually referred to as “symbols”) through their path. BTW: The thing that loads your code is “include()”.

So again, the first step is tor remove using .d:f - and, I suppose export f - unless you actually want f to be available directly from the A module:

@GlenHenshaw

module A
    include("d.jl")
    using .d: f
    export f

    include("e.jl")
    include("f.jl")
end

Step 2: Lesson on module

Julia module-s are not really “software modules”. They are actually software namespaces - and include()-ing a file does not automatically create said namespace. So, if you want the modules you wrote down in your example, you’d have to do the following:

A.jl:

module A
    module d
        include("d.jl")
    end #module d
    module e
        include("e.jl")
    end #module 3
    module f
        include("f.jl")
    end #module f
end

Then, you could do the following in package B:

module B
    import A
    #import A.d #Again, no point in importing if you qualify with explicit paths

    #Qualifying f() call with explicit path to A.d.f()
    #... so only need to "import A" to get "A" symbol
    A.d.f()
end

[MA_Laforge]

Step 3: no need for modules/namespaces

But my guess is that you never really wanted to create sub-modules d, e, & f.

Given that you were trying to do using .d:f from module A, my guess is that you really are just using separate files to break up your software into a more manageable solution:

Awesome; keep doing that . It’s much easier to find code if you split up large solution into separate files.

But that means you can stick with a simpler structure:

A.jl

module A
    include("d.jl")
    include("e.jl")
    include("f.jl")
end

Yup. That’s it. Now, all the code from d.jl, e.jl, and f.jl are now part of module A.

very_long_function_names_to_avoid_name_collisions

If you don’t need to split things into submodules - don’t. In julia, I find there is rarely a reason to do so.

In most languages, you need namespaces to avoid creating very_long_function_names_to_avoid_name_collisions - but that isn’t typically necessary in Julia.

Leverage multiple dispatch

…instead of relying on namespaces.

If you want to create a method of push!() that behaves differently for your values, you simply create your own struct MyType, and define a special Base.push!(::MyType, ...) method - and it automatically plays nice with the rest of Julia. The same can be done with any other function.

[MA_Laforge]

You might also look at the example here:
→ Dependencies of src files inside a package

[lmiq]

I think I understand what he is saying:

1. Create two packages, Package1 and Package2 with PkgTemplates.

2. Package1 contains:

module Package1
  module Package1_A
    f(x) = 2*x
  end
end

3. Package2 contains:

module Package2
  using Package1
  g(x) = Package1.Package1_A.f(x)
  export g
end      

4. Go to to dir/Package2, and start Julia with julia --project such that an environment for Package2 is created. Then, add Package1 :

(Package2) pkg> add ../Package1
    Cloning git-repo `/home/leandro/Drive/Work/JuliaPlay/packages/Package1`
   Updating git-repo `/home/leandro/Drive/Work/JuliaPlay/packages/Package1`
   Updating registry at `~/.julia/registries/General`
  Resolving package versions...
Updating `~/Drive/Work/JuliaPlay/packages/Package2/Project.toml`
  [231e8d6a] + Package1 v0.1.0 `../Package1#master`
Updating `~/Drive/Work/JuliaPlay/packages/Package2/Manifest.toml`
  [231e8d6a] + Package1 v0.1.0 `../Package1#master`

julia> using Package2
[ Info: Precompiling Package2 [422864b9-2805-4bb9-af4e-8ecfc857a12e]

5. Finally, Try to use the function g, which is defined in Package2:

julia> Package2.g(1)
ERROR: UndefVarError: Package1_A not defined
Stacktrace:
 [1] getproperty at ./Base.jl:26 [inlined]
 [2] g(::Int64) at /home/leandro/Drive/Work/JuliaPlay/packages/Package2/src/Package2.jl:5
 [3] top-level scope at REPL[4]:1

julia>

edit: It seems that the issue not related to the the nested modules. Simply the functions of Package1 are not loaded, it is like if Package1 was empty.

[PetrKryslUCSD]

What if one does

using Package1.Package1_A: f

? These types of references work for me.

[lmiq]

Actually what I have done turns out to hit in a simpler problem, which I do not know if it is the same as the OPs:

Given a single package, Package1, in directory ../Package1. Currently the main module of this package contains:

module Package1
  f(x) = 2*x
end

And then, if I try to use it, I get:

(@v1.5) pkg> add ../Package1
   Updating git-repo `/home/leandro/Drive/Work/JuliaPlay/packages/Package1`
   Updating registry at `~/.julia/registries/General`
######################################################################## 100,0%
  Resolving package versions...
Updating `~/.julia/environments/v1.5/Project.toml`
  [231e8d6a] + Package1 v0.1.0 `../../../Drive/Work/JuliaPlay/packages/Package1#master`
Updating `~/.julia/environments/v1.5/Manifest.toml`
  [231e8d6a] + Package1 v0.1.0 `../../../Drive/Work/JuliaPlay/packages/Package1#master`

julia> using Package1

julia> Package1.f
ERROR: UndefVarError: f not defined
Stacktrace:
 [1] getproperty(::Module, ::Symbol) at ./Base.jl:26
 [2] top-level scope at REPL[3]:1

julia>

Curiously, using dev seems to be reading the latest version (I changed the version to 0.1.1), but still the result is the same:

(@v1.5) pkg> dev ../Package1
Path `../Package1` exists and looks like the correct package. Using existing path.
  Resolving package versions...
Updating `~/.julia/environments/v1.5/Project.toml`
  [231e8d6a] + Package1 v0.1.1 `../../../Drive/Work/JuliaPlay/packages/Package1`
Updating `~/.julia/environments/v1.5/Manifest.toml`
  [231e8d6a] + Package1 v0.1.1 `../../../Drive/Work/JuliaPlay/packages/Package1`

julia> using Package1

julia> Package1.f
ERROR: UndefVarError: f not defined
Stacktrace:
 [1] getproperty(::Module, ::Symbol) at ./Base.jl:26
 [2] top-level scope at REPL[9]:1

Thus, if this is the problem, it is actually to know how to install and load a package that is in a local folder.

[MA_Laforge]

@lmiq

Thus, if this is the problem, it is actually to know how to install and load a package that is in a local folder.

Yes @lmiq. It is also true that you have to build your package folders correctly, and use the package manager correctly. This is definitely another possible source of issues (took me a while to understand what Julia wants).

Possible issues with your solution

Relative paths

My first suggestion would be to avoid using relative paths at all when you add or dev a package. Since I’m pretty sure you don’t know what will be the “current directory” (I sure don’t), to me, that seems like a bad way to learn how to use the package manager correctly:

@lmiq

(@v1.5) pkg> add ../Package1

By looking carefully at the package manager output, it would appear that you are using a relative path:

@lmiq

  Resolving package versions...
Updating `~/.julia/environments/v1.5/Project.toml`
  [231e8d6a] + Package1 v0.1.0 `../../../Drive/Work/JuliaPlay/packages/Package1#master`

I’m pretty sure this is a problem waiting to happen unless you really know what you are doing.

add vs dev

Indeed. pkg> add is not the best choice to to add packages under development. Each time you would want to test a change in your code, you’d have to:

1. bash$ git commit . (The changes to your package).
2. pkg> up (make Juila update which version of your package it uses).

Otherwise, you will be testing out old code.

Going back to relative paths:

@lmiq

(@v1.5) pkg> dev ../Package1
Path `../Package1` exists and looks like the correct package. Using existing path.

Look at the package manager output:

• It looks to me like you tried developing a package that was already registered by the package manager - so instead of pointing to your new directory, it kept using the old version of your package.

Once added, you don’t specify paths anymore

Since you already had a Package1, the dev command should have just been:

(@v1.5) pkg> dev Package1

That will correctly switch you over to dev mode on Package1.

If you wish to switch to a different “Package1”, then I suggest the following:

(@v1.5) pkg> rm Package1
(@v1.5) pkg> dev /abs/path/to/new/Package1

Works for me

I tried creating your Package1, just to make sure I am not misleading anyone:

@lmiq

module Package1
  module Package1_A
    f(x) = 2*x
  end
end

But on my side, I stuffed this code in a package I “generate”-ed with the package manager:

(v1.3) pkg> generate Package1

So, this code went into my new ./Package1/src/Package1.jl. Everything worked fine:

(v1.3) pkg> dev /abs/path/to/new/Package1

julia> using Package1
julia> Package1.Package1_A.f(3)
6

[lmiq]

Thank you for the answer! What was making things not work here was that I was not commiting the changes (git add -A; git commit -m "update message") in Package1.

The relative path seem to to work as well the the absolute path (not sure if that is safe, but it works, actually it would be good that it was, because no warning or anything is printed).

Concerning add or dev, both work in what concerns the path, but good point remembering that with dev there is no need to update the package.

Thus, concerning my previous posts: What I did wrong there is not commiting the changes when the packages were modified. Not sure if that was the same problem OP was facing, though.

[lmiq]

Anyway, that makes me wonder, what is the most practical way to develop simultaneously two packages, one that depends on the other? Revise won’t track the changes of the dependencies.

I guess that is not such an unusual scenario, is it?

[magister-ludi]

I do this all the time. I dev the packages that I’m working on. Julia recognises changes and recompiles at startup, Revise tracks changes in a REPL session.

EDIT: I always use the full path. I can think of too many ways things could go wrong with relative paths.

[lmiq]

@magister-ludi

Revise tracks changes in a REPL session

Of a module that is loaded with using from inside the main module of another package? Certainly not in general, does it for some specific setup?

[magister-ludi]

I’m not at a computer right now, so I can’t confirm, but I think that if the package that is pulled in by using has been dev’d, then it works. I won’t be able to check until tomorrow. If nobody has given you the answer by then, I’ll let you know.

[lmiq]

Yes, it does work. Cool.

So the way to go is to dev all packages being developed and using Revise.

One possible source of confusion is that when one does dev Package1 the source being tracked is that of the default development directory (usually .julia/dev/), even if the package was originally installed from another local directory. That behavior is natural if the package was installed from an online repository, but not so from a local version. Thus, we should be careful and use dev /path/to/Package1, otherwise two copies of the package will be available, causing confusion.

[magister-ludi]

@lmiq

we should be careful and use dev /path/to/Package1

Yes, I was going to mention that once I had done some experiments, but you got there first I’m pleased it’s working for you now.

[rafael.guerra]

@magister-ludi and @lmiq: in VS Code, do you have the two packages you are developing open in two separate windows (each with its own environment)?

[lmiq]

I don’t

[magister-ludi]

I don’t use VSCode much, but I’m not really sure what you are trying to understand.

If package A is a package that is used by package B (using dev, not add), then it shouldn’t matter if files from package A are open in an editor window. Once the files are saved to disk, the package B environment should notice and ‘revise’ the package. The environment that package A is using, shouldn’t make a difference.