Dependencies of src files inside a package

I am struggling with dependencies of several files inside a project (MWE package M). I have illustrated the current structure (of a MWE), which reflects the natural separation of different parts of the real project:
dependency

For the MWE the package is named M with top file M.jl.
It calls a function dfunc defined in d.jl.
dfunc calls bfunc and cfunc from b.jl and c.jl.
A common data type is Astruct, defined in a.jl and used directly in b.jl,c.jl and d.jl.

The complete MWE package can be installed with
add https://github.com/oheil/M.jl
Currently every file (a.jl to M.jl) defines a separate module (A,B,C,D,M). I include the files and do a using where its needed, e.g.:

include("a.jl")
using .A

With this setup, when I try to load the package M I get:

julia> using M
[ Info: Precompiling M [d58019f5-37a3-4da3-a62f-a442f1098211]
ERROR: LoadError: MethodError: no method matching bfunc(::M.D.A.Astruct)
Closest candidates are:
  bfunc(::M.D.B.A.Astruct) at C:\Users\oheil\.julia\dev\M\src\b.jl:8

Other setups, like only defining sub-module A, and just including other files do result in other errors or warnings, e.g.
cannot replace module A during compilation

I have tried several different setups, e.g. only sub-moduling A, but nothing gave an error or warning free working call to dfunc().

How do I solve this dependency problem? What is the proper way of sub-moduling, including and using? (I didn’t tried with import)

I hope I don’t have to rearrange everything, because the real project is much more complex and the above structure, as already said, reflects the different logical entities of the project.

2 Likes

The problem is that you’re include()ing the same file multiple times. Each time you include("a.jl"), you create a new module named A, with new, incompatible, types and functions. You need to ensure that each file is included exactly once.

For example, you could do:

## M.jl
module M

include("A.jl")
using .A
include("B.jl")
using .B

end

## A.jl
module A
  ...
end


## B.jl
module B

using ..A

end

Note that we can still use module A within module B without including it again there.

6 Likes

Technically, I agree with @rdeits. However, I’m not sure that’s really the solution you are looking for.

My experience

I have experienced the same confusion myself, and I have noticed that others seem to be stuck in the same mindset. I think it might stem from our ingrained C/C++'s include/header file methodology (assuming that’s your background).

In most cases, I don’t really want to break things up into different modules - especially inside a single package. Typically, when I need to break things up into separate modules, I really end up making one package per module.

If I’m not mistaken, there is a good chance that what you consider to be a package might actually be some sort of repository that collects somewhat related import-able modules (called “packages” in julia). Don’t get me wrong, Modules still exist in Julia, but they seem to correspond more to namespaces in C++.

Suggested structure, given your MWE

I tend use my .jl files (as you showed) just to separate out different functional blocks, and keep my files to a more (mentally) manageable size.

So yes, I would define files a.jl, b.jl, c.jl, and d.jl as follows:

a.jl:

struct Astruct
   #...
end

#more related code ...

b.jl:

bfunc(a::Astruct) = dosomething(a)

#more related code ...

c.jl:

cfunc(a::Astruct) = dosomethingelse(a)

#more related code ...

d.jl:

function dfunc()
   a = Astruct()
   bresult = bfunc(a)
   cresult = cfunc(a)
   return dosomethingwith_bandc_result(bresult, cresult)
end

#more related code ...

Finally, I would include all these files in the master M.jl file.

Note that [pkg_loadpath]/[PkgName]/src/[PkgName].jl is the file Julia automatically loads when the caller imports your module with either import PkgName or using PkgName.

See Code Loading · The Julia Language for more details on how packages are loaded.

Well, that’s at least true if you add [pkg_loadpath] to Julia’s LOAD_PATH variable, or if your CWD is in [pkg_loadpath]. See Constants · The Julia Language for more details on LOAD_PATH.

In any case, this is what my M.jl would look like:

#But yes, your "Julia package" code needs to be wrapped inside
#a Module of the same name:
module M

#Import some "Julia packages":
#(Which you probably typically think of as "modules")
using FFTW
using SomeOtherPackage1
using SomeOtherPackage2

#include the all the functionality broken up in separate files:
include("a.jl")
include("b.jl")
include("c.jl")
include("d.jl")

function run_the_fancy_calculation
   #setup constants
   result = dfunc()
   #more computations
   return result
end

#more related code ...

end

Hope this helps

9 Likes

Thank you, both solutions work (at least for the MWE).

Trying the version using modules is still not satisfactory and I remember now, why I initially started with sub-modules in the real project. The reason why I am still unhappy with this is that VSCode complains about missing references:
vscode
which is annoying despite that the complete package is compiling and running fine.
Thats what I got in the real project after some increasing complexity and I solved it using sub-modules and later on I ran into the dependency problem above.

When I try the includes only solution VSCode doesn’t complain at all (for the MWE), which is fine, but now I don’t understand why it doesn’t complain and why I had the issue in the real project and why I found that sub-modules help to make the LanguageServer know about the symbols.
Of course you can’t tell what happened in my real project.

But maybe someone know why the LanguageServer doesn’t complain about ::Astruct in the case, when no sub-modules exist and e.g. b.jl looks just like:

#module B
#export bfunc
#include("a.jl")
#using ..A

function bfunc(::Astruct)
end

#end

Hovering over ::Astruct shows the correct definition but I can’t “Go to definition”.

For now I try to make my real project running without sub-modules. Therefor, if you don’t mind, I am going to mark the includes only solution as this.

Unfortunately, I haven’t tried VSCode yet. I should: I hear it works well.

But from what I can gather, it sounds like the Julia language definition used in VSCode has a few limitiations.

In the “version using modules” problem, it almost sounds like the Language Server Protocol definition does doesn’t know about relative module paths. Note that I say this only because relative module paths is the least-typically-used construct in Julia – at least it is given the simplicity of your MWE.

In the “includes only solution”, I’m much more surprised. Maybe the Language Server Protocol implementation still has issues tracking struct definitions across files???

Raising an issue

You might need to raise an issue here: https://github.com/julia-vscode/LanguageServer.jl

Of course, I suppose the limitation might be in the implementation on the VSCode side, but if that’s the case, the maintainers of LanguageServer.jl will probably try to get that resolved if you inform them.

Thanks for that @oheil !

I didn’t even know about this Language Server Protocol, or that it was supported by VI!

1 Like

After cleaning up my original project, all is working now as expected. Actually I still have mixed sub-modules and includes and all is good.

Viewing modules (in my case a package internal sub-module, and the main module same as the package name) as namespaces helped a lot to do it right and to understand the former error with

bfunc(::M.D.A.Astruct)

vs

bfunc(::M.D.B.A.Astruct)

This is now totally clear as

## b.jl
module B
include("a.jl")
using .A

end

introduced the ...B.A.Astruct, which wasn’t intended to exist.

I think I missinterpreted something, so it’s more an issue on my side than at Language Server. Except for the “Go to definition” not working, but displaying the proper definition while hovering. After doing a “Find all References” the “Go to definition” worked again as expected. So, there seems to be a minor bug, I will investigate further to make it really reproducible, and if I succeed surely will open an issue.

Thanks again to you @MA_Laforge and @rdeits !

1 Like

Ah, so here it sounds like you actually were trying to create a multi-“module” solution, and its just that the above MWE was a toy example.

Building a Multi-Module Multi-Package Solution

First, I want to remind you that a Julia “Module” is actually more akin to a C++ namespace.

So if you wish to break down your solution into what is typically known as “modules” in computer science-talk, you need to start talking of Julia “packages”:

Computer
Science
Julia C++
Arbitrary
Scope
begin … end { … }
Function-level
Scope
function x() … end int x() { … }
Module-level
Scope
module MyMod … end namespace MyNS { … }
Module Package .h header file
w/compiled library

Package directory

So the first thing you need to do is find a location to develop your packages. Then, you need to tell Julia where to find them by adding it to the LOAD_PATH global variable.

You can automatically specify the package location either by setting the shell environment variable JULIA_LOAD_PATH, or by directly adding to LOAD_PATH in your ~/.julia/config/startup.jl file.

Structure of package “library”

The basic structure of a package “library” is as follows:

[path/to/shared/package/root]
├── A
│   └── src
│       └── A.jl
├── B
│   └── src
│       └── B.jl
├── C
│   └── src
│       └── C.jl
├── D
│   └── src
│       └── D.jl
└── M
    └── src
        ├── M.jl
        ├── logicalunit1.jl
        ├── ...
        └── logicalunitn.jl

Where the logicalunit?.jl files simply allow you to break down your package implementation with more manageable file sizes.

To access these packages, you would register the above “library” with your ~/.julia/config/startup.jl file:

push!(LOAD_PATH, "path/to/shared/package/root")

Package definitions & using/import

In the toy MWE you gave, there is no real need to break things down into logicalunit?.jl files. So it would now be important to point out that what I previously showed you to was how include multiple logicalunit?.jl files to form a single package.

However, when you wish to access the features of a Julia package within your code, you must either call import or using (as opposed to include). This is illustrated in the sample files below.

A/src/A.jl:

module A #Package code must be wrapped inside a "Module"

struct Astruct
   #...
end

#more code ...

#Export symbols you don't want the user to
#explicitly qualify with A.Astruct, etc.:
export Astruct
end

B/src/B.jl:

module B
using A #That's right, you don't "include" a package.

#You could also "import A" if you are ok with
#explicitly qualifying "exported" symbols, like
#"A.Astruct" each time!!!
#import A

bfunc(a::Astruct) = dosomething(a)

#If you don't "export" your functions explicitly, there
#is no risk of name collisions, so you could even call
#your function "f" in all your packages or modules!!!!
f(a::Astruct) = dosomething(a)

#more code ...
export bfunc
end

C/src/C.jl:

module C
using A

cfunc(a::Astruct) = dosomething(a)
#more code ...
end

D/src/D.jl:

module D
using A
using B
using C

function dfunc()
   a = Astruct()
   bresult = bfunc(a)
   #Must explicitly qualify cfunc here. It was not exported:
   cresult = C.cfunc(a)
   return dosomethingwith_bandc_result(bresult, cresult)
end

#more code ...
end

M/src/M.jl:

module M

#Here, you don't need to call "using" on A, B, or C
#because M never accesses those packages directly:
using D

#But you might want to break down package "M"
#across multiple files to make things more manageable:
include("logicalunit1.jl")
#...
include("logicalunitn.jl")

function run_the_fancy_calculation
   #setup constants
   result = D.dfunc()
   #more computations
   return result
end

#more code ...
end

That’s it for the most part!

4 Likes

I’m starting to think the Julia Docs might be lacking a bit with regards to “packages”, modules, include, using, import, etc.

I still see similar questions arising on Discourse, every now and then.

I remember I had a hard time creating package “libraries” when I first started. The naming convention used in Julia somewhat clashes with what I was previously familiar with in other programming environments. That seems to have set me down the wrong path at first.

In fact, I still have trouble not calling a “library of packages” a “repository of packages” - because I’m afraid people might confuse these concepts with how packages are registered in “General” from Git “repositories” on Github. FYI: Julia’s package registration system was designed to only have one “package” per Git “repository”. It appears Git repositories were never designed to provide multiple Julia “packages”.

The worse part of all this is that you are in no way new to Julia, @oheil. I can see you have made significant contributions to our community for many years.

I have to wonder how many more people are having trouble, and how we can help them get more efficient.

I don’t like the idea we might be loosing out on potential adopters simply because they can’t figure out how to create/use their own module package “library”. I almost gave up on Julia myself when I struggled with the module/package concepts a while back (and still do sometimes).

5 Likes

This is true, but until now I didn’t had very complex projects. I am not so good in digesting documentations when I don’t have a real problem to work on and to use what I read.
But your explanations are great and very helpfull! Thank you.
Currently I am on the “Module-level Scope”-complexity (more because I want it so not because I really am) and when I think “module” I mean the Julia construct module .... end. I am not sure how to improve documentation, maybe it is better served as a separate document, like a blog or something. Actually your answers here (and @rdeits) are perhaps already sufficient? It was for me. Whenever these question come again I will link to this thread.

Actually Julia is quite easy to start with and first results are mostly impressive. But mastering Julia is quite a task. The learning curve is somehow exponential and I don’t see the plateau yet :slight_smile:

I think people come and go, thats natural. I don’t like the idea, that programming languages/paradigms are somehow religions, where people have to be locked in. But wording like e.g. “evangelists” do imply that. The words form the thoughts. Programming languages are tools, they have to be used appropriately according to a problem. But enough of philosophy.

If some early adopters are leaving, than just because they didn’t ask here. We can’t force them to do that. But we can help those who ask. And thats what we do. Right on @MA_Laforge :+1:

1 Like

Just FYI, we do support having multiple Julia packages in a single Git repository.

4 Likes

Sorry, I should have said “originally”, I suppose. I’m still figuring out how to use those in an effective manner.

I hope that one day soon, there will be an fast, simple, elegant way to bundle multiple Julia packages into a simple Git repository, while simultaneously allowing developers to maintain minimum dependency requirements.

I have heard others suggesting it is already possible, but I personally seem to get snagged on little issues which I do not appear to communicate effectively.

2 Likes