How to structure project in julia

New to Julia
1

I’m new to Julia (2 weeks of exp). I would like to port and extend existing project from C++/python to julia. I’m having big difficulties in structuring my project. The project will contain roughly 50-100 files/modules, some modules will depend on other modules in the project. Coming from C++/python naturally I was expecting following layout to work out of the box:

#file ModuleA.jl
module ModuleA
...
end

#file ModuleB.jl
module ModuleB.jl
using ModuleA # or .ModuleA
...
end

To my surprise ModuleB will not import dependent ModuleA. I’ve been researching this subject for a week and haven’t found good solution. I found following options:

1. Add project directories to LOAD_PATH

Upsides:

  • local modules just work ™
  • push!(LOAD_PATH, …) can be added to startup file
  • project startup file can be added to vscode settings "julia.additionalArgs": "-L${workspaceFolder}/startup.jl"

Downsides:

  • feels hacky
  • project can’t have Project.toml, so can’t become a package
  • compiled modules pollute local\share\julia\compiled\v1.9\ with dlls
  • Pluto disregards julia -Lstartup.jl, so LOAD_PATH must be amended in every notebook

2. Use include

Include file followed by .using

#file ModuleB.jl
module ModuleB.jl
include("ModuleA.jl")
using .ModuleA
...
end

This doesn’t work when more than one dependency includes the same file due to lack of include once semantics, there is no #pragma once or #ifdef to my knowledge, I rate this option as worse than C.

3. FromFile

FromFile looks great at first glance

#file ModuleB.jl
module ModuleB.jl
@from "ModuleA.jl" using ModuleA
...
end

but there are a few problems:

  • module are wrapped in additional modules, this changes module tree
  • modules loaded into Main are wrapped differently than modules loaded from other modules so types often differ typeof(b.a) != typeof(a), potentially problematic for dispatching
  • @from … using … doesn’t work in Pluto

4. Create one monolithic file

Create one main julia file per project, all files are included into the monolith:

#file ModuleB.jl
module ModuleB.jl
# no dependencies declared here
...
end
#file Monolith.jl
include("ModuleA.jl")
include("ModuleB.jl")

Only downsides:

  • modules can’t declare their dependencies
  • files must be included in topological order
  • files/modules become just inlines, they can’t be executed/tested in julia
  • worse than C, it’s not 1970 anymore
  • over my dead body option

5. Create package per file/module

For each file/module create a package. Packages can be ]dev’ed or published in local repository.

  • requires nesting of every file/module into it’s own directory
  • doubles the amount of files (julia + toml)
  • even for a small project this will become a maintenance hell

Summary

Are there any other options that I missed?

Of all these options, I chose #1 as the least bad for now. I like the language but I feel extremely frustrated that there is no modern way to deal with local dependencies in Julia. The topic relative using/import should search current directory · Issue #4600 · JuliaLang/julia · GitHub is decade old, left unaddressed. Are there any plans to improve the situations? If not, it’s not too late for me to bail out.

14 Likes
2

Unfortunately, I think your summary is pretty accurate. As far as I can tell, 4. is the most common pattern and for some reason recommended by many, but I also do not understand that recommendation because it certainly looks and feels like poor organization. Of those options maybe the “cleanest” is 5., when independent concerns can be neatly sorted and separated, but it definitely adds a lot of boilerplate and maintenance burden, and doesn’t really make sense for many project topologies.

3

What is the meaning of “file/module” : is it file per module ?

Julia projects are most often defining a single module (== namespace in C++) which usually contains many files.

If the project is really large it would make sense to have sub-modules (::sub namespaces) but in this case, it is worth considering separate (and dependent) packages.

1 Like
4

Yes, the most typical structure is

module MyPackage
    using Dependency1
    using Dependency2

    export function1, function2

    include("file1.jl")
    include("file2.jl")
    include("file3.jl")

end

and file1.jl etc do not contain modules, just straight code.

As you have seen already, the alternatives are not without discussion. But the fact is that submodules are much less useful in Julia than in other languages. IMO it is just better to accept that and go for the one-module format. As one gets used to Julia, its not a big deal, it works well.

(and situations where submodules are useful do exist, particularly when the submodule provides a set of rather complete functionalities. In these cases it is also common just separate the module in a new package, as mentioned above).

11 Likes
5

I have been trying to get used to it for over a year, and it doesn’t seem likely to happen at this point…

I don’t think it works well, and it is a big deal to me. It makes code navigation impossible and when reading a file it’s frequently very difficult to see where various names are being defined since the code is just being injected into some top level. I also hate the fact that I have to think about the order of inclusion of all these files.

5 Likes
6

Yes, there are downsides, thus those long discussions you can find here in the forum. But many (I would say most with some confidence) developers ended up preferring this way. FromFile.jl is probably the best alternative to this.

1 Like
7

julia’s module is a counterpart of c++ class for me - a struct with accompanying functions

8

When I create a project, I develop within a package using PkgTemplates.jl. It creates the project structure, and I put my main code in src/project_name which is a module. As Imiq noted, these generally are not modules but just related functions and/or structs.

Here is an example of the main module. My scripts go in a folder called scripts or examples. Here is an example script. At the top, you can see that I activate the project environment with:

cd(@__DIR__)
using Pkg
Pkg.activate("..")

Environments are very useful because they provide version control and ensure your project will work months later. I’m not sure what a “modern” approach is for dependencies, but I have used many different programming languages, and I can honestly say that Julia’s package manager is the easiest and most flexible to use.

4 Likes
9

I know, which is why I protested with so much fervor :sweat_smile:
I often try to read through the code of packages I use, and it’s made much more difficult than it needs to be because there’s no way to discover where variables / functions are defined. Half my time is spent just grep’ing the entire package…

6 Likes
10

That is what I did suspected : module are equivalent to C++ namespace.
Large C++ project can involve many class but usually does not require many namespaces.

Class are replaced by struct/types and you pass from a single dispatch in C++(o.method) to multiple dispatch in Julia.

2 Likes
12

This is a great question. As you are discovering, many people will ignore what you wrote and insist that it’s best to dump all symbols into a single namespace. This is unfortunate, because it doesn’t really address the question at hand or move the discussion forward. It derails a discussion of possible solutions.

The bulk of threads like this become a series of comments like “having a single namespace (maybe split out some packages) is best”, “no its not”, “yes it is”. I find this terribly unfortunate. We all know that a single namespace is the status quo in Julia and that a lot of people prefer it this way. It’s their prerogative to continue to explore that direction. Can’t we leave threads like this to creatively discuss alternatives? I don’t think you are going to drown out the desire for something different by diluting threads with comments saying its an invalid desire.

6 Likes
13

Yes, I feel your pain. Discoverability is an issue in Julia, relative to let say, Python, not only because of that. Multiple dispatch and method extensions make things harder.

The up side of Julia is that packaging is very simple and can replace submodules with advantages (when the submodules make sense as providing separate clear functionalities).

3 Likes
14

I suspect that you can easily improve your workflow. I use a lot of files in my projects (included from the main module) and I have no navigation problem at all to jump from function to function defined in different files.

15

I use VSCode, and the “jump to definition” has at best a 60% success rate for me. If you have ideas as to how to improve my workflow I am certainly receptive. Also note I am not referring to code for which I am the writer. This pattern is reasonably navigable when you know the code and its structure very intimately already. The problem lies as a reader when I am trying to learn the internals of a new package

16

The OP did already a good research on the existing alternatives. And nobody is dismissing the question, it is just that people use the single namespace because it ends up being the most practical alternative. One can find 200+ message threads about this subject here. I don’t think that reviving those without bringing anything clearly new to the table will help anyone.

2 Likes
17

Yes, package manager in Julia is great, but packaging C++ class level of functionalities should not be required for a project to even work.

18

Well, it isn’t. There are many successful (large) packages in Julia. As I mentioned, the thing that in Julia modules are less useful than in other languages, because namespaces are less important because of multiple dispatch. Again, it is not that this doesn’t come with downsides, but trying to reproduce the workflow of C++ or Python in Julia will cause friction with the idiom and additional pain.

19

How many namespaces do you classically use in C++ project ?

20

Again, Modules in Julia are equivalent to C++ namespaces : not C++ classes.

I have a lot of structs and type hierarchies in my single module Julia projects.

3 Likes
21

Majority of packages in julia implement the monolith pattern (#4) which I strongly object. Files should be self contained and testable. I’ve used many languages over 20 years of my professional career and this is the first time I don’t get it.

1 Like