Using Julia for "large" scale projects?

New to Julia
1

Hello community,

I have rarely written relatively small (up to 500 lines of code) Julia programs and the long compilation time has prevented me from writing large scale programs. I have now decided to try writing a longer program (e.g. >10,000 lines of code; >50 files).

Which approach and which project structure would you recommend to me? It is important for me to optimize the compile time and to get “readable” error messages from the compiler. With Julia REPL I could not get comfortable but maybe I am using REPL wrong.

Regards,
Christoph

2

Which approach and which project structure would you recommend to me?

Regarding latency, latency does not depend on project structure. There is quite a bit you can do yourself to reduce latency: I made a list of recommendations here, and the performance tips in the Julia manual also has a section on mitigating latency.

At this time, in my opinion, it’s too difficult as an ordinary Julia user to get acceptable latency. Julia’s idioms guide your code towards high latency, and to the casual user, it’s unclear what is good, standard Julian coding practise, and what is optimising for the current implementation details of the compiler. Nonetheless, I recommend the following steps, in brief:

  1. Reduce the number of dependencies you take on, where possible
  2. Do not commit type piracy
  3. Write completely inferrible code
  4. Ditch your dependencies that violates points 1, 2 or 3
  5. Use PrecompileTools.jl to add a @compile_workload block for your package.

These are the most important things, and they will usually be enough. Then there are some more minor tricks:

  1. Do not use __init__.
  2. Be careful about excessive specialization. In particular, do not use compile-time constructs for runtime information (don’t use tuples when the length or element type is not known at compile time, don’t use Val when its value is not known at compile time)
  3. Use package extensions for optional support for third-party libraries
2 Likes
3

Highly biased answer:

  • Use VSCode as your IDE. It is the best supported and has many nice features for Julia development.
  • Use the TestItems framework for Julia in VSCode. This persists your test environment which makes the tests run quickly. Which means you can run tests frequently. I run tests after every change of more than a few lines of code, maybe 100 times/day. It catches dumb errors fast.
  • Put your code in a project. The other tools in the Julia ecosystem work better if your code is organized this way.
  • When you generate a project by default there will be a single file in the src/ directory with the same name as your project. Break your source code into logically consistent pieces and put each piece in a separate .jl file. Then in the project file do this:
module YourPackageName

include("File1.jl")
include("File2.jl")
include("File3.jl")
...
end #module

This structure works well up to several thousand lines of code. Once you get beyond that you will probably want to create subdirectories in the src/ directory to hold logically related source files.

For small projects I would advise against creating submodules:

module TopLevelModule

module SubModule1
#code here
end #SubModule1

module SubModule2
#code here
end #SubModule2

end #TopLevelModule

My experience has been that this makes for a more difficult to use API and is not worth the extra complexity for small projects. Maybe it makes sense when your projects get much bigger, say 20,000 lines or more.

6 Likes