Intermediate Development Workflow

New to Julia
1

Hi Everyone, I am relatively new to Julia, I’ve been using it for a few months. In the first month, I was able to write several thousand lines of code and solve some problems that took far longer in other languages. I was amazed.

For the last month, I have been trying to break my code into modules and see what a production system might look like. Before, I was using a flat folder with includes, which made rapid tinkering easy. Now I am moving things into modules, but those modules are still changing substantially. I tried a project, but that infrastructure didn’t seem to help development, just final packaging. So, after a month of rapid work, I feel blocked trying to iterate on module structure, and system architecture.

So, my question. I am beyond writing a few simple files and scripts, but not at the project publishing phase. I have 5 modules now, and will have 10+ once I refactor my flat prototype. But, I’ve spent A LOT of precious time, just battling scaffolding so that I can work in an agile way and refactor my module designs, and rapidly test the integration etc. Are there resources that explain how to work in this intermediate phase efficiently. I’m stuck here. I want be able to build and test multiple modules, and work in the way a user would to run the code quickly in an “eat your own dog food” way, with a fast feedback loop between coding and using the new code.

Thanks in advance for any help or resources. It seems like this has to be a solved problem, but most of what I see is for simple files or one module, or how to set up projects and push to github. I’m in the middle of those phases.

4 Likes
2

So currently, you have a single package with multiple modules? Your project’s file structure looks somewhat like

ThePackage/
 - Project.toml
 - src/
    - ThePackage.jl
    - module_a.jl
    - module_b.jl
 - test/
    - runtests.jl

Is that correct in principle? If so, can you be more specific about the shortcomings?

3

Julia provides three basic levels of organization for code that can used as tools, depending on what you’re after.

The module is a special kind of block that creates a global scope, and establishes namespacing using the dot notation. You’re always in some module, often in Main.

The project (environment) is a pair of files, Project.toml and Manifest.toml, that list a set of mutually compatible dependencies. The two files sit together in the same directory, which becomes the name of the project. The project need not contain any code. I have some scripts in /usr/local/bin that begin with lines like

#!/usr/local/bin/julia --project=/home/lee/juliaprojects/awesomeproject

using HTTP, Dates

     etc.

When I execute this script it will load the versions of HTTP and Dates specified in the Manifest.toml file in the given project directory.

The package is a project with two additional things: a file defining a module within a src directory and a few lines at the beginning of the Project.toml file with the package’s version, UUID, etc. You can generate all this quickly using the generate command in package mode.

If you want to be able to add the package you need to keep it in a Git repository, which you should be doing anyway. But you can dev a package that’s not under version control. Whether you dev or add a package depends on whether you want to use code from commits or from your filesystem.

If you’re putting your code in modules you’re probably intending to import that code into other code, so you might as well keep each module in its own package, which keeps everything neat and tidy. You can add or dev these local packages by referring to their paths, so you need not bother fiddling with LOAD_PATH.

It’s all pretty simple thanks to the package mode and commands like generate. And working this way makes it easy to share your projects later.

4 Likes
4

It is not so clear to me why you want to use multiple modules. Doing that is not well supported by the Julia tooling.

I would either use a Julia project (see: Working with Julia projects | Julia programming notes ), or one or more Julia packages.

If you have one package you can structure it using multiple includes.

Using multiple packages is a little bit of a pain, because you need to use Pkg.develop() for the sub-packages if you want to edit them and see the result immediately in the main package.

Julia 1.12 will make it easier to use “workspaces” that consist of multiple packages: 10. Project.toml and Manifest.toml · Pkg.jl

1 Like
5

This is a slightly edited version of a response I gave on another thread. Since you talk about improving your development workflow this answer includes suggestions beyond just structuring modules, folders, and files.

  • Use VSCode as your IDE. It is the best supported and has many nice features for Julia development.
  • If you have not already done so add a startup.jl file. Create a new directory config in your .julia directory. Put a file startup.jl in that directory. Here is a simple example:
Startup file 
using Pkg

let
    pkgs = ["OhMyREPL", "Revise"]
    for pkg in pkgs
        if Base.find_package(pkg) === nothing
            Pkg.add(pkg)
        end
    end
end

using OhMyREPL
using Revise

ENV["JULIA_EDITOR"] = "code.cmd"

Configure VSCode:

  • Set VSCode to save files on focus change. Then when you make a change in VSCode by the time you’ve moved the cursor to the VSCode julia REPL Revise will have detected the change and cause your code to recompile to be consistent with the text in the VSCode editor.

  • Set VSCode to format files on save. Cuts down on trivial github updates.

  • Use the TestItems framework for Julia in VSCode. This persists your test environment which makes the tests run quickly. Which means you can run tests frequently. I run tests after every change of more than a few lines of code, maybe 100 times/day. It catches dumb errors fast.

  • Put your code in a project. The other tools in the Julia ecosystem work better if your code is organized this way.

For small projects I would advise against creating submodules:

#Don't do this unless absolutely necessary to avoid namespace clashes

module TopLevelModule

module SubModule1
#code here
end #SubModule1

module SubModule2
#code here
end #SubModule2

end #TopLevelModule

My experience has been that this makes for a more difficult to use API -“was that TopLevelModule.foo() or TopLevelModule.Submodule1.foo()?” - and code that is harder to modify and update. It is not worth the extra complexity for small projects (<5000-10,000 lines of code).

Maybe it makes sense when your projects get bigger. Although even then I would be inclined to put those submodules in their own package.

2 Likes
6

A practical suggestion is to use PkgTemplates.jl to create the structure of a package.

I don’t get what the actual question/problem is. You say you’re “battling scaffolding” and are “stuck”, but didn’t say what any of the actual issues are?

To organize names into tidy name spaces. Which also makes it possible to use shorter names, as otherwise often the namespace is implicitly embedded in the name.

Works for me. EDIT: ah, actually I think I know what you mean. It seems like the current LSP (a new one is being developed, hopefully it fixes this) doesn’t know about bindings that are both in a different file and in a different module. Yeah, that sucks, although it’s just an annoyance.

IMO that sucks (without submodules). Even though it’s the most popular option in the Julia world.

How is that painful? You just do dev once and that’s it, in my experience.

1 Like
7

Here’s the customary pointer to https://modernjuliaworkflows.org/

6 Likes
8

Thanks to all who replied here, and sorry for my glacial response. I should have given more explanation of the difficulties I encountered, and thought I would be back on this thread a few hours after posting but got pulled away.

I have used Julia for only two months, and don’t know anybody else who uses it, so I should have asked question here earlier. @barucden I have tried a few different workflows to go from my early prototype with flat folder of files with no modules. I transitioned to a project, with multiple modules and submodules. And, wow. I did not know what I didn’t know and my first attempt at structuring the architecture started out with what @brianguenter and @ufechner7 cautioned against. Not only was I battling name space issues, every time I thought a package might be useful, and I tinkered with using it, it added it to my project, before I was certain I needed it and my project is a mess. As a noobie, I was also misusing include and using in a way that was not a good idea. I figured that out the hard way but get it now.

Right now, I have moved to using a folder of modules. No packages yet, and no projects. I have a good understanding of the module architecture at this point, but I am want an expressive consistent API between them, so interfaces are changing as I work on the design (which means tests would change, docs would change etc.) So, I have high level system tests that flex the system, but only targeted unit tests where the interfaces have stabilized.

I am using VS code with startup.jl and love it, and I need to internalize the link @gdalle posted: My high level go-forward plan is

  1. Refine the module APIs using iterative development and high level system tests for testing and flexing the system
  2. Once Module APIs stabilize, create a package for each module, unit test, doc
  3. Once packages are in good shape include in a project for distribution

(perhaps steps 2-3 should be in parallel, not sure). Not sure where to fit in optimize and test for differentiability probably end of step 1.(this is a physics simulator to be that I hope to integrate with other tools in the ecosystem).

Make sense?

9

As you already have a folder of modules, what would speak against converting the modules to packages right now?