Experience report after finishing a (reasonably substantial) Julia project in 2024

It does.

Another alternative is to create a shared environment that contains the project-specific development dependencies, with, for example:

(MyProject) pkg> # MyProject environment activated 

(MyProject) pkg> activate --shared MyProjectDevelopment
  Activating new project at `~/.julia/environments/MyProjectDevelopment`

(@MyProjectDevelopment) pkg> add StaticArrays
   Resolving package versions... 
   (...)


julia> using StaticArrays # MyProjectDevelopment activated here

(@MyProjectDevelopment) pkg> activate . # activate MyProject
  Activating new project at `~/Downloads/MyProject`

julia> x = SVector(1,1) # StaticArrays is available
2-element SVector{2, Int64} with indices SOneTo(2):
 1
 1

This if one has very project-specific development dependencies (I almost never do this). For benchmark, revise, etc, just use the global environment.

8 Likes

this is exactly what I do, with @dev

and I have the same problem as OP — I do not know a way to get it loaded into the VSCode repl by default. it’s quite annoying.

I guess I could set a startup.jl to push it into the LOAD_PATH, but that pattern just always bothers me as looking too ugly.

They are not easy to find, but you can debug from the VSCode REPL with @enter or @run.

Also, see this blog post for information on stacked environments. They can be used conveniently for debugging like this.

I share some of your frustrations but hope to help fix from within.

10 Likes

Maybe something like this does the trick, in the startup.jl, after installing the necessary development dependencies in the shared development environment:

if occursin("MyProject", Base.active_project())
    import Pkg
    Pkg.activate("MyProjectDevelopment", shared=true)
    using StaticArrays
    Pkg.activate("MyProject")
end

Test:

MyProject% julia --project
  Activating project at `~/.julia/environments/MyProjectDevelopment`
  Activating new project at `~/Downloads/MyProject/MyProject`
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.10.2 (2024-03-01)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> x = SVector(1,1)
2-element SVector{2, Int64} with indices SOneTo(2):
 1
 1

(MyProject) pkg>

Thank you. I will definitely use them next time.

Actually I just looked at SciML’s docs. I have used components of it, like diffeq, symbolics, flux, etc. Even Plots and Makie is part of it. Many of these packages are well documented.

Biggest problem for me is Symbolics, really. It has never successfully solved anything I fed to it that is not toy examples. The same systems of equations could be solved by SymPy, and was solved by hand. For whatever reason I had the impression that Symbolics.jl was not active, but I just looked at its repo and see that is is active… I don’t know what happened to my memory.

I definitively can understand and partially share some of these concerns/pain points. I’ll need to think a bit more about all of that.

For now let me point out two things that are incorrect:

First:

As written this is false. The local assignment in some_other_function does not overwrite the global curvature. In your case you must have done something differently.

Second:

This also seems wrong as written. In contrast to C include is not a macro replacing itself with the contents of another file but a function that evaluates some julia file. As such that Julia file needs to be syntactically correct. E.g. you cannot include the following:

module WontInclude
foo(x) = x^2
#no end for module
7 Likes

Hmmm, I just went on to test them again. I don’t remember what happened during development.

Maybe it’s just accidental local shadowing breaking a large function for the first case.

The second one could be breaking only the same file. I must have remembered incorrectly.

It doesn’t have very many symbolic equation solvers right now. It has more on the guts than that aspect of this moment, similar to SymEngine in status. But it’s a very active place right now.

1 Like

This was a good writeup! Thanks for sharing your experiences.

7 Likes

Not sure if it’s been mentioned here, but for functions julia does have forward declarations:

function foo end

bar(x) = foo(x)

foo(x) = 2x

should work as expected.

1 Like

It’s mostly the types that are the problems. I solved them by:

  • Pulling type definitions all the way up to the top of a module
  • Try to break up dependencies with intermediate types (like ControllerObservation)

Still require me to juggle multiple files, though.

A good takeaway from this discussion is that many of the problems @rongcuid had were the result of bad documentation. Bad doesn’t mean incomplete; it means hard for a new user to understand and navigate. The information is there, somewhere, but important ideas about how to set up and use your Julia workspace for maximum efficiency are not salient.

Two examples of this are the problems @rongcuid had understanding how to stack environments in the REPL and how to run the debugger on a single function from the REPL.

When beginners encounter these documentation problems they assume they are language or ecosystem problems. Then they write about their frustrations which discourages other new users from trying Julia. And us more experienced users feel Julia hasn’t gotten a fair shake.

I have never understood why there isn’t a getting started link on the Julia language home page the way Rust and Go have done

Right there, at the top of the home page where newbies can’t possibly miss it. ModernJuliaWorkflows is essentially this getting started page but I wager that few, if any, new users come across this.

Can we add a prominent link on the Julia language home page to ModernJuliaWorkflows? Just like the Rust and Go examples? And then do our best as a community improve it and keep it to date, as we do with the rest of the Julia documentation?

37 Likes

The project sounds very interesting. I didn’t know there were people writing racing game simulations in Julia. Just curious. Was it a personal project, academic project, or corporate project?

I don’t think, however, that these are disconnected. Getting the hang of an effective Julia workflow isn’t trivial, because of the nature of the language. Even when there is someone taking one by the hand, there is a initial steep learning curve, because many of the good practices of an optimal workflow are related to somewhat advanced uses of the language. Like the code structure, the persistent REPL, using environments effectively. Efforts like the modern Julia workflows are great, but as we can see from that page, the issue is that the amount of information quickly becomes a barrier that only goes out with experience.

(What I’m not sure is how different this is in other languages, my experience teaching with compiled languages is also full of first-use complications - possibly R and Matlab have an advantage there - I’m guessing - for having dedicated IDEs)

1 Like

It’s personal project. I have this random-number driven forum novel that’s coming to an end, and I say “screw it, you say my rule set is too complicated, now I write a program to fking simulate the final race with symplectic Euler one second per turn for 300 turns”.

So this is where the “one-off” for the project comes from. I only need it to run once (of course actually I run it many times, but it must be deterministic).

Spoiler alert, though it’s not like any of you are likely to go to a Chinese forum to read 500 thousand words:

Visualization, spoiler alert!

8 Likes

In general, if Makie has a problem, it is impossible to work around, because its API is wrapped in so many layers of abstraction.

Probably meaning “hard to workaround by yourself” (which I do relate to) ?
Because on the contrary it seems always possible to workaround Makie issues
(it is rapidly improving):
the support here on discourse is outstanding (especially thanks @jules !)
and the very low level control achievable is awesome.

1 Like

Thanks so much for posting this; this is indeed good feedback and a really cool project! Writing a 3000LOC simulator to top-off a Lord Of The Rings length forum novel is a beautiful thing.

As the discussion progresses here, I will be trying to split off concrete sub-threads for further discussion as it’s sensible. So let’s continue the discussion on Arrow and Parquet in Challenges with Arrow and Parquet in a (reasonably substantial) Julia Project.

21 Likes

Makie support is very responsive.

However, if you look closely you see that the skill lists on the right has huge line spacing and pops the layout all over the place. This is a bug and cannot be worked around. It also has problems handling empty lines.

In this case, RichText is abstracted a bit too high level to engineer a local fix. Maybe if there’s an intermediate level API which allows me to, say, draw each character directly, that issue could be worked around.

Plus I can’t do italics + bold.

Pluto.jl lets you redefine structs without restarting the Julia process, which can be useful when prototyping and iterating on struct layouts, even if the eventual home of the code will be a regular module and not a notebook.

2 Likes