Experience report after finishing a (reasonably substantial) Julia project in 2024

[rongcuid]

I use Pluto extensively in my research projects, but in this case it doesn’t scale.

I am thinking about going the OCaml way and abstract using modules:

module TypeA
  struct T ... end
  method_a(a::T) = ...
end
...
import TypeA
...
a = TypeA.T(a)
TypeA.method_a(a)
...

This is supposed to allow me to modify T and reload the whole module. I haven’t done it in practice.

[jules]

@rongcuid

Plus I can’t do italics + bold.

That’s done with font = :bold_italic for the default font face, mentioned here Fonts · Makie (you have to pick font faces, not layer bold + italic, because many fonts have multiple weights anyway).

@rongcuid

Maybe if there’s an intermediate level API which allows me to, say, draw each character directly

That exists but isn’t public, it’s called GlyphCollection and can be passed to text Makie.jl/src/types.jl at f51eb527806e184690b577e17e630e9c9c521cb9 · MakieOrg/Makie.jl · GitHub

And I’m not sure about the line spacing problem you mention, in principle that should be adjustable. Although could be that rich text doesn’t do this right now, I don’t remember

[danielwe]

@rongcuid

in this case it doesn’t scale

Hence the suggestion to integrate Pluto into your workflow at the prototyping stage of a module or type, rather than moving your entire project into a notebook. Place import Pkg; Pkg.activate(Base.current_project()) at the top of the notebook to make it run in the environment you’re developing in such that you can easily prototype on top of existing project code. And with Pluto.run(auto_reload_from_file=true), you don’t even have to leave your editor, Pluto can just be a monitor for the state of your code.

Just throwing this out as one possible way of working around the struct redefinition limitation. Of course, you may find that other workarounds suit you better, such as your suggestion of wrapping a module around each new type.

[rongcuid]

I didn’t make the connection between the rich text documentation under text() and the fonts documentation.

I totally didn’t see GlyphCollections when I dug through Makie’s code to look at whether I could work around the rich text issues. I searched for rich (which was where the problem was reported), landed on Makie.jl/src/basic_recipes/text.jl at f51eb527806e184690b577e17e630e9c9c521cb9 · MakieOrg/Makie.jl · GitHub, and then had completely zero idea where the entry point is by looking at this piece of code.

You see, every single parameter of every type is written in, you guess it, (args...; kwargs...), so I had a lot of trouble navigating Makie’s code base. Now that you mention the existence of GlyphCollection, I found the entry point (plot!, I suppose) immediately at the top of the file lol.

[rongcuid]

Ok. I don’t remember Pluto could do it, but maybe I had memories of old versions.

[mnemnion]

I’ll address one of your sections as a sort of jumping-off point to responding to the spirit of the whole.

@rongcuid

Enums

I used the @enum macros a few times in my code.

Don’t use them, really. They are not namespaced, there’s no multi dispatch, and there’s no pattern match. Abstract classes are more powerful.

Julia enums are much like C enums: they’re numbers which you can refer to by name. They are nothing like Rust or Java enums at all, so if you expect them to be, you’re going to have a bad time.

Julia has one namespacing mechanism: modules. Anything in the top scope of a given module shares a single namespace, enums included. For enums, I tend to do what we do in C, and use a prefix, so a VM I’m working on, the enums for instructions are all IAny, IReturn, and so on.

Is that ideal? It depends on what you’re working on, I suppose, or what you’re used to, or both. If you want a namespaced enum, EnumX.jl provides this. It uses a module, in fact.

You can use enums for dispatch, although how to do it isn’t at all obvious. Using the example enum from the documentation, here’s how.

julia> @enum Fruit apple=1 orange=2 kiwi=3

julia> f(::Val{apple}) = "I'm an apple"
f (generic function with 1 method)

julia> f(::Val{orange}) = "I'm an orange"
f (generic function with 2 methods)

julia> f(::Val{kiwi}) = "I'm a kiwi"
f (generic function with 3 methods)

julia> f(fruit::Fruit) = f(Val(fruit))
f (generic function with 4 methods)

julia> f(apple)
"I'm an apple"

julia> f(kiwi)
"I'm a kiwi"

If you’re using enums in the common fashion, where they get written as constants, directly into the text file, then this will be fast. If the enums are being passed around in variables, you’re better off dispatching in a different way. Using a collection of zero-width subtypes is another option, and it might be a better one, depending on what you’re doing.

At least, I think that using an enum by name is treated like a constant, not a variable, by the compiler. It’s difficult to be sure of these things.

Pattern matching: there are a few packages, the obvious one is named Match.jl, and it works fine with enums. It is not exhaustive, you won’t get an error if you don’t match all branches. That’s pretty normal with Julia, but for enums specifically, which can’t be extended, it would be possible to write an exhaustive pattern-matching macro which threw an error if you didn’t handle all cases. I’m not aware of any, however.

The point of my answer here, however, is that this is a pervasive problem with the ecosystem. How is a beginner supposed to know you can dispatch on enums using the Val pattern? There are breadcrumbs in the manual, but that’s it.

I think it’s great that someone who wants namespaced enums can write a macro and make it a package, and that there several competing implementations of pattern matching based on macros as well. I think less things should be in stdlib, not more.

But discoverability is really rather bad. And it isn’t great that something as basic as pattern matching doesn’t have a clear winner which everyone uses.

This is a combination of the size of the user base and the essential complexity of the language. The only way out is much better supporting documentation, and more of it, and the way to get that is to keep growing and stimulate enthusiasm in the user base. There aren’t easy answers here.

Meanwhile, getting to know Julia requires some determination to trawl through juliahub and old Discourse threads. It is simply harder to figure things out than it is in many other languages, and that needs to change.

[rongcuid]

I used prefixes:

@enum SkillRarity begin
    SkillNormal
    SkillRare
    SkillEvolved
    SkillSpecial
end

And didn’t find out there was no pattern matching until I implemented 20 skills in ~800 LoC, finished the whole simulation, and finally needed these enums for visualization. So in the future I will treat them like C enums.

@mnemnion

The point of my answer here, however, is that this is a pervasive problem with the ecosystem. How is a beginner supposed to know you can dispatch on enums using the Val pattern? There are breadcrumbs in the manual, but that’s it.

This is my first time seeing a Val pattern. I have thought about if it could be done because some other lisps could do it, but I never found it despite actively searching for it in the official documentation, especially through the methods and dispatching docs.

[gdalle]

@brianguenter

Can we add a prominent link on the Julia language home page to ModernJuliaWorkflows ? Just like the Rust and Go examples? And then do our best as a community improve it and keep it to date, as we do with the rest of the Julia documentation?

Co-creator of ModernJuliaWorkflows here: contributions are always welcome!
The initial goal was to contribute the posts to the Julia blog, but instead maybe our website deserves a life of its own? I could definitely see it becoming a less formal, more hand-holding companion to the detailed documentation. Of course, putting a link to MoJuWo in the important links section would make it seem more official than it is… but in Julia world there really isn’t such a thing as “official”, there is only “useful to the community”.
If you’re wondering why all this stuff is not already in the docs, see this previous discussion:

Write documentation PRs not blog posts

I will start by saying I have read many useful blog posts and course notes that have been linked on this forum, and I greatly appreciate the effort that goes into making them. The problem is they are not discoverable. It is my opinion that these guides would be much better served as a “Tutorial” or “Usage Guide” section in the official documentation for each package. I would like to have a discussion about that stance and what we can do to improve discoverability of fundamental knowledge for new…

[rongcuid]

Val is in fact, not in the documentation:

https://docs.julialang.org/en/v1/manual/methods/

It’s there in the API:

https://docs.julialang.org/en/v1/base/base/#Base.Val

However, I will never find it if I haven’t read the entire Base API documentation.

[mnemnion]

@rongcuid

And didn’t find out there was no pattern matching until I implemented 20 skills in ~800 LoC, finished the whole simulation, and finally needed these enums for visualization. So in the future I will treat them like C enums.

Well, and that’s the thing: there is pattern matching, but it uses Julia’s excellent metaprogramming facility, so it’s a library. Several, in fact. But everywhere else, it’s a core part of the language. This isn’t quite the dreaded Lisp Curse, but it’s related.

@rongcuid

This is my first time seeing a Val pattern. I have thought about if it could be done because some other lisps could do it, but I never found it despite actively searching for it in the official documentation, especially through the methods and dispatching docs.

There is a section on value types, but it doesn’t connect them with enums in any way.

In larger languages you’d search for “Julia enums” and find a blog post called, like, “Julia enums: more than you ever wanted to know” which lays it all down on the table for you. That search does yield some interesting stuff, including a sum-types project SumTypes.jl, which I’d completely forgotten about, and uses the macro system to make Rust-style enum types.

But there isn’t an exhaustive treatise. It’s the size of the community, more than anything.

[rongcuid]

@mnemnion

There is a section on value types, but it doesn’t connect them with enums in any way.

Good note. I was looking at “Methods” page all that time.

I’ve came across SumTypes.jl in this project. I deemed it overkill at the time and thought nothing about it.

[CameronBieganek]

You can get a certain level of pattern matching just by using dispatch and argument destructuring:

foo((x, )::Tuple{Any}) = x
foo((x, y)::Tuple{Any, Any}) = x, y
foo((; a)::NamedTuple) = a

julia> foo((1, ))
1

julia> foo((1, 2))
(1, 2)

julia> foo((a=10, b=20))
10

(The type annotations for the arguments are necessary because otherwise the argument would be typed Any and each definition would overwrite the previous one.)

[ericphanson]

@rongcuid

Another problem is that when reading another person’s code, especially on GitHub or other places where an LSP is not available, it’s very difficult to find where a symbol is defined: if I am reading b.jl, anything there could be defined in ModuleA.jl or a.jl.

This doesn’t really help with reading other people’s code, but for your own code, it sounds like you might like my ExplicitImports.jl package. It helps convert using Foo into using Foo: x, y, z, based on whatever names x, y, z, you are actually using in your code.

[rongcuid]

This is not good for research code, because I am refactoring so frequently that I cannot imagine myself maintaining not only an export list, but also an import list.

If Julia’s LSP is as good as rust-analyzer or IntelliJ Java, then we are talking.

[ericphanson]

Yeah, it is mostly useful once the code is stable, so you can convert to an explicit list at that point.

[indymnv]

Nothing to add, but just show appreciation for the nice article you wrote. Hope this kind of feedback can help to improve Julia.

[liuyxpp]

@rongcuid

Another problem is that when reading another person’s code, especially on GitHub or other places where an LSP is not available, it’s very difficult to find where a symbol is defined: if I am reading b.jl, anything there could be defined in ModuleA.jl or a.jl.

I’ll just straight up say that it’s worse that #include in C. In C, there’s at least forward declaration that allows me to break cycles without shuffling stuff between files. Unfortunately, Julia doesn’t have that.

I strongly agree with this. It is so painful to read other’s codes just in Github.

@rongcuid

Makie is very powerful and very fast (once warmed up), but not mature. I know it well enough to navigate its documentation, but when I started out I had no idea how to read its documentation. Especially problematic is all those kwargs used in its API, which require a lot of digging to figure out. I think its documentation needs some reorganization. It is, however, a very powerful library, allowing me to make very complex plots.

And this one is also frustrating for me.

[dylanxyz]

@rongcuid

Naming is hard [citation needed]. Let me reiterate: use snake_case for functions and PascalCase for types. Please don’t use nocase, even though the official docs recommend it.

Well, i personally find that it is more convenient to mix different conventions, and julia makes that easy:

Dot case? no problem

julia> var"foo.bar"() = println("_ are overrated, use . instead")
foo.bar (generic function with 1 method)

julia> var"foo.bar"()
_ are overrated, use . instead

For people who prefer the _ to be centered with the text:

julia> var"foo-bar"() = println("like snake_case, but aligned with the text")
foo-bar (generic function with 1 method)

julia> var"foo-bar"()
like snake_case, but aligned with the text

This is also a good one:

julia> var"foo bar"() = println("use ' ' as separator, because why not?")
foo bar (generic function with 1 method)

julia> var"foo bar"()
use ' ' as separator, because why not?

And my personal favorite, for when i started learning programming with CSS, and had to use double-dash naming convention for classes:

julia> var"my--very--long--variable--name" = nothing

[rongcuid]

You might have noticed already… but all of these are better than nocase.

myverylongvariablename = nothing

[merlin]

Sevi’s next post has this. I think you mean something like poetry specifying dev dependencies for python packages, for example I use the ptpython repl in my development workflow in python but dont want this as part of my package dependency.

So yes, this is easy in the Julia REPL as Sevi’s post shows.

I like to keep several named ‘global’ environments, these are stored by default in the ~/.julia/environments folder. For example mine looks like this:

ls -1 ~/.julia/environments/
__pluto_boot_v2_1.7.1
__pluto_boot_v2_1.9.2
data-work
stats-plotting
v1.0
v1.1
v1.4
v1.5
v1.6
v1.7
v1.8
v1.9

I handle dev dependency additions in my startup.jl file, but for me in REPL it would be:

] activate ~/.julia/environments/data-work
] add CSV TerminalPager Chain
julia> using CSV, TerminalPager, Chain
] activate .
julia> rand(100, 100) |> pager

I should also add that CSV.jl is extremely well supported.