Discussion on "Why I no longer recommend Julia" by Yuri Vishnevsky

I have been fortunate to have recently moved to a Software Engineer role from being a research academic while being employed at an institution willing to invest in open source infrastructure. We recently launched an Open Science Software Initiative:

We have a ways to go to raise the standards for academic open source software, but I do think we are starting to some funding for this purpose. That said Julia’s package and Github architecture make this a lot easier. We just need to expand the tooling as we gain collective experience with Julia.

15 Likes

As such, when iterating over an entire array, it’s much better to iterate over eachindex(A) instead of 1:length(A) . Not only will the former be much faster in cases where A is IndexCartesian , but it will also support OffsetArrays , too.

Ok, I guess if that was a part of the official documentation, then that’s the solution – we should consider 1:length(A) as bad practice, or at least highlight eachindex(A) as the best practice, especially in conjunction with @inbounds.

I’ve just summited PRs to change this to “notes” and “warnings” in the docs (this info was not visible enough imho). Maybe the docstrings in @inbounds could be edited as well…

7 Likes

I mean, it technically was always as dangerous as unsafe_load, obvious under the assumption that --check-bound=no, but that is the default.

I just filled an issue to deprecate it and change it to @unsafe_inbounds.

3 Likes

This would be breaking. It would probably need to be a Julia 2.0 feature if implemented.

Sorry, I forgot to mention that it would be deprecated, still usable but with a warning about using the new name. :sweat_smile:

2 Likes

What exactly should an interface promise? One level could be that of type signatures that have to be available for certain functions. So for example an AbstractArray would have to show that you can call size() on it and receive let’s say a tuple of integers. But type signatures of functions, while already quite useful if they’re being automatically checked, don’t save you from the OffsetArrays issue mentioned here, which is more about the logical interface. And this seems much harder to pin down, and test automatically in generic fashion.

I for one would already like a type-signature-checking interface option. Maybe just implemented via macros so you can say @declare_interface_compatible SomeInterface MyType and then it’s checked that a bunch of previously declared function signatures can be correctly inferred when using MyType. And if not it would error and say “MyType doesn’t fulfill SomeInterface because of …”.

And where you use interface functions they could check that the types being used have been compile-time checked for interface compatibility. Maybe with Tricks.jl or something like that.

4 Likes

ouch …
checking the public github repos with the new (beta) “GitHub code search”

https://cs.github.com/ AND with:

  • language:julia "=1:length("
  • language:julia "1:length("
  • language:julia "for i=1:length("
  • language:julia "@inbounds for i=1:length("

there are many hits…

6 Likes

It will be interesting to check how many are left in a couple weeks. Thousands of bugs solved all at once!

2 Likes

I mean, it depends on the code.

I have used that pattern in a lot of my research code, but that code does the data analysis for my papers, and it isn’t meant to be used with any types other than what I have written and therefore tested (usually just Array).

I would imagine a lot of the hits you see are in the same boat. The issue here is that StatsBase is an extremely popular package that can be reasonably assumed to be called with any AbstractArray.

4 Likes

I would assume variations of in 1:length is more common than =1:length. There’s also = 1:length,, = 1 : length, etc.

2 Likes

Since almost everyone starts to know the danger of @inbounds for i = 1:length(v) and OffsetArrays. Here’s some additional advice based on my JuliaImages experience:

https://github.com/JuliaArrays/OffsetArrays.jl/pull/281


But try to think broadly, Yuri doesn’t mean to blame the particular OffsetArrays or Zygote packages – he tries to point out that the Julia ecosystem/community don’t yet have a good tooling/mechanisim to allow non-experts build large but reliable Julia package – that’s all I understand.

16 Likes

thanks for the tips!

limiting the search:

org:julialang

  • org:julialang language:julia "1 : length"

    • JuliaLang/julia‎ : 3 files
      • repo:julialang/julia language:julia "1 : length"
    • ‎JuliaLang/Pkg.jl‎
  • org:julialang language:julia "1:length"

    • JuliaLang/julia‎
    • JuliaLang/Compat.jl‎
    • JuliaLang/IJulia.jl‎
    • JuliaLang/JuliaParser.jl‎
    • JuliaLang/Pkg.jl‎
    • JuliaLang/TOML.jl‎
    • JuliaLang/MbedTLS.jl‎
    • JuliaLang/METADATA.jl‎

org:JuliaComputing :

  • org:JuliaComputing language:julia "1:length"
1 Like

I believe that some people find lots of bugs in Base and the ecosystem. I must use Julia in a different way because I very rarely run into bugs. I don’t recall ever finding one in Base, but I might have forgotten something. So whether Julia is usable might depend on the kind of user. Julia has generally been very reliable for me. But, if it isn’t for important segments of users, that’s a big problem.

These past few days I could have really used “zero based” Array indexing. But, I knew it would probably be more trouble than its worth. I kinda thought it was general knowledge that , while Julia has some support for this, it’s nothing you can rely on in general. I mean this is a feature, not a core feature, that is not really well supported (in practice; the tools are there). There are issues of broader import like interfaces.

Poorly defined and enforced or tested interfaces is a big problem. It hasn’t been a pain point in my work. But, it clearly needs a big shift in culture/tools.

I started using JET and Aqua. They found bugs . This is great. My goal is to use them in all my packages. Everyone should use tools like these. Almost no one does.The registrator complains about all kinds of things, but not that I haven’t passed some analysis requirements (eg JET)

35 Likes

First of all, please don’t do this here. Trolling and flamebaiting are not welcome

Secondly, most of issues in the blog-post were identified and fixed long before the blogpost was ever posted. The community takes fixing bugs pretty seriously, and the forum is a big part of how they coordinate and approach that work.

27 Likes

I think this brings up some other good points.

Yuri, and others who report similar issues, ran into these problems at least partially because they were developers of packages that try to innovate within the ecosystem. Such package development will naturally find the corner cases.

There are reasons I don’t recommend Julia to some people but bugs are not one. There’s a lot of software I use that is difficult to even get working on windows let alone design novel packaging on top of without creating many bugs

2 Likes

I tend to disagree with the OA on this – I don’t want to blame the entire language/ecosystem because of silent bugs involving OffsetArrays and @inbounds.

I still think OffsetArrays are a bad idea, though, and I don’t see the point for them.

In my own code, I usually need to iterate over a matrix, and I normally do things like:

for i ∈ 1:size(A,1), j ∈ 1:size(A,2)
    A[i,j] = sin(x[i]) * cos(y[j])
end

which is very succinct and easy to understand… and if I want to traverse the diagonal of A, this is simply:

for i ∈ 1:size(A,1)
    A[i,i] = A[i,i] + 1
end

To support OffsetArrays, I should do:

for I ∈ CartesianIndices(A)
    A[I[1],I[2]] = sin(x[I[1]]) * cos(y[I[2]])
end

Which is ok. But right now, I don’t know how to traverse the diagonal of a matrix in a way that supports OffsetArrays.

There seems to be an explanation in the OffsetArrays.jl home page about how to use Diagonal from the LinearAlgebra package (which looks like it’s converting the OffsetArray to a standard array), but what really caught my eyes was:

Certain libraries, such as LinearAlgebra, require arrays to be indexed from 1

ERROR: ArgumentError: offset arrays are not supported but got an array with index other than 1

I believe that would be the quickest/wisest solution, just make more base libraries incompatible with OffsetArrays.

Unless, of course, somebody kindly illustrates what is the real need for OffsetArrays, I think that should be the best solution for most packages.

8 Likes

it’s just a package, there’s no “bad idea”, if you don’t like handle it in your package, don’t. You don’t even have to make a warning, because Julia never made the promise that because the multi dispatch would allow you to run generic code on previously unknown types, it would work correctly logically speaking. One just have to know what they’re using / doing – like in any language

9 Likes

I’m sympathetic to Yuri’s larger point: Julia as a language and ecosystem is permissive by default. You can easily combine packages in ways that are unlikely to work and get mired in the weeds. For example, I can easily imagine attempting to differentiate a distributed SVD of a BlockedArray filled with Unitful Quaternions. Sure, that’s an absurd example and it’s unlikely to work, but I can imagine how it might work and may even be able to construct the problem statement in a few lines of code. And heck, if I’m intrepid, I think I could make it work.

That’s also the promise of Julia: you can (sometimes) take two unrelated packages that know nothing about each other and the combination suddenly provides powerful new functionality. Often with no code changes. And it’s amazing.

But sometimes a package doesn’t quite do what the other expects. It’ll typically just error, but OffsetArrays are a very obvious and salient example where it can result in silent corruption due to @inbounds.

53 Likes