I’m a 25-year veteran of Microsoft’s C#, and I’ve become very discouraged by the status of the Machine Learning (ML.NET) project at Microsoft. It seems to be dying, and developers are left holding an empty bag.
So, I’ve come over to Julia to see if life is better. My early experience was both good and bad.
Let me talk about the bad side of Julia. Like Python, Package neglect is a problem. For example, being interested in Smoothing Splines, I naturally looked at “SmoothingSplines.jl”. The GitHub REPO shows that this package has not been maintained in two years, and it can no longer be compiled by Julia 1.8.5.
My question to the Julia community is: “What is your policy on Package neglect”?
EDIT: TL;DR, there was a practical problem, but no more.
I’m surprised, since that’s claimed to be not true. Since you’re new, if the package has a Project.toml file it should work (I’m not aware of any packages for pre-1.0 Julia only, that are registered are registered). You can see in that file [compat] and there under julia = “1.0”. That doesn’t mean only 1.0 works, it refers to oldest version that that should work. And I’m not sure it’s even needed in that case (is 1.0 automatically implied?).
There are very rare examples where code doesn’t work in a later Julia (if relying on undocumented API), so except for that, it doesn’t matter if packages are no longer updated. They should work, but of course might have bugs that do not get fixed, then you could fork them.
An older system uses REQUIRE file, and it may also work, though maybe not, unless there’s alo a Project.toml file.
There’s no policy as there can’t be one, and it’s not needed [EDIT: There might be a need for a policy to take over packages, for future development, but in this case the repo was changed quickly, and in some strict sense it wasn’t broken as is, never is.]. All Julia code that ever worked, should continue to work forever (or at least until Julia 2.0). If people abandon packages, there nothing to do, since you can’t force people to work on open source. And you can’t drop packages from the registry either, that would break things.
Even code from before Julia 1.0 should work, but then you likely need the exact Julia version it was made for. In general you might also need the same exact dependencies of your package. If the Project.toml file is incorrect that is, otherwise newer non-breaking version of dependencies are ok.
That seems to mean DataFrames.jl has a newer breaking version. In that case you need to figure out the version from that time that works, if the package manager doesn’t do it for you, as it should. [A Manifest.toml files is also for that situation, I believe.]
I would say, think about the Heartbleed bug from years back. It wasn’t negligence, it was that two developers were responsible for code that basically the entire internet relied on for security. You get what you pay for in software. If no one is willing to pay to have someone maintain SmoothingSplines.jl, then why would they do it?
That’s not neglect, just a reality of people moving on to make a living.
If you’re referring to (from the docs, a different error, not a compilation issue per se):
julia> X = map(Float64,convert(Array,cars[:Speed]))
ERROR: ArgumentError: syntax df[column] is not supported use df[!, column] instead
and what I got and:
then it could solved alternatively by:
pkg> add DataFrames@v0.21.7
Why that version number? I just chose some radum one from approximately the time frame of that package… The point I’m making is that you CAN run old code… but yes, you’re in a rather impossible situation (as a new user). It’s not that the package strictly depended on the older pre-1.0 DataFrames.jl, only its docs did… but it could have done so. I could run the tests successfully for the package with (both the old and the new) DataFrames.jl installed:
pkg> test SmoothingSplines
Now, if the code of the package itself would have depended on DataFrames, it should have been mentioned in the Projects.toml file (and the code in the docs would have worked).
In statically compiled languages you would have high (not full) confidence the code worked at runtime. In Julia the tests are meant for that, but they are not always sufficient.
I suppose the problem in the screenshot is similar, and installing an older version of some package should help.
It turned out to be because of DataFrames as I explained, and I wouldn’t say it was non-trivial for me as somewhat experienced with Julia (people give me way more credit than that, I wouldn’t claim Julia expert myself).
Note, as soon as you get any error, you want to look at the first error you get (this applies to any programming language), here for this line:
X = map(Float64,convert(Array,cars[:Speed]))
not start with the subsequent errors, that (here) all stem from it, e.g. not look at that (or last) error:
julia> predict(spl, 20.0)
ERROR: UndefVarError: `spl` not defined
The fix is (relatively) easy, while the debugging the root cause (for new users) may not be… On the plus side, the Julia community is often helpful though.
To solve this for working with the latest DataFrame version, a PR to the package (docs) is needed as mentioned, but it also needs to be merged… and if the package is actually abandoned, then a policy of taking over may be needed(?). I would propose e.g. two years if enough for it to be justified by anyone taking over… The problem could happen for any arbitrary short time though. But even with other packages having braking changes (as can happen for other languages too), and spoiling things for other (older) packages, in practice, I want to stress there’s always a solution; at least having the same package environment (of dependencies).
I did make a PR for this (I saw other PRs not yet merged, and actually, another user made the same PR just now…, and it was merged).
Neither of us can make the PR be merged, so it wouldn’t have been sufficient to fix the problem. Until then, and since the registry doesn’t point to either of our forks, it’s still useful to know you can refer to those forks (had this been a problem with the actual code of the package not just its docs), even after them outdated, now, this way: