Package negligence

New to Julia
1

Hello.

I’m a 25-year veteran of Microsoft’s C#, and I’ve become very discouraged by the status of the Machine Learning (ML.NET) project at Microsoft. It seems to be dying, and developers are left holding an empty bag.

So, I’ve come over to Julia to see if life is better. My early experience was both good and bad.

Let me talk about the bad side of Julia. Like Python, Package neglect is a problem. For example, being interested in Smoothing Splines, I naturally looked at “SmoothingSplines.jl”. The GitHub REPO shows that this package has not been maintained in two years, and it can no longer be compiled by Julia 1.8.5.

My question to the Julia community is: “What is your policy on Package neglect”?

Charles

5 Likes
2

I installed SmoothingSplines on Julia 1.8.5 without any problem. Also the tests passed.

So in this case I do not see any problem. I am using Linux, though. What are you using?

Make sure you install it in a clean project, e.g.

mkdir Splines
cd Splines
julia --project="."

and then in Julia:

using Pkg
Pkg.add("SmoothingSplines")
Pkg.test("SmoothingSplines")
using SmoothingSplines

Does this work for you, or which step fails?

8 Likes
3

EDIT: TL;DR, there was a practical problem, but no more.

I’m surprised, since that’s claimed to be not true. Since you’re new, if the package has a Project.toml file it should work (I’m not aware of any packages for pre-1.0 Julia only, that are registered are registered). You can see in that file [compat] and there under julia = “1.0”. That doesn’t mean only 1.0 works, it refers to oldest version that that should work. And I’m not sure it’s even needed in that case (is 1.0 automatically implied?).

There are very rare examples where code doesn’t work in a later Julia (if relying on undocumented API), so except for that, it doesn’t matter if packages are no longer updated. They should work, but of course might have bugs that do not get fixed, then you could fork them.

An older system uses REQUIRE file, and it may also work, though maybe not, unless there’s alo a Project.toml file.

1 Like
4

Can confirm package works on 1.9.0-rc2.

Note that DataFrame indexing has changed since the example on that README was written. You would use X = map(Float64,convert(Array,cars[:,:Speed])) instead, for example.

1 Like
5

I’m not aware of any policy.

Perhaps 2 years isn’t too long a time for a package to go untouched - it might have been that good. But it’s perhaps something to investigate before you stake your business on it :thinking:

4 Likes
6

There’s no policy as there can’t be one, and it’s not needed [EDIT: There might be a need for a policy to take over packages, for future development, but in this case the repo was changed quickly, and in some strict sense it wasn’t broken as is, never is.]. All Julia code that ever worked, should continue to work forever (or at least until Julia 2.0). If people abandon packages, there nothing to do, since you can’t force people to work on open source. And you can’t drop packages from the registry either, that would break things.

Even code from before Julia 1.0 should work, but then you likely need the exact Julia version it was made for. In general you might also need the same exact dependencies of your package. If the Project.toml file is incorrect that is, otherwise newer non-breaking version of dependencies are ok.

That seems to mean DataFrames.jl has a newer breaking version. In that case you need to figure out the version from that time that works, if the package manager doesn’t do it for you, as it should. [A Manifest.toml files is also for that situation, I believe.]

2 Likes
7

I would say, think about the Heartbleed bug from years back. It wasn’t negligence, it was that two developers were responsible for code that basically the entire internet relied on for security. You get what you pay for in software. If no one is willing to pay to have someone maintain SmoothingSplines.jl, then why would they do it?

That’s not neglect, just a reality of people moving on to make a living.

3 Likes
8

Here is a screen shot of trying to run the SmoothingSpline,jl code that is on the GitHub page:

test
test1980×1416 211 KB

As you can see, lines 9 and 10 show a problem.

9

Lines 9 and 10 are merely possible call errors, that might also be the linter being too strict.
The actual error after ERROR Undef is the really interesting part :slight_smile:

PS: Welcome to the Julia Forum! :wave:

1 Like
10

True. But I just became a Sponsor. Was that a mistake?

1 Like
11

I’m running Windows 11. The latest release.

12

If you’re referring to (from the docs, a different error, not a compilation issue per se):

julia> X = map(Float64,convert(Array,cars[:Speed]))
ERROR: ArgumentError: syntax df[column] is not supported use df[!, column] instead

and what I got and:

then it could solved alternatively by:

pkg> add DataFrames@v0.21.7

Why that version number? I just chose some radum one from approximately the time frame of that package… The point I’m making is that you CAN run old code… but yes, you’re in a rather impossible situation (as a new user). It’s not that the package strictly depended on the older pre-1.0 DataFrames.jl, only its docs did… but it could have done so. I could run the tests successfully for the package with (both the old and the new) DataFrames.jl installed:

pkg> test SmoothingSplines

Now, if the code of the package itself would have depended on DataFrames, it should have been mentioned in the Projects.toml file (and the code in the docs would have worked).

In statically compiled languages you would have high (not full) confidence the code worked at runtime. In Julia the tests are meant for that, but they are not always sufficient.

I suppose the problem in the screenshot is similar, and installing an older version of some package should help.

1 Like
13

For me it runs, just by fixing the notation of the dataframe, i. e, these lines:

X = map(Float64,convert(Array,cars[!,:Speed]))
Y = map(Float64,convert(Array,cars[!,:Dist]))

(add the !, there).

In fact, if you run that line by line in the REPL, you get a message explaining that change.

Thus, what is outdated there is the example in the README.

1 Like
14

In this case it is simpler. The fix is just that the authors have to update the example in the README file. (ps: PR created)

15

Hmmmm… Well as a brand-new user, I can tell I have a lot of work to do. Silly me, I thought this would be easy.

4 Likes
16

For me the docs work (see my previous comment how, still the DataFrames version seems not to be the cause):

julia> predict(spl, 20.0)
59.82139857976935
17

Don’t worry, I’m sure you’ll face much harder problems than this one :blush: . Welcome to the forum!

6 Likes
18

It turned out to be because of DataFrames as I explained, and I wouldn’t say it was non-trivial for me as somewhat experienced with Julia (people give me way more credit than that, I wouldn’t claim Julia expert myself).

Note, as soon as you get any error, you want to look at the first error you get (this applies to any programming language), here for this line:

X = map(Float64,convert(Array,cars[:Speed]))

not start with the subsequent errors, that (here) all stem from it, e.g. not look at that (or last) error:

julia> predict(spl, 20.0) 
ERROR: UndefVarError: `spl` not defined

The fix is (relatively) easy, while the debugging the root cause (for new users) may not be… On the plus side, the Julia community is often helpful though.

To solve this for working with the latest DataFrame version, a PR to the package (docs) is needed as mentioned, but it also needs to be merged… and if the package is actually abandoned, then a policy of taking over may be needed(?). I would propose e.g. two years if enough for it to be justified by anyone taking over… The problem could happen for any arbitrary short time though. But even with other packages having braking changes (as can happen for other languages too), and spoiling things for other (older) packages, in practice, I want to stress there’s always a solution; at least having the same package environment (of dependencies).

19

I did make a PR for this (I saw other PRs not yet merged, and actually, another user made the same PR just now…, and it was merged).

Neither of us can make the PR be merged, so it wouldn’t have been sufficient to fix the problem. Until then, and since the registry doesn’t point to either of our forks, it’s still useful to know you can refer to those forks (had this been a problem with the actual code of the package not just its docs), even after them outdated, now, this way:

pkg> add https://github.com/PallHaraldsson/SmoothingSplines.jl
3 Likes
20

Everything fixed, PR merged: GitHub - nignatiadis/SmoothingSplines.jl: Cubic smoothing splines in Julia.

5 Likes