Lost in transition: the current state of the Julia ecosystem?

General Usage
1

I really don’t know what to think about the current state and direction of the Julia ecosystem…

I’ve been holding back support for Julia v0.6 in Genie because some dependency packages were not yet v0.6 compatible. I was hoping that their maintainers would upgrade them soon. No luck.

But instead:

  1. since the release of v0.6, every few days I’ve been checking for updates with Pkg.update(). At the same time I kept working on Genie on the v0.5 branch so I’ve been checking for package updates there too. To my surprise, at least for the packages that I’m using, I see more updates on the v0.5 branch than on the v0.6!

  2. even low level, core packages that are developed by the JuliaComputing contributors (for example @Keno’s Gallium and AbstractTrees) aren’t updated yet. AbstractTrees is a dependency of a few packages that Genie is using and I made a PR with the v0.6 compatibility. Even more, I’ve seen the v0.6 support is already coded by @Keno but the package has not been tagged. What’s happening?

  3. at the same time I see quite a lot of talk about v0.7. Really, are we there yet? Cause to me it looks like we’re still limping transitioning from v0.5.

Thus, I’m reaching out in the hope of finding some answers…

  1. does anybody have more info than me regarding the status of packages like AbstractTrees? @Keno?

  2. I’m ready to go ahead and fork every single package that Genie needs and is not updated, and maintain my own versions. But is this the way to go? AbstractTrees is a dependency of Gallium - is it not going to get tagged? Why aren’t things properly communicated so we know what to expect?

  3. I hope that the perspective is different depending on the packages you’re using. Does anybody have a rosy view of things? Cause from where I’m standing it looks like the ecosystem is de facto still at v0.5. Anybody had a positive experience with the migration to v0.6? Any tips for the rest of us?

Thanks

1 Like
2

What packages specifically? The only package that I know of that isn’t v0.6 compatible is Knet.jl (technically Autograd.jl?) because of the broadcast change affecting their autodifferentiation. The ecosystem is definitely not de facto v0.5: it’s clearly v0.6 with many packages even dropping support for v0.5. Is the only package you’re talking about here AbstractTrees.jl, or are there more?

Why not just submit a PR to them? If you submit a few PRs many times (especially if it’s in an org) you’ll just be invited to be a contributor and help maintain the project. There are some libraries which have no devs that do need this, but most at least do version updates.

3
1 Like
4

If I’m reading this correctly, that’s just to one of the many libraries? I’m now sure how a PR to one of the libraries leads to the conclusion that many other (unnamed) libraries need to be forked.

5

My suggestion is copy (provided the license is compatible) the package as a sub-module and fix it as needed. Once the original package is fixed, then you can delete your copy.

My rule of thumb is that The “0.6” in Julia means “0.6”: it’s alpha- quality software and it’s important to manage your expectations on Julia itself, as well as the ecosystem. Once it’s 1.0 ones expectations may be different.

6

Sorry about that. There’s a lot on my plate, so sometime routine package maintenance falls by the wayside. I’ll try to look at this tomorrow.

1 Like
7

@ChrisRackauckas Well, this is embarrassing… but you’re right.

About two months ago I put my repos on hold waiting for updates on the dependencies. But until now, I haven’t actually looked at the GitHub repos. Indeed, with the exception of AbstractTrees and Gallium, they have all been updated. I don’t know why I didn’t get the new versions in Pkg.update(), maybe I messed up something in METADATA.

Now I’ve done a clean install of Julia v0.6 and successfully added all the packages from scratch.

I sincerely apologize, I didn’t mean to step on anybody’s toes. (But at the same time I’m quite happy that I opened up my mouth and got this thing working again, I really didn’t know what else to do!)

Thanks everybody for your help and sorry again for wasting your time! :flushed:

5 Likes
8

@Keno Thank you - I’m not directly using AbstractTrees. But it was a dependency of Gumbo, which is a key package in the view layer of Genie, hence my frustration.

But it turned out that Gumbo removed AbstractTrees as a dependency over a month ago and tagged a new release. But for some reason, I didn’t get it in Pkg.update(). However, it’s all good now - fresh install of Julia and everything worked perfectly again.

Sorry for the commotion

9

Actually, the problem with Pkg persists even on a new install :zipper_mouth_face:

Does anybody know why I ended up with
- AbstractTrees 0.0.4

if pkg.julialang.org show it as
Julia v0.6: 0.1.0 (6 months ago)

and Gumbo, the package that’s using it, has in its REQUIRE
AbstractTrees 0.1.0

10

Similarly, Gumbo itself is at v0.3.0 on Julia v0.6, despite the fact that pkg.julialang.org lists

Julia v0.5: 0.3.0 (7 months ago) /  Tests pass. 
Julia v0.6: 0.4.0 (a month ago) /  Tests pass.

How can this be possible on a brand new Julia v0.6 install?

julia> versioninfo()
Julia Version 0.6.0
Commit 903644385b (2017-06-19 13:05 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin13.4.0)
  CPU: Intel(R) Core(TM) i7-6820HQ CPU @ 2.70GHz
  WORD_SIZE: 64
  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Haswell)
  LAPACK: libopenblas64_
  LIBM: libopenlibm
  LLVM: libLLVM-3.9.1 (ORCJIT, skylake)
11

OK, I did a new clean install starting with a clone of Genie. It turns out that that’s how Pkg can resolve the current requirements, by choosing lower versions - especially Gumbo 0.3.0 instead of Gumbo 0.4.0 (which in turn depends on AbstractTrees 0.0.4 instead of AbstractTrees 0.1.0).

Attempting to force Gumbo to 0.4.0 in Genie results in a dependency tree that can’t be satisfied.

So, in regards to my initial query: how is a developer expected to deal with this? If I wait for packages to be updated but Pkg skips updating them because one package breaks the dependency tree, how do I even know new packages versions were released?

If Pkg would show a notice and say:

There is a newer version of Gumbo at 0.4.0 but it can't be upgraded due to unsatisfied dependencies.

at least I would know that some packages have been updated and I need to dig deeper into the dependency tree.

12

So something is forcing Gumbo to downgrade. What does Pkg.dependents("Gumbo") show

2 Likes
13

IIUC, the following shell command should point to packages with dependency version ceilings:

egrep '[-.0-9]+ [.0-9]+' $HOME/.julia/v0.6/*/REQUIRE

Perhaps this will help you find the culprit.

1 Like
14

I dug into the Gumbo/AbstractTrees/ASTInterpreter issue a while ago and tried to document as specifically as I could why the downgrade was happening. Probably related to the issues you’re seeing:

https://github.com/Keno/ASTInterpreter.jl/issues/69#issuecomment-310210108

For my use-case the issue was more or less resolved with Atom stopped depending on ASTInterpreter/Gallium.

15

This happens to me routinely. I asked why, but did not get replies yet. I moved my own packages out of ~/.julia/v0.6 so that I can wipe and reinstall it every month or so, and I am in the process of figuring out how to run tests for them.

“Running Pkg.update() routinely” turned out to provide the illusion of being up to date. Some packages were 2-3 (!) minor versions behind.

16

Sorry, eventually I had to go to sleep :slight_smile:

@avik @ssfrr Yeah, in my case it’s a Gallium > ASTInterpreter > AbstractTrees > JuliaParser dependency structure that holds back the Gumbo > AbstractTrees versions.
Removing all the dependencies on Gallium made Pkg update to latest versions.

@Tamas_Papp It would be helpful if Pkg3 would show a warning if a package that is not pinned can’t be upgraded due to dependencies. It can tip off the developer to go down the dependency graph and try to solve it manually.

17

Have you tested Pkg3? Following Stefan’s talk on juliacon i’d understand as: Pkg3 enables you to hold different package versions in parallel if needed.

18

Not thoroughly and not recently, it was very early version last time I checked.

That sounds cool - do you have more details about what does that mean?

Will it allow having multiple versions of, say, AbstractTrees and use one with Gallium and one with Gumbo but not in the same app? Or will it allow a package to encapsulate/namespace its dependencies so that in the same app multiple versions will coexist, say Gallium.AbstractTrees@0.0.4 and Gumbo.AbstractTrees@0.1.0?

19

OK, i think the design still follows:
https://github.com/JuliaLang/Juleps/blob/master/Pkg3.md
and i really recommend the video from juliacon (this year). Part of the demonstration is failing, but it’s visible what it could be.

20

One of the issues seems to be that many packages are one-person efforts, and when that person is busy, development stalls. Then later on many issues get fixed in bursts.

Also, some packages are still hammering out the ideal architecture/API, and my impression is that they want to hold off on other things in the meantime.

1 Like