The State of the Julia Ecosystem

I am extremely thankful to the core developers for developing the new package manager Pkg3 which helps a lot in terms of easy install and reproducibility. A couple of text files (Project.toml and Manifest.toml) are sufficient to reproduce a work environment with a one-line command (I actually wish I had it when developing the code for my PhD paper, which is on Julia 0.6 - I am afraid to touch anything in my Julia 0.6 install as it could be quite tricky to restore the current configuration…).

I’m completely ignorant as to the functioning of JuliaPRO and whether or not it includes any packages, but I would like to make the case that including packages by default is almost unnecessary now: one could simply create curated Project.toml and Manifest.toml files and the user could instantiate them and have a “batteries included” environment.

3 Likes

I am not sure what a “standard download” is. But you can start the notebook with a cell that gets your project and instantiates everything.

Unless you do this, I expect it is inevitable that some packages experience at least minor syntax changes and break code within a few months (at most). But this problem is orthogonal to having a curated set of packages or similar ideas.

If this is the problem that prompted this topic, perhaps it would have been better to start with a request for help (“I am writing a book, how can I ensure that code runs?”) instead of a somewhat meandering criticism of the 1.0 release process, Julia Observer, scoping, Julia Pro, etc.

3 Likes

If I may read between the lines here, and please take this constructively, you seem a bit disgruntled because you were an early Julia adopter and things have changed rapidly—more rapidly than you feel comfortable with. Julia 1.0 was released faster than you would have liked, and contained one change that you really didn’t care for—global scope behavior. Packages that you grew fond of, such as Winston, have been abandoned—for years now. Why not switch to one of the many excellent, actively maintained plotting packages instead of remaining unhappy about Winston’s lack of maintenance?

The global scope “debacle” was so-called by @jeff.bezanson himself in his typical, charmingly self-deprecating style, recognizing that many people—but not all of them, and perhaps not even most—are vocally unhappy about it. Frankly, if that’s the only thing people have to complain about in 1.0—which seems very close to being the case—then we really stuck the landing. Given the number of very hard decisions that went into the 1.0 release, getting only one thing wrong is a spectacular track record. There’s a proposed solution for global scope, which we’re not rushing out the door because botching the fix would be worse than not fixing what many people have argued not to be a mistake in the first place.

The package ecosystem is now largely recovered, which is pretty impressive given the extremely short time frame. As @ssfrr has already said, I don’t think that would have happened without lighting the fire under the proverbial asses of package developers by putting 1.0 out there when we did.

I agree that package search and discovery remains an issue. The https://pkg.julialang.org page is abandoned because the person who created and maintained it decided it was no longer something they wanted to maintain. Julia Observer is unofficial, and while Dan has done a great job with it, I worry that it also has a bus factor of one. What we need is a solution which guarantees active and ongoing support and maintenance. We do have a plan, which is under active development, so bear with us on the package search front. In the meantime, the version bounds in the General registry are accurate: if you want to know what’s compatible with Julia 1.0, it’s all in there. The biggest remaining usability problem is that the Pkg version resolver gives pretty inscrutable errors when no compatible versions exist. That too will be improved in the near future.

Taking a step back, I’d like to encourage you to change your perspective. Rather than looking back and being annoyed by things you grew used as a very early Julia adopter which have changed or broken, instead look forward. The era of uncomfortably rapid change is over: the things that work well in Julia now are going to continue working—we guarantee it. We have also successfully made the transition from an old, traditional style package manager that gave no real ability to separate projects, snapshot or reproduce work, or have multiple public and private registries, to a new package manager that was designed to do all of those things and do them well. It hasn’t been a painless transition by any means, nor is it complete—package users are living in the promised land, but package developers are still awkwardly straddling the old and new worlds (but not for long).

From where I’m standing, the future looks very bright. Come join me up here :smiley:. Thank you for all your past and present support and evangelization of Julia.

48 Likes

Stefan

Thanks for your lengthy response.

Firstly I’m not disgruntled and there is no need for me to sign-up, I did that 5 years ago. It is true that I was not pleased at the GSD, and although I am not a frequent contributor to forums such as this did in fact raise the issue, immediately I encountered it. I am pleased to hear that it has been taken on board.

The main concern I expressed in the initial posting was that by not amending http://pkg.julialang.org, not getting NewPkgEval up and running, and using Julia Observer as a primary reference you do yourselves less than the justice than you deserve. I am aware of the great amount of work that has gone into the development of Julia and still continues.

However the information provided from the above on the Ecosystem is less than useful and I hope will be rectified soon.

I deal with people who view Julia from the boundary, not on the field, and I am trying to ensure that they do not go off and find a different boundary and a different game.

I’m confused about this claim you’ve made a few times now. NewPkgEval has been working since before 1.0 and has regularly been used to check for package breakage since then. That’s how the issue listing 27 packages with failures on 1.1-DEV that I linked to above was generated and that’s how we make sure the point releases 1.0.1-1.0.3 didn’t accidentally break anything.

To get constructive here: what do you think we should do differently at this point? Is having a better way of discovering packages the main thing?

I understand why you might be frustrated, as writing a book when the target is moving must be difficult. However, I think it was all worthwhile as from my “user” perspective 1.0 is eminently usable and joy to program in.

As a user I don’t necessarily see all the problems with the “ecosystem” as my needs are relatively straightforward and I only see a limited neighborhood within the ecosystem. So that perspective might also be something to incorporate into evangelizing talks, perhaps?

By the way, I really liked your previous book on Julia. I hope the new one is going to be as good! Best wishes.

Petr

So where can a see the results of the latest run of NewPkgEval, I personally have had difficulty in finding this from the main website.

I think I have discussed what is helpful to me, I wish to tell punters how to download Julia for their favoured platform, find and install a set of packages that concern them and start to use the language constructively.

I do, on the other hand, think the documentation you have provided is excellent and praise this whenever I can. Working on documentation can be a mind-bogglingly tedious task.

In fact writing this edition is virtually a total rewrite, as I keep telling the publisher when they ask me why progress is so slow.

I’m glad you enjoyed my first attempt, there are large parts of it I’m dissatisfied with and would like to make a better stab at it this time; in particular the last chapter which is pretty poor and I hope the muse grabs me when (if?) I eventually get there.

1 Like

This is not run regularly, published or definitive—we do not have the resources to continuously CI the entire package ecosystem. The version bounds in the registry should be accurate: if a version claims to be compatible with 1.0 in the registry then you can presume that it is.

I think I have discussed what is helpful to me, I wish to tell punters how to download Julia for their favoured platform, find and install a set of packages that concern them and start to use the language constructively.

So the original post was primarily asking about what version of Julia and packages to recommend? JuliaPro is still a good starter option. The default registry is the same as the set of packages that were previously shipped with it and those are known to work well with the shipped Julia.

It would also be possible to create a list of packages that have some version that is compatible with 1.0+ if that would be helpful. These days I think that people are mostly just trying to install packages and of the resolver lets you do it you’re good to go, otherwise the package is not yet compatible with 1.0.

So in fact NewPkgEval is not run regularly.

The last version (of the spreadsheet) that I found was not timestamped and still had JuliaDB marked as skipped.

As I said if J-Pro installed a bundle of packages (as per v0.6.x) then this woud be helpful.

And yes it would be extremely useful to have a list of packages which work with v1.0+ and perhaps if it could replace the page I’ve reference to a couple of times in this blog - which suggests that v0.6.3 remains the best gal on the block.

Yes, it is run regularly but manually—specifically before releases to make sure that they are non-breaking as guaranteed. You seem to be misunderstanding the purpose of PkgEval. It is not some ecosystemwide CI system; it is for the Julia devs to evaluate if we have broken packages. Packages are responsible for their own CI: when a package makes a release, it should run CI to make sure that it works with the versions of Julia that it claims to work with and either fix any breakages or update the compatibility claims in the registry accordingly. I’m not sure where you got the idea that PkgEval was for public consumption—it is not. Sure, we will post results because it’s not secret information, but it is really only relevant to Julia developers, not Julia users.

There were a lot of complaints that the JuliaPro download was too big, so we have changed the way it works. Now it ships without preinstalled packages but with a registry that lets you easily install the JuliaPro packages and which prompt if you want to stray off of “the guaranteed path” and use non-JuliaPro packages. If you install all of the package in the JuliaPro registry, that’s equivalent to the old JuliaPro.

This information is available in the registry, it should be simple to write a script that pulls out packages that have some version that’s compatible with 1.0+, I guess I could sit down and do that.

4 Likes

It’s not difficult to argue that Julia users would find this information relevant also. You make fair points, it’s not your responsibility to be constantly testing all those packages, since packages are doing their own CI, but it would still be relevant information to Julia users regardless, since Julia users are some kind of developer.

1 Like

I want to point out that the NewPkgEval.jl as well as the old PackageEvaluator.jl (which is the old code that generated the pkg.julialang.org page) are both completely open source with MIT licenses. Anyone can run NewPkgEval! It’s actually a fairly small package. Of course, it does take a non-trivial amount of computation resources. It also looks like it could use a Readme with a quick-start guide, but what better way to get involved than helping out with that?

I would also absolutely love to see the pkg.julialang.org site updated and set up to run with regular updates. Indeed, it should be much easier to do and maintain now that Julia is at 1.0 and has stability guarantees.

Now, @sherrinm, please note that I am not suggesting that you be the one to do this work and I know that the fact that it’s open source does not help you directly. But I do imagine there are folks looking for ways to get more involved in the project — and I can say that doing the work to update PackageEvaluator.jl to lean on NewPkgEval and update its site regularly would be quite appreciated.

4 Likes

The code is public. Anyone who has some available hardware or cloud credits to spend and wants to run this regularly and publish the results is very welcomed to do so.

2 Likes

It would be nice if I could do testing of my own package via a CI cron job, and have it submit a report to pkg.julialang.org automatically to update the listings, similar to codecov. Distributed workload.

Can you please share what the plan is? The reason I’m asking is that during the development of the current Julialang.org I was discussing with @viralbshah about a website in the line of https://rubygems.org, https://hex.pm, https://www.scala-lang.org, etc. With a REST API as well. Unfortunately, we didn’t have time to do this for v1 so we scraped it (we even had a nice package search box in the design of the JuliaLang home page). I was planning on starting a side-project in this direction, but I would rather not do it if other work is being already undertaken.

5 Likes

pkg.julialang.org is a luxury — we only have it because one person decided to make it, and we are not entitled to that person’s continued efforts for free. You make it sound like not updating pkg.julialang.org was some kind of official decision. Regularly running PkgEval will also require a large amount of compute resources. Well, in fact we are discussing all of this and share your goal of getting it all working nicely. But it will take time, effort, and money. In the future, when posting about things like this I just ask that you phrase it more like “Hi, I noticed that pkg.julialang doesn’t seem to be updated. Are there any plans to do so? It would be great to have it running again.”

Posting this without any supporting data is disheartening to see.

9 Likes

Here’s a script that generates a CSV file with the name of each package and true or false indicating whether it has a version that is compatible with some 1.x Julia release:

import Pkg
import Pkg.TOML
import Pkg.Types: VersionSpec
import UUIDs: UUID

function compatible(pkg::String, version_spec::String)
    vers = VersionSpec(version_spec)
    registry_path = joinpath(DEPOT_PATH[1], "registries", "General")
    package_index = TOML.parsefile(joinpath(registry_path, "Registry.toml"))["packages"]
    compatible = Dict{String,Bool}()
    for (uuid_str, info) in package_index
       uuid = UUID(uuid_str)
        name = info["name"]
        path = joinpath(registry_path, info["path"])
        compat_file = joinpath(path, "Compat.toml")
        isfile(compat_file) || continue
        compats = Pkg.Compress.load(compat_file)
        compatible[name] = any(compats) do (ver, compat)
           !isempty(vers ∩ VersionSpec(compat[pkg]))
        end
    end
    return compatible
end

open("tmp/compat_julia_1x.csv", "w") do io
    println(io, "package,compatible")
    compats = compatible("julia", "1.0-1.1")
    for (name, compat) in sort!(collect(compats), by=lowercase∘first)
        println(io, "$name,$compat")
    end
end

The output of running this right now is here:

https://gist.github.com/StefanKarpinski/0a83266ca3db15dc5cb3b7b2d98da02d

8 Likes

I realise the NewPkgEval is open-source and infact pulled down a copy at the weekend with the thought of running a copy. However while engaged on my 2nd edition it did strike me that this might be more appropriate as a job undertaken by the Julia Computing organisation.

I am interested in the idea of creating a script to pull out information from the registry - I believe that were once a Perl Monger (?), as am I, and probably such a script might be an approach I would take. Perhaps you can send me some information in a private communication

I am sorry to dishearten you.

If you no longer have any use for pkg,julialang,org, then it should be retired.

I believe that there have been specific enquires as to the future of this particular page, which was only one aspect of my posting - of which I took a great deal of care and feel is entirely fair.

I come from the land of Milton, Shakespeare and Fergus McGonigal and am sorry if my prose style is closer to the third of these Great British poets than the first two.