# The State of the Julia Ecosystem

**URL:** <https://discourse.julialang.org/t/the-state-of-the-julia-ecosystem/18650>\
**Category:** Community\
**Created:** [December 14, 2018, 9:18am UTC](https://discourse.julialang.org/t/the-state-of-the-julia-ecosystem/18650 "2018-12-14T09:18:23Z")\
**Posts on this page:** 20\
**Page:** 3

<div class="post-metadata">

**Author:** ![piever](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/piever/32/1815_2.png) [@piever](https://discourse.julialang.org/u/piever)\
**Post date:** [December 14, 2018, 3:48pm UTC](https://discourse.julialang.org/t/the-state-of-the-julia-ecosystem/18650/41 "2018-12-14T15:48:31Z")

</div>

I am extremely thankful to the core developers for developing the new package manager Pkg3 which helps a lot in terms of easy install and reproducibility. A couple of text files (Project.toml and Manifest.toml) are sufficient to reproduce a work environment with a one-line command (I actually wish I had it when developing the code for my PhD paper, which is on Julia 0.6 - I am afraid to touch anything in my Julia 0.6 install as it could be quite tricky to restore the current configuration…).

I’m completely ignorant as to the functioning of JuliaPRO and whether or not it includes any packages, but I would like to make the case that including packages by default is almost unnecessary now: one could simply create curated Project.toml and Manifest.toml files and the user could instantiate them and have a “batteries included” environment.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [December 14, 2018, 3:57pm UTC](https://discourse.julialang.org/t/the-state-of-the-julia-ecosystem/18650/42 "2018-12-14T15:57:07Z")

</div>

> [@sherrinm](#):
>
> However it is my intention, for the time at least, to ship the code as a series on Jupiter notebooks / Juno scripts, and these need a clean _standard_ download for the readers to work with.

I am not sure what a “standard download” is. But you can start the notebook with a cell that gets your project and instantiates everything.

Unless you do this, I expect it is inevitable that some packages experience at least minor syntax changes and break code within a few months (at most). But this problem is orthogonal to having a curated set of packages or similar ideas.

If this is the problem that prompted this topic, perhaps it would have been better to start with a request for help (“I am writing a book, how can I ensure that code runs?”) instead of a somewhat meandering criticism of the 1.0 release process, Julia Observer, scoping, Julia Pro, etc.

---

<div class="post-metadata">

**Author:** ![StefanKarpinski](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stefankarpinski/32/24_2.png) [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)\
**Post date:** [December 14, 2018, 3:59pm UTC](https://discourse.julialang.org/t/the-state-of-the-julia-ecosystem/18650/43 "2018-12-14T15:59:37Z")

</div>

If I may read between the lines here, and please take this constructively, you seem a bit disgruntled because you were an early Julia adopter and things have changed rapidly—more rapidly than you feel comfortable with. Julia 1.0 was released faster than you would have liked, and contained one change that you really didn’t care for—global scope behavior. Packages that you grew fond of, such as Winston, have been abandoned—for years now. Why not switch to one of the many excellent, actively maintained plotting packages instead of remaining unhappy about Winston’s lack of maintenance?

The global scope “debacle” was so-called by @jeff.bezanson himself in his typical, charmingly self-deprecating style, recognizing that many people—but not all of them, and perhaps not even most—are vocally unhappy about it. Frankly, if that’s the _only_ thing people have to complain about in 1.0—which seems very close to being the case—then we really stuck the landing. Given the number of very hard decisions that went into the 1.0 release, getting only one thing wrong is a spectacular track record. There’s a proposed solution for global scope, which we’re not rushing out the door because botching the fix would be worse than not fixing what _many_ people have argued not to be a mistake in the first place.

The package ecosystem is now largely recovered, which is pretty impressive given the extremely short time frame. As @ssfrr has already said, I don’t think that would have happened without lighting the fire under the proverbial asses of package developers by putting 1.0 out there when we did.

I agree that package search and discovery remains an issue. The [https://pkg.julialang.org](https://pkg.julialang.org) page is abandoned because the person who created and maintained it decided it was no longer something they wanted to maintain. Julia Observer is unofficial, and while Dan has done a great job with it, I worry that it also has a bus factor of one. What we need is a solution which guarantees active and ongoing support and maintenance. We do have a plan, which is under active development, so bear with us on the package search front. In the meantime, the version bounds in the General registry are accurate: if you want to know what’s compatible with Julia 1.0, it’s all in there. The biggest remaining usability problem is that the Pkg version resolver gives pretty inscrutable errors when no compatible versions exist. That too will be improved in the near future.

Taking a step back, I’d like to encourage you to change your perspective. Rather than looking back and being annoyed by things you grew used as a very early Julia adopter which have changed or broken, instead look forward. The era of uncomfortably rapid change is over: the things that work well in Julia now are going to continue working—we guarantee it. We have also successfully made the transition from an old, traditional style package manager that gave no real ability to separate projects, snapshot or reproduce work, or have multiple public and private registries, to a new package manager that was designed to do all of those things and do them well. It hasn’t been a painless transition by any means, nor is it complete—package users are living in the promised land, but package developers are still awkwardly straddling the old and new worlds (but not for long).

From where I’m standing, the future looks very bright. Come join me up here 😃. Thank you for all your past and present support and evangelization of Julia.

---

<div class="post-metadata">

**Author:** ![sherrinm](https://avatars.discourse-cdn.com/v4/letter/s/a88e4f/32.png) [@sherrinm](https://discourse.julialang.org/u/sherrinm)\
**Post date:** [December 14, 2018, 4:20pm UTC](https://discourse.julialang.org/t/the-state-of-the-julia-ecosystem/18650/44 "2018-12-14T16:20:14Z")

</div>

Stefan

Thanks for your lengthy response.

Firstly I’m not disgruntled and there is no need for me to sign-up, I did that 5 years ago. It is true that I was not pleased at the GSD, and although I am not a frequent contributor to forums such as this did in fact raise the issue, immediately I encountered it. I am pleased to hear that it has been taken on board.

The main concern I expressed in the initial posting was that by not amending [http://pkg.julialang.org](http://pkg.julialang.org), not getting NewPkgEval up and running, and using Julia Observer as a primary reference you do yourselves less than the justice than you deserve. I am aware of the great amount of work that has gone into the development of Julia and still continues.

However the information provided from the above on the Ecosystem is less than useful and I hope will be rectified soon.

I deal with people who view Julia from the boundary, not on the field, and I am trying to ensure that they do not go off and find a different boundary and a different game.

---

<div class="post-metadata">

**Author:** ![StefanKarpinski](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stefankarpinski/32/24_2.png) [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)\
**Post date:** [December 14, 2018, 4:30pm UTC](https://discourse.julialang.org/t/the-state-of-the-julia-ecosystem/18650/45 "2018-12-14T16:30:54Z")

</div>

> [@sherrinm](#):
>
> not getting NewPkgEval up and running

I’m confused about this claim you’ve made a few times now. NewPkgEval has been working since before 1.0 and has regularly been used to check for package breakage since then. That’s how the issue listing 27 packages with failures on 1.1-DEV that I linked to above was generated and that’s how we make sure the point releases 1.0.1-1.0.3 didn’t accidentally break anything.

> [@sherrinm](#):
>
> I deal with people who view Julia from the boundary, not on the field, and I am trying to ensure that they do not go off and find a different boundary and a different game.

To get constructive here: what do you think we should do differently at this point? Is having a better way of discovering packages the main thing?

---

<div class="post-metadata">

**Author:** ![PetrKryslUCSD](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/petrkryslucsd/32/215825_2.png) [@PetrKryslUCSD](https://discourse.julialang.org/u/PetrKryslUCSD)\
**Post date:** [December 14, 2018, 5:00pm UTC](https://discourse.julialang.org/t/the-state-of-the-julia-ecosystem/18650/46 "2018-12-14T17:00:20Z")

</div>

I understand why you might be frustrated, as writing a book when the target is moving must be difficult. However, I think it was all worthwhile as from my “user” perspective 1.0 is eminently usable and joy to program in.

As a user I don’t necessarily see all the problems with the “ecosystem” as my needs are relatively straightforward and I only see a limited neighborhood within the ecosystem. So that perspective might also be something to incorporate into evangelizing talks, perhaps?

By the way, I really liked your previous book on Julia. I hope the new one is going to be as good! Best wishes.

Petr

---

<div class="post-metadata">

**Author:** ![sherrinm](https://avatars.discourse-cdn.com/v4/letter/s/a88e4f/32.png) [@sherrinm](https://discourse.julialang.org/u/sherrinm)\
**Post date:** [December 14, 2018, 5:03pm UTC](https://discourse.julialang.org/t/the-state-of-the-julia-ecosystem/18650/47 "2018-12-14T17:03:43Z")

</div>

So where can a see the results of the latest run of NewPkgEval, I personally have had difficulty in finding this from the main website.

I think I have discussed what is helpful to me, I wish to tell punters how to download Julia for their favoured platform, find and install a set of packages that concern them and start to use the language constructively.

I do, on the other hand, think the documentation you have provided is excellent and praise this whenever I can. Working on documentation can be a mind-bogglingly tedious task.

---

<div class="post-metadata">

**Author:** ![sherrinm](https://avatars.discourse-cdn.com/v4/letter/s/a88e4f/32.png) [@sherrinm](https://discourse.julialang.org/u/sherrinm)\
**Post date:** [December 14, 2018, 5:07pm UTC](https://discourse.julialang.org/t/the-state-of-the-julia-ecosystem/18650/48 "2018-12-14T17:07:13Z")

</div>

In fact writing this edition is virtually a total rewrite, as I keep telling the publisher when they ask me why progress is so slow.

I’m glad you enjoyed my first attempt, there are large parts of it I’m dissatisfied with and would like to make a better stab at it this time; in particular the last chapter which is pretty poor and I hope the muse grabs me when (if?) I eventually get there.

---

<div class="post-metadata">

**Author:** ![StefanKarpinski](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stefankarpinski/32/24_2.png) [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)\
**Post date:** [December 14, 2018, 5:09pm UTC](https://discourse.julialang.org/t/the-state-of-the-julia-ecosystem/18650/49 "2018-12-14T17:09:27Z")

</div>

> [@sherrinm](#):
>
> So where can a see the results of the latest run of NewPkgEval, I personally have had difficulty in finding this from the main website.

This is not run regularly, published or definitive—we do not have the resources to continuously CI the entire package ecosystem. The version bounds in the registry should be accurate: if a version claims to be compatible with 1.0 in the registry then you can presume that it is.

> I think I have discussed what is helpful to me, I wish to tell punters how to download Julia for their favoured platform, find and install a set of packages that concern them and start to use the language constructively.

So the original post was primarily asking about what version of Julia and packages to recommend? JuliaPro is still a good starter option. The default registry is the same as the set of packages that were previously shipped with it and those are known to work well with the shipped Julia.

It would also be possible to create a list of packages that have some version that is compatible with 1.0+ if that would be helpful. These days I think that people are mostly just trying to install packages and of the resolver lets you do it you’re good to go, otherwise the package is not yet compatible with 1.0.

---

<div class="post-metadata">

**Author:** ![sherrinm](https://avatars.discourse-cdn.com/v4/letter/s/a88e4f/32.png) [@sherrinm](https://discourse.julialang.org/u/sherrinm)\
**Post date:** [December 14, 2018, 5:24pm UTC](https://discourse.julialang.org/t/the-state-of-the-julia-ecosystem/18650/50 "2018-12-14T17:24:03Z")

</div>

So in fact NewPkgEval is not run regularly.

The last version (of the spreadsheet) that I found was not timestamped and still had JuliaDB marked as _skipped._

As I said if J-Pro installed a bundle of packages (as per v0.6.x) then this woud be helpful.

And yes it would be extremely useful to have a list of packages which work with v1.0+ and perhaps if it could replace the page I’ve reference to a couple of times in this blog - which suggests that v0.6.3 remains the best gal on the block.

---

<div class="post-metadata">

**Author:** ![StefanKarpinski](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stefankarpinski/32/24_2.png) [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)\
**Post date:** [December 14, 2018, 5:41pm UTC](https://discourse.julialang.org/t/the-state-of-the-julia-ecosystem/18650/51 "2018-12-14T17:41:42Z")

</div>

> [@sherrinm](#):
>
> So in fact NewPkgEval is not run regularly.

Yes, it is run regularly but manually—specifically before releases to make sure that they are non-breaking as guaranteed. You seem to be misunderstanding the purpose of PkgEval. It is not some ecosystemwide CI system; it is for the Julia devs to evaluate if we have broken packages. Packages are responsible for their own CI: when a package makes a release, it should run CI to make sure that it works with the versions of Julia that it claims to work with and either fix any breakages or update the compatibility claims in the registry accordingly. I’m not sure where you got the idea that PkgEval was for public consumption—it is not. Sure, we will post results because it’s not _secret_ information, but it is really only relevant to Julia developers, not Julia users.

> [@sherrinm](#):
>
> As I said if J-Pro installed a bundle of packages (as per v0.6.x) then this woud be helpful.

There were a lot of complaints that the JuliaPro download was too big, so we have changed the way it works. Now it ships without preinstalled packages but with a registry that lets you easily install the JuliaPro packages and which prompt if you want to stray off of “the guaranteed path” and use non-JuliaPro packages. If you install all of the package in the JuliaPro registry, that’s equivalent to the old JuliaPro.

> [@sherrinm](#):
>
> And yes it would be extremely useful to have a list of packages which work with v1.0+

This information is available in [the registry](https://github.com/JuliaRegistries/General), it should be simple to write a script that pulls out packages that have some version that’s compatible with 1.0+, I guess I could sit down and do that.

---

<div class="post-metadata">

**Author:** ![chakravala](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chakravala/32/6832_2.png) [@chakravala](https://discourse.julialang.org/u/chakravala)\
**Post date:** [December 14, 2018, 6:22pm UTC](https://discourse.julialang.org/t/the-state-of-the-julia-ecosystem/18650/52 "2018-12-14T18:22:21Z")

</div>

> [@StefanKarpinski](#):
>
> Sure, we will post results because it’s not _secret_ information, but it is really only relevant to Julia developers, not Julia users.

It’s not difficult to argue that Julia users would find this information relevant also. You make fair points, it’s not your responsibility to be constantly testing all those packages, since packages are doing their own CI, but it would still be relevant information to Julia users regardless, since Julia users are some kind of developer.

---

<div class="post-metadata">

**Author:** ![mbauman](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mbauman/32/31082_2.png) [@mbauman](https://discourse.julialang.org/u/mbauman)\
**Post date:** [December 14, 2018, 6:27pm UTC](https://discourse.julialang.org/t/the-state-of-the-julia-ecosystem/18650/53 "2018-12-14T18:27:08Z")

</div>

I want to point out that the [NewPkgEval.jl](https://github.com/JuliaComputing/NewPkgEval.jl) as well as the old [PackageEvaluator.jl](https://github.com/JuliaCI/PackageEvaluator.jl) (which is the old code that generated the [pkg.julialang.org](http://pkg.julialang.org) page) are both completely open source with MIT licenses. Anyone can run NewPkgEval! It’s actually a fairly small package. Of course, it does take a non-trivial amount of computation resources. It also looks like it could use a Readme with a quick-start guide, but what better way to get involved than helping out with that?

I would also absolutely love to see the [pkg.julialang.org](http://pkg.julialang.org) site updated and set up to run with regular updates. Indeed, it should be much easier to do and maintain now that Julia is at 1.0 and has stability guarantees.

Now, @sherrinm, please note that I am not suggesting that you be the one to do this work and I know that the fact that it’s open source does not help you directly. But I do imagine there are folks looking for ways to get more involved in the project — and I can say that doing the work to update PackageEvaluator.jl to lean on NewPkgEval and update its site regularly would be quite appreciated.

---

<div class="post-metadata">

**Author:** ![StefanKarpinski](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stefankarpinski/32/24_2.png) [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)\
**Post date:** [December 14, 2018, 6:27pm UTC](https://discourse.julialang.org/t/the-state-of-the-julia-ecosystem/18650/54 "2018-12-14T18:27:56Z")

</div>

The code is public. Anyone who has some available hardware or cloud credits to spend and wants to run this regularly and publish the results is very welcomed to do so.

---

<div class="post-metadata">

**Author:** ![chakravala](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chakravala/32/6832_2.png) [@chakravala](https://discourse.julialang.org/u/chakravala)\
**Post date:** [December 14, 2018, 6:31pm UTC](https://discourse.julialang.org/t/the-state-of-the-julia-ecosystem/18650/55 "2018-12-14T18:31:55Z")

</div>

It would be nice if I could do testing of my own package via a CI cron job, and have it submit a report to `pkg.julialang.org` automatically to update the listings, similar to codecov. Distributed workload.

---

<div class="post-metadata">

**Author:** ![essenciary](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/essenciary/32/210469_2.png) [@essenciary](https://discourse.julialang.org/u/essenciary)\
**Post date:** [December 14, 2018, 6:36pm UTC](https://discourse.julialang.org/t/the-state-of-the-julia-ecosystem/18650/56 "2018-12-14T18:36:17Z")

</div>

> [@StefanKarpinski](#):
>
> Julia Observer is unofficial, and while Dan has done a great job with it, I worry that it also has a bus factor of one. What we need is a solution which guarantees active and ongoing support and maintenance. We do have a plan, which is under active development, so bear with us on the package search front

Can you please share what the plan is? The reason I’m asking is that during the development of the current [Julialang.org](http://Julialang.org) I was discussing with @viralbshah about a website in the line of [https://rubygems.org](https://rubygems.org), [https://hex.pm](https://hex.pm), [https://www.scala-lang.org](https://www.scala-lang.org), etc. With a REST API as well. Unfortunately, we didn’t have time to do this for v1 so we scraped it (we even had a nice package search box in the design of the JuliaLang home page). I was planning on starting a side-project in this direction, but I would rather not do it if other work is being already undertaken.

---

<div class="post-metadata">

**Author:** ![jeff.bezanson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jeff.bezanson/32/48_2.png) [@jeff.bezanson](https://discourse.julialang.org/u/jeff.bezanson)\
**Post date:** [December 14, 2018, 6:37pm UTC](https://discourse.julialang.org/t/the-state-of-the-julia-ecosystem/18650/57 "2018-12-14T18:37:14Z")

</div>

> [@sherrinm](#):
>
> The main concern I expressed in the initial posting was that by not amending [http://pkg.julialang.org](http://pkg.julialang.org), not getting NewPkgEval up and running, and using Julia Observer as a primary reference you do yourselves less than the justice than you deserve.

[pkg.julialang.org](http://pkg.julialang.org) is a luxury — we only have it because one person decided to make it, and we are not entitled to that person’s continued efforts for free. You make it sound like not updating [pkg.julialang.org](http://pkg.julialang.org) was some kind of official decision. Regularly running PkgEval will also require a large amount of compute resources. Well, in fact we are discussing all of this and share your goal of getting it all working nicely. But it will take time, effort, and money. In the future, when posting about things like this I just ask that you phrase it more like “Hi, I noticed that pkg.julialang doesn’t seem to be updated. Are there any plans to do so? It would be great to have it running again.”

> [@sherrinm](#):
>
> In August/September various questions were raised as to which packages work with v1.0; these requests have become less frequent of date, not because the subject has gone away but I suppose because the proponents are running out of stamina.

Posting this without any supporting data is disheartening to see.

---

<div class="post-metadata">

**Author:** ![StefanKarpinski](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stefankarpinski/32/24_2.png) [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)\
**Post date:** [December 14, 2018, 6:37pm UTC](https://discourse.julialang.org/t/the-state-of-the-julia-ecosystem/18650/58 "2018-12-14T18:37:21Z")

</div>

Here’s a script that generates a CSV file with the name of each package and `true` or `false` indicating whether it has a version that is compatible with some 1.x Julia release:

```julia
import Pkg
import Pkg.TOML
import Pkg.Types: VersionSpec
import UUIDs: UUID

function compatible(pkg::String, version_spec::String)
    vers = VersionSpec(version_spec)
    registry_path = joinpath(DEPOT_PATH[1], "registries", "General")
    package_index = TOML.parsefile(joinpath(registry_path, "Registry.toml"))["packages"]
    compatible = Dict{String,Bool}()
    for (uuid_str, info) in package_index
       uuid = UUID(uuid_str)
        name = info["name"]
        path = joinpath(registry_path, info["path"])
        compat_file = joinpath(path, "Compat.toml")
        isfile(compat_file) || continue
        compats = Pkg.Compress.load(compat_file)
        compatible[name] = any(compats) do (ver, compat)
           !isempty(vers ∩ VersionSpec(compat[pkg]))
        end
    end
    return compatible
end

open("tmp/compat_julia_1x.csv", "w") do io
    println(io, "package,compatible")
    compats = compatible("julia", "1.0-1.1")
    for (name, compat) in sort!(collect(compats), by=lowercase∘first)
        println(io, "$name,$compat")
    end
end

```

The output of running this right now is here:

[https://gist.github.com/StefanKarpinski/0a83266ca3db15dc5cb3b7b2d98da02d](https://gist.github.com/StefanKarpinski/0a83266ca3db15dc5cb3b7b2d98da02d)

---

<div class="post-metadata">

**Author:** ![sherrinm](https://avatars.discourse-cdn.com/v4/letter/s/a88e4f/32.png) [@sherrinm](https://discourse.julialang.org/u/sherrinm)\
**Post date:** [December 14, 2018, 6:37pm UTC](https://discourse.julialang.org/t/the-state-of-the-julia-ecosystem/18650/59 "2018-12-14T18:37:27Z")

</div>

I realise the NewPkgEval is open-source and infact pulled down a copy at the weekend with the thought of running a copy. However while engaged on my 2nd edition it did strike me that this might be more appropriate as a job undertaken by the Julia Computing organisation.

I am interested in the idea of creating a script to pull out information from the registry - I believe that were once a Perl Monger (?), as am I, and probably such a script might be an approach I would take. Perhaps you can send me some information in a private communication

---

<div class="post-metadata">

**Author:** ![sherrinm](https://avatars.discourse-cdn.com/v4/letter/s/a88e4f/32.png) [@sherrinm](https://discourse.julialang.org/u/sherrinm)\
**Post date:** [December 14, 2018, 6:47pm UTC](https://discourse.julialang.org/t/the-state-of-the-julia-ecosystem/18650/60 "2018-12-14T18:47:59Z")

</div>

I am sorry to dishearten you.

If you no longer have any use for pkg,julialang,org, then it should be retired.

I believe that there have been specific enquires as to the future of this particular page, which was only one aspect of my posting - of which I took a great deal of care and feel is entirely fair.

I come from the land of Milton, Shakespeare and Fergus McGonigal and am sorry if my prose style is closer to the third of these Great British poets than the first two.

[Previous page](https://discourse.julialang.org/t/the-state-of-the-julia-ecosystem/18650.md?page=2)

[Next page](https://discourse.julialang.org/t/the-state-of-the-julia-ecosystem/18650.md?page=4)