Post list of all registered packages

So I know this has probably been discussed, but I am very surprised that

  1. JuliaObserver is broken and may have been for some time – see Julia Observer: Request for Help , or just try clicking the ALL tab on https://juliaobserver.com/

  2. PackageEvaluator listing is stuck on v0.6 – see https://pkg.julialang.org/

  3. NewPkgEval results aren’t posted anywhere I could find – see GitHub - JuliaCI/PkgEval.jl: Keeping tabs on the julia ecosystem

How can we evangelize Julia if we cannot point new users to a good list of all available registered packages?

I am using various versions of Julia on Mac and Linux platforms, and am willing to run list generators and upload to a site, if that’s all we need to do. There should be a site where one can choose the Julia version, OS platform, and get a simple long list like PackageEvaluator produced, with links to the github home pages. If we can get a few other volunteers to run the generators on other platforms (e.g. MS Windows), it should be straightforward to keep a list updated, perhaps weekly.

Please advise.

2 Likes

I don’t quite see how the list of all available package is useful to a new user—it’s like drinking from a firehose. A small, curated list would be ideal, and one can always ask on the forum, too. I would say that the packages I would confidently refer a new user to are not more than a hundred, maybe less.

Also, the lists you link to covered all registered packages — with the new Pkg framework, unregistered packages can be used quite easily, too.

Even currently abandonned projects are the result of effort put in by volunteers. I appreciate that you feel strongly about this (even though I am not sure I understand why), and it is fine if you want to revive one of these projects, or start a new one, but please consider using a different tone.

3 Likes

Tamas,

Thanks for your reply. I edited my top post to change the tone as you suggested.

For me, the long comprehensive list is best. I like to be able to search on a term, e.g. “protobuf” or “gpu” and find all the registered packages working on various things. Curated lists are fine too, but no substitute for a regularly-updated comprehensive one, IMO.

1 Like

Hi @ronubi, builder of julia observer here. Will try to have it back up and running soon. It just broke while another project’s hitting the fan.

If you want to donate to the cause, I’d probably move it a notch or two higher on my todo list for the week. :stuck_out_tongue:

// also, i accept your dare. i clicked on a lot of the links on julia observer – and almost all of them are broken

3 Likes

Thanks for the update and the humor. I clicked around on your github page and home website, but didn’t find a donate link anywhere . . .

On Github, seaching for protobuf language:Julia works fine. Given that most open source packages on Julia are on Github, this gives you fair coverage; but Gitlab has similar functionality.

1 Like

Just added a donate button to the github readme. The thought never really crossed my mind before this. Will try to have the website fixed soon.

Cheers,
Dan

1 Like

Even better: Just add a jl to your search term. Since Julia packages are named with a trailing .jl by convention, simply searching for protobuf.jl or gpu jl works great.

5 Likes

For all registered installable packages you can use, https://github.com/Nosferican/DependenciesParser.jl.

using DependenciesParser
DependenciesParser.data
1 Like

When I started to play around with julia, I scrolled through several times the list of packages at PackageEvaluator. I think when you don’t search for something, but you are interested in what is possible with the language, it is useful to have a complete list.
And I also agree that a smaller curated list would be ideal to show the state-of-the-art of the ecosystem.

I find this works nicely with your package:

using DependenciesParser

function lookingfor(str::String)
    found = Array{String, 1}()
    for pkg in DependenciesParser.data
        if occursin(str, pkg)
            push!(found, pkg)
        end
    end
    return found
end

julia> lookingfor("Color")
11-element Array{String,1}:
 "AnsiColor"
 "Color"
 "ColorBrewer"
 "Colors"
 "ColorSchemes"
 "ColorSchemeTools"
 "ColorTypes"
 "ColorVectorSpace"
 "EchogramColorSchemes"
 "NamedColors"
 "NoveltyColors"

julia> lookingfor("Cache")
4-element Array{String,1}:
 "CacheServers"
 "DispatcherCache"
 "LRUCache"
 "RequestsCache"
3 Likes

In general, the appropiate solution in my opinion is to have package metadata in the Project.toml, such as author, license, tags, description, etc. That way one can filter and discover packages based on the metadata (e.g., permissive license with maintainer with x reputation that does y). Hopefully, that gets incorporated soon with the new Registrator and full transitioned Pkg era. Other tools like a centralized review system or recommender systems would be nice too.

2 Likes

I agree with the need for metadata, but I don’t think that browsing, filtering, and other display functions are within the scope of Registrator.

I totally agree with @ronubi here, and this if my major complaint about the whole Julia thing these days.

It is very very odd for pkg.julialang.org to be outdated. Many people access that and think that the language is just dead. I had to tell people about juliaobserver.com, which is a nice site, but sounds like not official to me, since it’s not under julialang.org . And if it’s offline, than it is useless.

It is also very odd to suggest users to write code to search for packages. How the hell can an outside user find out on his own that DependenciesParser exists and you can use it to search for packages?

Regarding searching for packages on github, I know many people who wants to use Julia but doesn’t even know that github exists. If we want outsiders to get to know Julia, it must exists one straightforward way to look for packages. Many people want to check if there is a package for something before even installing Julia. And I would guess that if we want the user base to increase, we must make it easy for non-developers (the data-scientists that are not developers that share code on github) to find out what is available in the Julia ecosystem.

5 Likes

Or, alternatively, just document how to search for packages on Github, Gitlab, etc. Maybe provide a simple website that does this in one step.

There is an important distinction between:

  1. all Julia packages,
  2. all registered Julia packages,
  3. registered Julia packages which were selected as stable/useful (“curated list”),
  4. above plus some security assurances thrown in (eg if you report malicious code, it will be taken down).

Most package lists maintained by various languages fall between 2 and 4 somewhere. But also some older languages have a single centralized package registry, either by design or de facto.

The advantage of Github and similar is that they may have a lot of readily available information that should aid the user in picking a package (outstanding issues, contribution stats, stars). It is not clear that duplicating this is a good investment of effort.

Nor is the idea that Julia has to be evangelized. Use it because it solves YOUR problem, then share your solution with others. Finding ways to inject that into every conversation is IMO counterproductive.

1 Like

There are two items in the recent JuliaTeam vision blog post that will solve this problem: Centrally hosted documentation as well as search for docs and code.
A public instance of this should be out very soon.

4 Likes

Thanks. This bullet point on Search is informative:

Search [in progress]. Currently search and discovery of packages is a serious pain point in the Julia ecosystem. JuliaTeam will provide integrated search of documentation and code for all packages. This will let you find the package that does what you need, whether it’s a public open source package or a private package that your organization uses—they’ll all be searchable in a single place. Code search will match the capabilities of the Google Code Search of old.

And I wasn’t aware of the METADATA.jl (old way) vs General (new way) registry issue, which probably has a lot to do with why pkg.julialang.org hasn’t been updated.

A bit of context,

@Tamas_Papp, while metadata is not within the scope of Registrator in general, this needs to happen first in order for METADATA/Attobot/REQUIRE to be drop so people can submit Project.toml and not have these bare-generated. Registrator does gets a bit into the maintainer-like with the user permissions to tag a release. Another issue is to find the universe of packages. In my own research I do this, but I usually care about production ready maintained documented packages for instance. These are mostly from the subset that is registered with a main registry (CRAN/CTAN/PyPI/General). The way to check those is to crawl the registry. However, due to the current state of things a few things can happen, a package might not be compatible with that Julia version, its repository might have been deleted or turned private, it might have been deprecated, etc. I usually run a script for JuliaEcosystem that uses Pkg to assess whether a package is installable (package and all dependencies compatible and available). The next step is to check whether it is working as intended which means to run the test matrix à la CI/PkgEval. The plan for new-era registries is for forking the release tag and hosting a copy making it immutable (i.e., author forfeiting right to take it down similar to a journal submission and such).

@felipenoris PackageEvaluator has been outdated since Julia v0.7/v1.0 came out (never supported beyond v0.6). The new project is NewPkgEval, but it is not run CRON and used only for assessment of breaking changes in the language to check what in the ecosystem (registered packages) might break. I have my packages CRON with Travis on the release version (decentralized), but hopefully a centralized system comes up at the registry level. PkgEval as well as JuliaObserver are contributed tools not affiliated/supported by the core language or JuliaComputing.

Looks like New pkg.julialang.org - #10 by juliohm solves the discussion on this thread. :partying_face:

5 Likes