What about a service that tracks every new registered package?

proposal

#1

I see that the community has found ways of sharing a new package that they have registered via discourse threads. They tag the thread with the announcement and package keywords and whoever is interested will follow the tags and get notifications.

What if instead of this manual process, we had an integrated service creating posts in discourse? A bot that creates a post for every newly registered package. Every time I go to pkg.julialang.org I discover a new cool package that would have saved me a lot of time if I knew of its existence.

What are your thoughts on this?


#2

For me, the ideal announcement explains motivation, implementation details when relevant, and gives examples. For updates, they talk about what has changed, etc, again with examples. This way I get some perspective of improvements and deprecations since the last release without going through the changelog. I hope that even if automated announcements are implemented, people continue to write informative announcements of this kind here.


#3

I agree the current approach is super nice, perhaps I have chosen the wrong words in the question. I am not proposing to replace the current approach, just add to it. It is good when developers take the time to post on Discourse and write a beautiful rationale for the changes, that is too ideal though.

Besides the current approach, I believe we would benefit from knowing every new package out there. It is also a way of motivating package authors because they will get instant feedback.


#4

You can just follow METADATA, and also help review.


#5

Thanks @ChrisRackauckas, I tried that in the past, it is just too much traffic. I am not interested in new tags, but only newly registered packages (i.e. v0.0.0).


#6

I see. The ability to filter METADATA PRs for registrations would handle this.

But I think it’s okay to wait for authors to do an announcement. Sometimes registrations are earlier than a package is fully ready.


#7

You should be able to set up an email filter in your email reader to filter for METADATA and Registration, to only see package registrations.


#8

Yes, the ability to filter METADATA PRs would do it, and that is my proposal sort of.


#9

@dpsanders I believe we can do better than a hack like that?

What I have in mind is something like a RSS feed for pkg.julialang.org that tracks new registered packages and nothing more.


#10

Yes that’s a fair point and a good idea.


#11

Who has the skills to implement it? Unfortunately I don’t have the expertise for doing it myself.


#12

If you don’t need it super quickly, I can do the thing. I’m thinking of the following parts:

  1. Tracker for METADATA.jl, triggering new announces every time a package registration PR is merged.
  2. A nice web page listing latest packages + maybe some beautiful graphs.
  3. RSS feed for old-schoolers.
  4. Note sure about re-posting news here, but if it makes sense, it should be easy to implement as well.

Any other features?


#13

Thanks @dfdx, that would be awesome! I would say keep it simple, the RSS feed could go first in your priority list, the rest is decoration. Also, the main feature we are discussing here is being able to filter out normal PRs from PRs that introduce a new package. Perhaps you could compare the diff to see if any new folder was created by the PR? Something on these lines.


#14

Some info on this topic is already available here: http://pkg.julialang.org/pulse.html. This shows both new and updated packages in the last 7 days, and has links to the packages if you want more info. How often is this updated?


#15

I decided to start with parsing PRs title and checking if it starts with “Register” - it requires a way fewer API calls and still seems to be quite reliable. We can improve it later if needed.

Any suggestions on how to name or where to place such a service?


#16

@tkelman do you have suggestions for where this service can be hosted and how it should be named?


#17

Hosting is not an issue - I can just put it to one of my already running servers. As for the name, it might make sense to integrate with some existing package listing. I can provide an API so that simple redirect would be enough to integrate such an RSS feed. (cc: @djsegal)

BTW, I’m somewhere 80% done with the first version, so expect to see it at the end of the week.


#18

juliaobserver.com seems like it could probably satisfy a decent amount of these requirements.

how exactly do you want everything to work together?


right now i’m just scraping github’s api ~5x a day. i could probably tie into a RSS feed or something


#19

Depending on your spare time and wish we can go 3 ways:

  1. Client-side integration: you can just add a <link> element pointing to my RSS to your main page.
  2. Server-side integration: you may want to provide your own endpoint (and thus nicer url like “juliaobserver.com/rss”), and then on server side call my API.
  3. Or you may want to rewrite RSS feed code to Ruby to avoid dependency on 3rd party APIs at all. You will only need to make a few more GitHub API calls, parse results and build XML document, and I can provide all the pieces and knowledge I’ve got so far.

#20

That also sounds like a pretty good plan to get to full integration.

Just let me know when you have code or an api live (maybe leave an issue on github)