lhe/Biofast benchmark | FASTQ parsing [Julia,Nim,Crystal,Python,...]

There are packages that have different versions that span both registries, and I don’t trust myself to manually sort that out in a PR without human error creeping in. But for the packages with no previous entries in General it would be easier.

Manually sounds barbaric but let there be tools. Try this and see if you like the result in /tmp/General (adjust the paths as necessary if you’re not running Linux):

using Pkg
pkg"add LocalRegistry#merge"
using LocalRegistry
run(`git clone https://github.com/JuliaRegistries/General.git /tmp/General`)
run(`git clone https://github.com/BioJulia/BioJuliaRegistry.git /tmp/BioJuliaRegistry`)
LocalRegistry.merge("/tmp/General", "/tmp/BioJuliaRegistry", merge_packages = true)
1 Like

I tried this this morning - I totally did not clock the merge_packages option facepalm.

It probably didn’t exist at the time. :slight_smile:

OMG that merge function has saved my sanity. Thank you!

3 Likes

Ok, everything in the BioJuliaRegistry has been merged into General. I’m going to update the website, but there may be references to the registry in READMEs and old doc builds whilst we get around to hunting them out and correcting them. In the meantime it would help if any users could remove the BioJuliaRegistry and try and do their thing with just General and see if it breaks - it shouldn’t but you know how it is. Soon-ish, say end of bank holiday, I will archive the BioJuliaRegistry.

12 Likes

New Getting Started instructions are available on the website, as is a new post explaining Bio.jl is deprecated, and which packages should be used instead.

3 Likes

What should I do to migrate from BioJuliaRegistry to General ?

Assuming it worked as expected, I think just delete the BioJulia registry (~/.julia/registries/BioJuliaRegistry/), then resolve? I did that (well, I moved the registry instead of deleting, just in case), and that seems to have worked.

I just did pkg> registry rm BioJuliaRegistry and then for my few active projects on my desktop deleted any manifests and instantiated them again. Resolve seems to work as @kevbonham says. Any packages - mostly BioJulia ones that still get the BioJuliaRegistry in their CI in order to work, should work without that step now and so it can be removed from the Travis (or other CI) files.

I think this

Also importantly, the Julia developers do not value backward compatibility.

is a pretty spectacular statement. Are you basing it on version breakage pre v1.0? I think that seems overly harsh, even unfair.

2 Likes

I would go so far as to say it’s so spectacular it’s not worth engaging.

2 Likes

It does me no good to provoke the Julia community here. I will only ask a question: do you anticipate a python2-to-3 like massive breaking changes in the next 5 to 10 years? By “massive breaking changes”, I mean a big fraction of existing code will stop working. I saw brief responses from some of you on hacker news/reddit. The answer was yes. Is that still true? Or maybe those responses don’t represent the opinion of the core Julia devs?

EDIT: also this thread really worries me. Like “Julia 2.0 will come with features that people are dying to have that aren’t possible without breaking things”.

My understanding is that there most likely will be breaking changes to the language within the next 5-10 years. That fact has little to do with “not valuing backwards compatibility” and does not doom us a Python2 / Python3 situation.

The problem with Python 3 was that it introduced changes that many in the community felt were unnecessary or unwanted and also introduced major performance penalties while simultaneously not offering really any new features that people found enticing.

The Julia developers will probably eventually make breaking changes to the language, but everything I’ve seen suggests they take the downsides of breakage very seriously. There is no indication or evidence they’ll break things for no reason and I strongly beleive that they’ll only introduce breakage when there is a very compelling reason to do so and the community overwhelmingly supports it.


As @ChrisRackauckas mentioned, the Julia community in the past created a ton of machinery to make the 0.6 to 1.0 transition straightforward, including bots, compat tools and deprecation warnings. There’s no reason to think that machinery won’t be present for a hypothetical 1.0 to 2.0 transition, and it’ll likely be more advanced.

4 Likes

My impression is that the core devs are taking great care to avoid problems related to breaking changes, and in particular try to provide an easy upgrade path. The Python 2->3 was a debacle that lasted a decade, and no one wants that. I can’t read minds, though, so I don’t know exactly how much they care.

What I am saying is that your statement is extremely strong if it is based only on the development of the language in the pre version 1.0 phase. And if it isn’t, then I am curious what has happened post 1.0 that has strengthened you impression.

I can’t see that happening. I talk with these folks daily and all of the discussion is always how to add features and improvements without breaking code. Literally every package is tested before a new Julia release comes out.You can have a package sit for years without having to update it: I know that because I went back to a few packages last touched in the v1.0 upgrade and they were just fine. v1.0 was big, and the reason it was big was so that those big changes wouldn’t have to happen again. Will there be a 2.0 and a breaking change? Yes, probably. But it’ll come with depwarns, a bot that automatically updates 90% of code, a Compat.jl package that supports both versions, and blog posts telling you exactly what to do.

Any suggestion that people have something like that in mind is completely unsubstantiated and FUD unless you point to an issue by one of the maintainers that says “yes, we are intend to make such a major breaking change”.

4 Likes

Just to be more explicit about the complete logical fallacy here, extrapolation from pre-v1.0 makes absolutely no sense. In the last few months before v1.0 was out, all of the dev discussions were like “okay, we have 6 months left to break anything. After v1.0, they are fixed forever. So what are all of the syntax choices you don’t like? What are all of the naming choices you think are inconsistent? What are all of the weird workarounds you think should be removed? Okay, let’s triage, make decisions, and make breaking change fix all of these right now, and set in stone a good v1.0. After that, our choices will be fully documented for the future to know why this is decided, and we can move on without having to ever argue the name of function X”.

Looking at that and going “they broke a lot before v1.0, therefore they will break a lot after v1.0” is a fallacy that completely ignores all of the reasons for why v1.0 was such a breaking release in the first place.

8 Likes

Ok. I get the answer. I will point people to this and the other thread and let them interpret by themselves.

2 Likes

@StefanKarpinski had a really nice explanation regarding the Python 2-to-3 mess and how it doesn’t apply to Julia. I’m having trouble finding it right now. Can someone find that comment and link it? I think it was a comment somewhere on Julia Discourse.

1 Like

Okay, here is one of @StefanKarpinski’s good comments on the matter. If you search the Julia Discourse, you can also find many other good comments by Stefan on this matter.

10 Likes