Compatibility across Julia 1+ versions

I don’t actually think any additional work has been done on this, so it seems unlikely that any fix will be in 1.3 which is due for feature freeze quite soon.

Probably not very helpful, but this issue is kind of scary in “big data” world. Spark is increasingly ubiquitous and JDBC was a really good fall-back for database interaction. Not having JavaCall.jl will seriously endanger Julia’s usability for a fairly wide audience. I’m not overly fond of Spark myself, but I have had cause to worry about this a bit lately, I’m a little scared to tell people that it’s an ordeal for me to call the JVM.

I wish I could offer help with this issue, but my knowledge falls well short, I don’t think I even understand what the problem is.

3 Likes

I find this surprising, given that 1.2 is still unreleased.

As far as I have heard, the release date of 1.2 will also be the feature freeze of 1.3 and it will happen around the 10th of August (~ 4 month after feature freeze of 1.2).

I have to admit that the 1.3-alpha business also confuses me. Having 3 versions discussed at the same time is not optimal. But I guess this is just because 1.3 brings the new threading architecture and people were curious/trying it out already.

My point with asking here, was is there’s a more general problem? This is the only package I recall stuck on Julia 1.0.x (so I don’t think so, except for all packages depending on JavaCll); however people feel it’s a problem for JDBC (and I guess Taro.jl, Spark and more; e.g. promoted at Julia Computing: Julia and Spark, Better Together).

Despite my title, there’s also a problem with compatibility for packages for Julia 0.6 (or earlier), e.g. ROOTS.jl stuck there. I know nothing was promised, and good tools to help upgrade, still people need to do the/some work.

This got me thinking, is it a crazy idea to implement JuliaCall.jl, to help with JDBC (Julia calling older Julia, calling JVM…)? SImilar to Yuri D'Elia / Polyglot.jl · GitLab (or just an extension to it; yes, that package is stuck on Julia 0.6 it seems, but should be an easy update). One of the point of pybond which Polyglot.jl is based on, was Mixing Python 2/3 code bases.

I see pros and cons:

  1. Old code will just work.
  2. No pressure to upgrade old code (a con if you will).
  3. Allows remote, and while slower/inferior (to e.g. PyCall that doesn’t), you don’t always need mixing data in the same process as PyCall does, nor would it work here for Julia-to-Julia? Even if you run Polyglot not remotely, it’s slower, but may not be a problem for e.g. JDBC. Probably not for all code, e.g. ROOTS.jl?
1 Like

There are a few things going on:

  • The 1.2 RC process has taken an unusually long time—this happens some times
  • We wanted 1.3-alpha out so that people can start testing the multithreading work early

Given the amount of time it can take to get through the RC process, we need to move to a model where we are concurrently working on stabilizing one release while also working on features for the next release, so this will become the norm, except that there will not usually be alpha releases. In this case, 1.2.0 final will be released around the same time as we feature freeze 1.3 and create its release branch.

6 Likes

We want package authors to be as strongly encouraged to upgrade to 1.0+ so allowing people to keep using 0.6 packages indefinitely seems like it would be actively damaging to Julia’s ecosystem. Being able to run two different Julia versions in the same process would be a huge amount of work, would have a significant detrimental performance impact, and wouldn’t fix the problem with Java—it would probably make it worse, in fact—all of which makes this not seem like a very appealing plan.

5 Likes

Not what, I meant. Not doing that would be I think trivial work. I was probably editing my comment when you answered, see on adding to Polyglot.jl that doesn’t use the same process.

Sure, if someone can get RPC between different versions of Julia working, that would be possible, but it does not really seem worth the effort. One of the good things about open source though is that whoever thinks something is worth doing can just go ahead and do it.

1 Like

I’m looking into that. FYI: Indirectly to help me with at, I’m trying to use this package (that must have worked at some point), and I get a segfault when precompiling in Julia 1.0.4, but not in Julia 1.1 (can someone explain the ERROR there?), while still not working… so I opened an issue:

Is the package intended to support Julia 1.0.x?

[EDIT: Not sure about 1.0.x, but still should precompiling for “using Py2Cl” segfault? Possibly some missing “1.1” in some files, I’m mostly familiar with REQUIRE. Initial version was released 19 Aug 2018 days after Julia 1.0.0 released then with Manifest.toml Py2Jl.jl has Project.toml etc. and never had REQUIRE, so I read that to be yes to at least 1.1+ (or 1.0.x) but no Julia version, I’ve checked, works.

I’ll try this in Julia 1.0.x. There’s no reason not to support LTS version.

Supplementary:
I’ve checked Julia 1.0.4, yes it produced a segmentation fault. This reminds me of an old issue by me:
https://github.com/JuliaLang/julia/issues/31240

This only occurs in Julia 1.0.4. About four months ago I used to try debugging julia source code and then it seems that it’s a string comparison in codegen.cpp causing the segmentation fault.

However at that time very people have encountered this, so I think the best option for me is just using Julia 1.1+.

1 Like

If you have a reproducer for the crash, please do report. Since 1.0 is LTS, it should be fixed.

2 Likes

I do have several ways to reproduce this, but it’s really sophisticated and too package specific.
Like via using MLStyle; using DataFrames and then simply doing something with the two packages.

However I think it must be myself who triggered the segmentation fault: Py2Jl doesn’t have DataFrames.jl dependency, but it still produce the segmentation fault.

https://github.com/JuliaLang/julia/blob/v1.0.4/src/codegen.cpp#L1191

if (strcmp(f, "jl_fptr_args") && strcmp(f, "jl_fptr_sparam"))

Should && be ||?

[quote=“thautwarm, post:20, topic:27196”]

I have submitted the reproducer (too, same cause, similar enough segfault?). I just noticed now:

(julia-1.0.4) pkg> st
    Status `~/julia-1.0.4/Project.toml`
  [e53ffe30] Py2Jl v0.1.0 #0dbff54 (https://github.com/JuliaCN/Py2Jl.jl)

[EDIT: after exiting Julia and entering again they’re all there, the segfault seems to have trimmed the list… maybe a clue to someone, should I report that too? Done it.] None of the dependencies are there (or any other package). That’s strange in of itself to me, but at least shows that only this one package is needed to reproduce (and the fault is Julia, not it).

|| would be a trivially true logic. strcmp returns the three way comparison result, I.e. 0 if the inputs are the same.

I just don’t know why this line raises segmentation fault.

Errr, well, if anything || or && is definitely not the reason. If the backtrace is real, it just means f is a bad pointer.