I thought I’d advertise this on here, since the repo on github is so neglected. There is practically no documentation, PR’s sit around forever without getting closed or merged, it’s not registered on PyPl and many of the issues have few to no responses. On top of that, as you can see from the issues, many of these are catastrophic (even segfaults), and prevent the user from using pyjulia at all.
I’d love to help out, but I’m hesitant to spend much time digging into this, not really knowing much about how it works, and there being basically zero documentation. I don’t think I’m alone, apparently a number of weird things have to happen to make this work as is evidenced by the fake-julia subdirectory and its contents.
I feel rather silly going around begging for a package to get attention, in most cases my attitude is just “fix it myself or don’t use it”, but I think pyjulia deserves some special consideration from anyone in a position to help. Thing is, I encounter a lot of people both in real life and online who pretty much refuse to use anything except python (and that exclusion is hardly limited to Julia… we’re just in that period of human history I suppose). It’s hard for me to write a Julia package that does some really cool stuff and get other people to use it if they can’t call it from python. If pyjulia were as robust as PyCall, the rate of Julia adoption would be much higher because before long even those people who refuse to touch anything other than python would find that things that they are using call Julia under the hood somewhere. I think the gradual creep of Julia into python is something we should very much promote, as there probably are a lot of people out there who would much prefer writing Julia code rather than C or even Cython code for their performance critical tasks in Python. I think it would be highly beneficial for Julia and the Julia community if pyjulia were well-maintained and well-advertised.
Again, I feel like a moron for begging people to work on something that I’m not working on myself, but I think if the package matures some, people like me who are woefully ignorant of the Julia interoperability stuff could make more significant contributions.
I recently ran into this problem too. In particular, I think in many python packages Julia could replace C as the high performance language, and there are beginning to be some Julia packages that can fill gaps in the python package ecosystem. Unfortunately, pyjulia is undocumented, and doesn’t seem to work well with custom Julia types.
In my case, there’s a python package (Dedalus) that could benefit from using BandedMatrices.jl for its solvers ( currently using sparse matrix solvers to solve banded systems), but unfortunately pyjulia seems to convert any AbstractMatrix into a python matrix.
The difficulty here is in finding someone who’s familiar with writing Python packages, C extensions, and using the Julia embedding API. That’s a pretty small intersection. Ideally it would be maintained by someone who uses it regularly, and currently I don’t believe anyone does. Many people who start using Julia tend to stop using Python, hence PyCall being so much better maintained than pyjulia.
Given the small amount of code in pyjulia and its current maintenance level, it may actually be more practical for a few Julia package authors who’d like to make their packages callable from Python to rewrite a from-scratch replacement for pyjulia using ctypes or cython or cffi, whatever the most modern / performant method for doing this is nowadays in the Python world (x-ref The PayPal Technology Blog – Medium). I believe cmpb, a Python wrapper for MathProgBase, used ctypes directly instead of going through pyjulia.
Many people who start using Julia tend to stop using Python, hence PyCall being so much better maintained than pyjulia.
This is certainly the case with me personally. Still, I think it’s hard to understate how important maintenance of pyjulia could be for the future of Julia itself. In my experience, there’s an enormous set of people who will simply refuse to ever use Julia for any reason if they cannot embed it in their Python, Java, Ruby, Scala or whatever other language they use. Look at the success of Scala: much of it is due to the fact that it is trivial to use with Java, and much work has been done to make it work nicely with Python. Go is more complicated, but as google has more money than they can possibly know what to do with, it seems like rather a different case.
Many of us who have made the plunge into Julia have done so because we like it so much that we have no desire to return to Python, C++ or anything else. I’m lucky enough to be at a place that is flexible enough so that I can use Julia for my work as a data scientist - but that can certainly change. If it did, I’d probably still be able to use Julia if it were trivial to call it from Python, but not otherwise. I suspect a lot of people are in positions like this.
If any of the current pyjulia maintainers is listening - could we please get a bit of documentation on how this thing works under the hood? I think I get the gist, but am confused by some of what I’m seeing, particularly what the whole “fake-julia” thing is. I have lots of Julia, Python, C-by-way-of-C++ experience (though all this interoperability stuff is new to me), so I could probably make some contributions if I felt I had a somewhat firmer grasp of what’s happening here. Perhaps I’ll post an issue there asking for this…
another voice of support for this. Some of my friends in materials science have started using my Julia codes only because they can call them from Python.
Just provide another use case. Some deep learning frameworks has strong python supports, e.g., writing layers/processors/etc. in python. We can use Julia to write the computing codes and wrap them in python if calling Julia from python is easy. I tried pyjulia months ago, however, it segfaults regularly. Then have to give it up.
By the way, I may have fixed a segfault that was one of the major obstacles to making pyjulia practically useful. See here. At the moment the only other segfault I get is this, which occurs when one calls using on any package that depends on PyCall.
Obviously a ton of work remains to be done after the basic functionality is fixed. Julia objects imported to python are still rather difficult to use, and pyjulia doesn’t import any type constructors.
Everyone agrees that this would be good to have, but as @tkelman says, the basic problem is that many of the people with the technical ability to work on it don’t use it themselves. (e.g. I work on PyCall because I use it myself to call Python from Julia, but I never need the reverse. And even then there are plenty of weird corner cases that I don’t need myself and have limited time to work on.) Asking for volunteers on the mailing list is unlikely to change this.
The ideal situation would be to have someone who is simply paid to work full time on interoperability between Julia and Python (and other languages). But that requires money — either a grant or a corporate sponsorship or donations from someone with deep pockets. Any ideas?
I’m using PyTorch at the moment for DL, but would love to be able to move over to Julia & MxNet - the problem is convincing team members that there’s a mature 2-way transfer between Julia & Python
For me personally, I think what’s going to happen is that once I am in a position where it gets very hard for me to use Julia (because everyone around insists on using only Python, et cetera), I will probably pour a significant amount of my personal time into making pyjulia usable, at least for whatever I have to do. This issue of interoperability, especially from python, still seems critical to the long-term success of Julia to me, but, like everyone else, right now I am very happy using Julia as much as possible and Python as little as possible (at least directly), and I’m in an environment that’s flexible enough that I can do that. But that won’t last. I expect it won’t last for most people.
Note that right now, after fixing the cache issue, pyjulia seems ok at least as a bare minimum, though there is an issue with v0.6 that has me stumped.
I started a little cython interface last weekend. It seems to be working pretty well, but only handles primitive types. For small arrays and dictionaries I’m passing JSON strings. It doesn’t crash (much), and doesn’t require PyCall.
I agree about the importance of PyJulia, RJulia, and similar.
I’m in an environment where everyone else uses R, and only R if they can get away with it. When speed is too important, they’re willing to start diving into C++ with Rcpp.
If RJulia were an option, those still unfamiliar with C++ are likely to choose Julia thanks to being in the same ball park of computer speed, but much faster in human-time. If they keep using it, before long they’ll realize it’s actually easier than R, and start wondering…
Fortunately for me I’m pretty much in the same situation I was back in March: there aren’t to many obstacles to my using Julia at the moment. Again, this could change quickly, if and when it does I’ll be more motivated to contribute to pyjulia.
Since this is an R package, it may be easier to get feedback on R forums. Of course if you have Julia-specific questions, this is the right place to ask them.
As a former R user, I am not convinced that “Julia as a C++ replacement for low-level code” confers any significant benefits. You still face the “two-language problem”, just that language 2 is arguably nicer. But you still have to learn a new language (unless you already know Julia), the performance model of Julia is different (takes a bit of learning to write fast code), and don’t get most of the benefits (Julia is much better for organizing high-level code than R).
I know extremely little about C++, but seeing how some programmers have described it:
Blockquote The rest of the coding was just banging my head against C++. I still spend some time doing this, but luckily C++ usually cracks before my head these days. That was definitely not the case in the old days; Daniel and I were so frustrated trying to get early versions of Stan to build that I felt alternatively like crying and pitching my computer out the window and going back to Java. I still think C++ is designed by sadists with no value for human sanity and only detail-freak masochists can tolerate the pain levels it produces.
I would much rather just use Julia than Julia + R, but if people are willing to dabble in C++ for performance critical code, surely messing with Julia wouldn’t be as rough? My thought was that this would then start convincing people to switch to Julia full time.
Although I do agree that it is easy to write slow Julia code.
Failing that, this would make it easier for some of us to write R packages. All my classmates use R, and the language is still vastly more popular overall.
julia <- julia_setup()
Julia version 0.7.0-DEV.1856 found.
Julia initiation...
WARNING: Error during initialization of module LinAlg:
ErrorException("could not load library "libmkl_rt"
libmkl_rt.so: cannot open shared object file: No such file or directory")
Finish Julia initiation.
Loading setup script for JuliaCall...
UndefVarError(:include) Error in .julia$cmd(paste0("include(\"", system.file("julia/setup.jl", :
Error happens when you try to execute command include("/home/celrod/R/x86_64-pc-linux-gnu-library/3.4/JuliaCall/julia/setup.jl") in Julia.
In addition: Warning messages:
1: In utils::compareVersion(x, y) : NAs introduced by coercion
2: In utils::compareVersion(x, y) : NAs introduced by coercion
A friend with Julia 0.6 had the same error.
I could try switching symlinks around to see if it works with my OpenBLAS install.
Thank you for the feedback!! Is it okay to open a new issue on Github for this? The shared library error may be related to LD_LIBRARY_PATH or something like that. Although I compile julia together with R with the flags as in official embedding julia documentation. Due to some limitations in R, sometimes the .so file that julia needs still have to be in LD_LIBRARY_PATH.
As a long time C++ programmer I would describe that quote as a rather extreme embellishment (of course I am often guilty of these myself!). There are some areas of the language that are overly complicated and horrific (particularly inheritance), but using it to write some performance critical code for R shouldn’t be that bad. I suspect you’d have a hard time selling Julia primarily as an alternative to writing C++ R plugin code to people who are reasonably used to doing that sort of thing.
That said, as I have discussed extensively above, I think being able to run Julia from R and especially Python is crucial for the long term health of the language.
Yes, this is the key. Julia is great for writing packages, but all of the “very popular” software is using C++ or Fortran and with MATLAB/R/Python wrappers. Until we can easily distribute packages to the other scripting languages, writing a package has a distinct disadvantage compared to C++ and Fortran.