Many of you already knows me as a great Julia follower/developer/advocate in many platforms. So what I mention here is not criticizing Julia’s way of managing legacy or foreign code but my goal is to shed light on it if there are possible new developments.
I started to use R when I was a PhD. student in 2005. Then in many years, I adopted my C and C++ code in a hard way. After using the Rcpp package, calling into C and C++ binaries were as easy as calling R citizen counterparts.
In Julia, I can not see the same straightforward approach when it comes to talking to strangers. Here is my first attempt to migrate an R package’s Fortran part to Julia and what I got is total fail.
Using the C/C++/Fortran compiler manually and generating a shared library is not a problem for us Julians. By using the FFI feature, it is also easy to call into binary functions. But I think creating a yggdrasil pull request is not that easy.
Please change my mind, if I am getting it wrong.
In R, the packaging system always interacts with the local C/C++/Fortran compiler, generates the binaries and gets everything done for the end-user. In Windows and Mac systems pre-compiled binaries are served in a way similar to Julia’s one. But the source code is hosted in a single package. A package developer can write/test his both R and C code in the same development stage. A single
R CMD build/test/install
makes everything ready.
Can we borrow same functionality from R? What do you think?
In fact, Julia used to do that (typically with BinDeps.jl). And you can still do that if you want (either via BinDeps.jl or manually in your deps/build.jl script). But it’s pretty fragile — there are just so many ways for people’s local build systems to be broken or idiosyncratic, and debugging user problems was a huge pain. The BinaryBuilder.jl style of cross-compiled prebuilt binaries (e.g. via Yggdrasil), in comparison, has been relatively painless and I think most of us found it to be a huge improvement … for users, it almost always just works, and for developers you only need to get a build script working once and then forget about it. But cross-compiling is a bit of a different art than usual software builds, so on the developer side it takes a bit of getting used to.
BinaryBuilder also allows you to build the JLL package locally so you can test on your platform before making a pull request to Yggdrasil, then you can pkg> dev the JLL to work with in other packages locally.
Using the run_wizard() function from BinaryBuilder is actually easy, if you have a reasonable Makefile (particularly if you have a cmake one). The hard part is to convince Mosè that your build script is not an insane mess (which in my case always is). But when you pass over Mosè’s bar, you’ll probably have a very solid build for all platforms, something that I was never able to do with my standard packages that are written in other languages. At the end I currently I’m just writing simple parsers for all my other-language packages in Julia, and they are much easier to distribute that way.
Thank you for the response. Did you have a chance to take a look at the link I provided in the first entry (a github issue about migrating R package to Julia).
First, I opened the site of the package and didn’t immediately find a fortran code there, so it is not clear to me exactly what are you compiling with the command you provide there. But that is probably easy to clarify.
If you provide somewhere detailed instructions on how to download and compile the package in one machine, it will be fairly easy to create a build script for the BinaryBuilder wizard.
You will basically need a script with more or less this content, which is created by the wizard (and here already adjusted by the tips of Mosè):
All that working (I did all this for the first time a couple of weeks ago, by the way), you will create a Julia package to provide the interface for that function. Since you know already how to call the function you want from Julia, you just need a package that exports a function with that enclosed, given the user inputs.
There might be some differences if your package provides a library, not an executable, etc, and yes, building these can be cumbersome at first, but when done you’ll have a very robust distribution.
I recommend joining the the binarybuilder channel on Slack for interactive assistance.
The https://binarybuilder.org/ is much less demanding and dependent on the user’s environment and delivers a much more consistent cross platform experience for users.
Thank you @mkitti, it seems the binary package is available in Yggdrasil repository. I will use this as a dependency in LinRegOutliers.jl package and we will have one more robust regression technique for linear regression.
By the way, I once again saw how difficult and complex creating such a package could be. However, once the difficulty is overcome, we have a library that works on all platforms, and that’s what’s beautiful about it.
The JLL package was ultimately published and registered here:
The main gist was the recipe below. The main thing that needed be added was a copy of the compiled shared library into ${WORKSPACE}/destdir/lib or $libdir. We also needed to abstract out the fortran compiler rather than hard coding it, and install a proper license.