Hey, I have been working on libjulia-clj and it is time to announce what little I have so far ;-).
This is way that you can seamlessly call Julia from Clojure, a language hosted on the JVM. This allows you to use Julia for what it is good for and the JVM for what it is good for, thus sidestepping issues about which one is good at what.
If you have never heard of Clojure, then I think there is plenty out there but here is a new to Clojure document.
I am a huge fan of Julia and need help building out language bindings that are safe and effective. For instance, while zerocopy of dense arrays has been tested a good interface for AbstractArrays has not been developed. This library really is small enough (less than 1,500 lines) that I think it is a good time to get direction from the larger Julia community.
If this interest you in any way please let me know, try it out, file some issues – then fix them ;-).
Greetings, btw, from the Clojure community. Some of us are huge fans!
Very nice! I heard lots of good things about Clojure, I have heard of companies that use Clojure as their main language for microservices. Are you interested in some particular application?
We do application development with a data focus in general and I have always been trying to push the edge of what is possible in terms of high performance computing with the JVM. Lots of times I would just like to use some system or library from Julia with a lot less drama that shelling out or write something in Julia in order to get a lot more perf out of some bespoke piece of code.
The JVM is great because it handles really generic code well and it has a very powerful JIT and garbage collector. My opinion is that this is what makes Clojure possible as that style of functional programming generates a lot of ephemeral garbage as a side effect.
By comparison, I like the tradeoffs that Julia has made; much weaker GC and more emphasis on numerics and powerful (but simply expressed) compilation of the type that is impossible using the JVM regardless of language.
So they seem very complimentary. Many non-numeric algorithms can be expressed with sort of radical simplicity in Clojure and will get sufficient performance but there is a growing need for really great numerics support and instead of attempting to build out the foundation in Java where your performance bar is fairly low for this sort of thing I look to bring the best in the world in range of your normal Clojure programmer. Julia, IMHO, is head and shoulders above everything else including Python, Nim, and R in language design and focus on great numerics. It is setting itself up to be the one true Fortran replacement :-).
So that is the why.
I think at this point libjulia-clj is working fine for us although it is crashing when I mess with NextJournal but that is a whole other can of worms. It is easy for me at this point to pull a piece of code, write it in Julia and call it getting a major speed boost like what is demonstrated by the README on libjulia-clj all without changing the larger systems we have built.
It is also easy for us to use Julia implementations of algorithms such as UMAP and BayesianOptimization. The other option, use Python, also works however I do not enjoy writing Python myself nor do I enjoy the systematic brittleness of the Python ecosystem. In the long run I see Julia as a much better option.
So I am satisfied that for us, Julia is now an option to use in our application development pathways and most importantly when we are doing research and exploring systems.
Another example of research along these lines is our support of the tvm compiler.
I think that probably (more than) answers your question but let me know if not :-).
The Julia ecosystem has benefited greatly from Clojure inspiration (primarily the work of @tkf): JuliaFolds · GitHub
JuliaFolds is whole set of composable packages around the transducers concept, that work on GPU, multithreaded CPU, distributed, with the same user code.
Also extensible to custom types, and leverages macros and user defined compiler passes to desugar loops as well, for the FP averse among us.
There are also efforts to leverage the compiler infrastructure to emit standalone slim binaries (https://github.com/JuliaGPU/GPUCompiler.jl/issues/124) . That + a future reduced runtime (with LLVM codegen etc removed) means that interop might get even easier on constrained systems.
Don’t let the GPUCompiler name fool you, it’s useful for more generally emitting static code.
@Akatz - I have often wondered what transducers would look like in a more typed context! Very interesting!
@Tord - I stopped early on this one to keep things understandable. The repository is like 1,200 lines of code with comments. I don’t have great support for non-dense arrays but if I had added support for all the various types then the repository wouldn’t be understandable for new people…I do believe as I said that what is done is enough to show real potential.
Importantly, you can interact with pointers in Julia. It can be difficult to deal with memory allocated within Julia since you have to make sure that it does not become garbage collected. For this GC.@preserve is really useful macro: Essentials · The Julia Language
In particular, look at Base.unsafe_wrap as documented here. You can use that to wrap a Julia array around externally allocated memory such as a directly allocated java.nio.Buffer. Maybe you were trying to ask me about this earlier?
I want to point out that Mark was the one who found the key piece that he pointed out above. The JVM uses SIGSEGV signals during the normal course of operation. These signals were causing Julia to exit with a memory read error and turning this option off solved that issue…so far without causing other obvious issues :-).
I think zerocopy is important for some things and it is important in order to enable copy fastpaths from jvm heap memory (arrays) into native heap memory. My experience is that most likely, if you are switching languages, you are doing enough work that a single memcpy, as long as it is a true low-level memcpy, isn’t going to be the issue. In order to enable the memcpy, however, you have to know the layout of the target hence zerocopy sets up the actual accelerated copy :-). Rarely would I recommend a system where both the JVM and Julia are, for instance, writing to a shared block of memory but if that makes sense in order to enable the accelerated memory copy operations a zerocopy representation of native heap memory layout is of course important.
Thanks for the pointers to the various systems for doing this and for the pointer into the differential equations library. That is a great target to start looking into and to see what the actual issues are with a non-trivial integration and exactly what I was looking for :-).
One important piece of the matrix that I haven’t attempted yet is to call a Clojure function from Julia. I believe this is possible due to these lines in Julia.h.
Basically I would like to declare a function to Julia and have Julia call it. This is a crux piece of the python integration that is tough to get working but was great once it worked; it kind of completes the loop. I do not, however, see any sort of register_function or new_function exposed via the C API so perhaps I will have to scan the PyJulia integration and see how they handle this. Any help here would be great.
My assumption is that calling a Clojure function would involve the JNI unless Clojure is exporting callable functions to C. In either case, the direct way to do this from Julia would be via ccall or the newer and convenient @ccall.
If you are using JNI to call back into Clojure, you should be able to take advantage of JavaCall.JNI. If you follow that link, you’ll see that we just built a very light wrapper that uses ccall to call the JNI C API.
using JavaCall
using JavaCall.JNI
path_to_libjvm_so = "/path/to/libjvm.so" # Derived from java.library.path
JNI.init_current_vm( path_to_libjvm_so )
Clojure = JNI.FindClass("clojure/java/api/Clojure");
var = JNI.GetStaticMethodID(Clojure, "var", "(Ljava/lang/Object;Ljava/lang/Object;)Lclojure/lang/IFn;");
load_string = JNI.CallStaticObjectMethodA(Clojure, var, Int64.([JNI.NewStringUTF("clojure.core"), JNI.NewStringUTF("load-string")]) )
load_string_invoke = JNI.GetMethodID( JNI.GetObjectClass(load_string), "invoke", "(Ljava/lang/Object;)Ljava/lang/Object;");
JNI.CallObjectMethodA(load_string, load_string_invoke, Int64.( [ JNI.NewStringUTF("(prn (+ 1 2 3 4 5) )") ] ) )
I think what you were trying to do initially was exposing some C function as a built-in function in Julia as one might do in Python. However, in Julia, we just call the C function since LLVM does the work for us.
The conclusion of the following thread is just use ccall .
ccall is how many Base functions are implemented. For example,
Hmm, interesting ccall will definitely work; I hadn’t considered that.
For sure, one my users will attempt to call Julia’s map or something like that with a clojure-defined function so I am just working through the details of how that would work.
Clojure’s IFn interface really is something like a function of Any…->Any so mapping that into a C call where it passes a value_t** and a uint32 n_args would be ideal and then I could specialize from there.
Thanks for the pointer to ccall thread, I will read up on that. Talking about PyMethodDef brings back some rough memories :-).
It looks like you already figured out a way to use JNA’s callback mechanism to create a C function pointer. If you can create a Ptr{Nothing} with the pointer address, you can then pass that to ccall. ccall’s other args are the return type, argument types as a Tuple, and finally the argument values themselves.
It sounds like you may want to create a Julia function that returns a closure that in turn invokes ccall.
Clojure is a beautiful lisp dialect, most of the 3rd scientific libraries are useful but didn’t update for a long period of time. Using Julia from Clojure is the best method for both language ecosystems. I try lib python-clj but it did work with lein. I installed clojisr, Clojure and R are not Interoperability as good a part of Clojure. Can I use Julia and Clojure with lein on OSX?
@madeinquant - Please let us know the how libpython-clj is failing for you here. We use lein for the travis unit tests and I use lein almost exclusively out of habit so I am pretty sure we can get that working.
For Clojisr (R/Clojure interop) - we also use zulip for discussions and I am certain the authors would like to hear from you and we can make it better.
The libjulia-clj unit tests on Travis all use leiningen and those are setup for linux, mac, and sort of windows. So it may work for you. We are currently ironing out details of how the two systems can work better with each other.
Hope this helps, we are happy to help you out if you reach out to us :-).
@cnuernber Thanks for your comments. After reading the usage the section of julia-clj on github, it is a solution what I am looking for. Incanter is my personal favourite, Clojure’s primary statistical computing library, I like the syntax. Unfortunately, the library didn’t update over the last few years.
Opened an account in zulip and I try to install the library in mac. Thanks for your efforts to implement libjulia-clj.
@cnuernber libjulia-clj is installed for mac successfully.
Steps for OS X, hope it can help for installation.
Install julia with brew. brew install julia
A symlink to a Julia version (here 1.5) which has already been created by brew. We don’t need to create a symlink manually. ln -s /Applications/Julia-1.5.app/Contents/Resources/julia/bin/julia /usr/local/bin/julia
Setting the path in ~/.bash_profile export JULIA_HOME=/Applications/Julia-1.5.app/Contents/Resources/julia export PATH=${PATH}:${JULIA_HOME}/bin
Leiningen/Boot in project.clj [cnuernber/libjulia-clj "0.01"]
Connecting to the REPL
Visual studio code has been my main text editor, I did my Clojure development with extension nREPL. lein repl :connect
In your repl, load the julia base namespace and initialize the system.
Memoizing progress on the signaling front from Slack:
@cnuernber reported that preloading libjsig.so works if preloaded. This allows the Java Virtual Machine and Julia to cooperatively use signals so that both Java and Julia garbage collection runtimes can function properly.
From the OpenJDK GPLv2 source code, documentation, and Java RFE 4381843 , this library interposes sigaction, signal (deprecated), and sigset (deprecated) so that the JVM signal handlers can still function along side other signal handlers. It must be loaded before libc, libthread, and before JVM installs its signal handlers since it looks for a flag that indicates the JVM is installing its signal handlers. If the signals are not relevant for the JVM, it will then relay the signal event to the signal handlers that were installed before or after libjsig was loaded. This applies on POSIX (Linux, Mac, etc.) only.
libjuila-clj usage:
user> (System/getenv "LD_PRELOAD")
"/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjsig.so"
user> (require '[libjulia-clj.julia :as julia])
nil
user> (julia/initialize! {:signals-enabled? true :n-threads -1})
10:52:43.045 [nRepl-session-a4fb6d18-0c4a-450d-8422-cc0cb547edbd] INFO libjulia-clj.impl.base - Attempting to initialize Julia at /home/chrisn/dev/cnuernber/libjulia-clj/julia-1.5.3/lib/libjulia.so
10:52:43.176 [nRepl-session-a4fb6d18-0c4a-450d-8422-cc0cb547edbd] INFO tech.v3.jna.base - Library /home/chrisn/dev/cnuernber/libjulia-clj/julia-1.5.3/lib/libjulia.so found at [:system "/home/chrisn/dev/cnuernber/libjulia-clj/julia-1.5.3/lib/libjulia.so"]
:ok
user> (System/gc)
nil
user> (System/gc)
nil
user> (System/gc)
nil
user> (System/gc)
nil
user> (System/gc)
nil
user> (System/gc)
nil
user> (def ignored (julia/eval-string "Threads.@threads for i in 1:1000; zeros(1024, 1024) .+ zeros(1024, 1024); end"))
#'user/ignored
Given these results, I will also explore loading libjsig.so in JavaCall.jl as well.
