Using Julia from the JVM (Clojure)

@Akatz - I have often wondered what transducers would look like in a more typed context! Very interesting!

@Tord - I stopped early on this one to keep things understandable. The repository is like 1,200 lines of code with comments. I don’t have great support for non-dense arrays but if I had added support for all the various types then the repository wouldn’t be understandable for new people…I do believe as I said that what is done is enough to show real potential.

2 Likes

It has been wonderful seeing this develop. For me the journey started with the TWIML AI podcast:

One thing that bugs me about modern programming is that we have difficulty making our tools work together, so the more interop the better!

It might be useful to take a look at other projects that access Julia from other languages.

@tkf has been interfacing Julia into the Python scene.

@Non-Contradiction has developed a package for calling Julia from R:

@ChrisRackauckas has been building on the above to make scientific machine learning in Julia accessible from other languages:

It would be interesting to see what would be required for a diffeq package to be made available for Clojure.

For implementing zerocopy operations, I recommend taking a close look at how Julia interacts with C and Fortran:

Importantly, you can interact with pointers in Julia. It can be difficult to deal with memory allocated within Julia since you have to make sure that it does not become garbage collected. For this GC.@preserve is really useful macro: https://docs.julialang.org/en/v1/base/base/#Base.GC.@preserve

In particular, look at Base.unsafe_wrap as documented here. You can use that to wrap a Julia array around externally allocated memory such as a directly allocated java.nio.Buffer. Maybe you were trying to ask me about this earlier?

An alternative is just allocating memory via Libc.malloc which would not be subject to garbage collection:

@jamii has a write up on exploiting that here:

There is a lot there, thanks Mark – @mkitti!

I want to point out that Mark was the one who found the key piece that he pointed out above. The JVM uses SIGSEGV signals during the normal course of operation. These signals were causing Julia to exit with a memory read error and turning this option off solved that issue…so far without causing other obvious issues :-).

I think zerocopy is important for some things and it is important in order to enable copy fastpaths from jvm heap memory (arrays) into native heap memory. My experience is that most likely, if you are switching languages, you are doing enough work that a single memcpy, as long as it is a true low-level memcpy, isn’t going to be the issue. In order to enable the memcpy, however, you have to know the layout of the target hence zerocopy sets up the actual accelerated copy :-). Rarely would I recommend a system where both the JVM and Julia are, for instance, writing to a shared block of memory but if that makes sense in order to enable the accelerated memory copy operations a zerocopy representation of native heap memory layout is of course important.

Thanks for the pointers to the various systems for doing this and for the pointer into the differential equations library. That is a great target to start looking into and to see what the actual issues are with a non-trivial integration and exactly what I was looking for :-).

One important piece of the matrix that I haven’t attempted yet is to call a Clojure function from Julia. I believe this is possible due to these lines in Julia.h.

Basically I would like to declare a function to Julia and have Julia call it. This is a crux piece of the python integration that is tough to get working but was great once it worked; it kind of completes the loop. I do not, however, see any sort of register_function or new_function exposed via the C API so perhaps I will have to scan the PyJulia integration and see how they handle this. Any help here would be great.

My assumption is that calling a Clojure function would involve the JNI unless Clojure is exporting callable functions to C. In either case, the direct way to do this from Julia would be via ccall or the newer and convenient @ccall.

If you are using JNI to call back into Clojure, you should be able to take advantage of JavaCall.JNI. If you follow that link, you’ll see that we just built a very light wrapper that uses ccall to call the JNI C API.

using JavaCall
using JavaCall.JNI
path_to_libjvm_so = "/path/to/libjvm.so" # Derived from java.library.path
JNI.init_current_vm( path_to_libjvm_so ) 
Clojure = JNI.FindClass("clojure/java/api/Clojure");
var = JNI.GetStaticMethodID(Clojure, "var", "(Ljava/lang/Object;Ljava/lang/Object;)Lclojure/lang/IFn;");
load_string = JNI.CallStaticObjectMethodA(Clojure, var, Int64.([JNI.NewStringUTF("clojure.core"), JNI.NewStringUTF("load-string")]) )
load_string_invoke = JNI.GetMethodID( JNI.GetObjectClass(load_string), "invoke", "(Ljava/lang/Object;)Ljava/lang/Object;");
JNI.CallObjectMethodA(load_string, load_string_invoke, Int64.( [ JNI.NewStringUTF("(prn (+ 1 2 3 4 5) )") ] ) )

JavaCall.jl makes this easier:

using JavaCall
JavaCall.JNI.init_current_vm( path_to_libjvm_so ) 
Clojure = @jimport clojure.java.api.Clojure
IFn = @jimport("clojure.lang.IFn")
load_string = jcall( Clojure, "var", IFn, (JObject,JObject), "clojure.core", "load-string")
jcall( load_string, "invoke", JObject, (JObject,), "(prn (+ 1 2 3 4 5) )")

This should be easier once I get around to making a @jcall macro analogous to @ccall.

I think what you were trying to do initially was exposing some C function as a built-in function in Julia as one might do in Python. However, in Julia, we just call the C function since LLVM does the work for us.

The conclusion of the following thread is just use ccall .

ccall is how many Base functions are implemented. For example,

Hmm, interesting ccall will definitely work; I hadn’t considered that.

For sure, one my users will attempt to call Julia’s map or something like that with a clojure-defined function so I am just working through the details of how that would work.

Clojure’s IFn interface really is something like a function of Any…->Any so mapping that into a C call where it passes a value_t** and a uint32 n_args would be ideal and then I could specialize from there.

Thanks for the pointer to ccall thread, I will read up on that. Talking about PyMethodDef brings back some rough memories :-).

It looks like you already figured out a way to use JNA’s callback mechanism to create a C function pointer. If you can create a Ptr{Nothing} with the pointer address, you can then pass that to ccall. ccall's other args are the return type, argument types as a Tuple, and finally the argument values themselves.

It sounds like you may want to create a Julia function that returns a closure that in turn invokes ccall.

See the section on Indirect Calls:

@cnuernber We (@vchuravy , @Oscar_Smith, @mattBrzezinski, @baggepinnen, and others ) were having a discussion Julia Slack about Java / Julia.
.

One concern that arose is that by disabling Julia’s signaling, we may have interfered with Julia’s garbage collection which seems to use SIGINT.

We may need to take a serious look at signal chaining.

Clojure is a beautiful lisp dialect, most of the 3rd scientific libraries are useful but didn’t update for a long period of time. Using Julia from Clojure is the best method for both language ecosystems. I try lib python-clj but it did work with lein. I installed clojisr, Clojure and R are not Interoperability as good a part of Clojure. Can I use Julia and Clojure with lein on OSX?

@madeinquant - Please let us know the how libpython-clj is failing for you here. We use lein for the travis unit tests and I use lein almost exclusively out of habit so I am pretty sure we can get that working.

For Clojisr (R/Clojure interop) - we also use zulip for discussions and I am certain the authors would like to hear from you and we can make it better.

The libjulia-clj unit tests on Travis all use leiningen and those are setup for linux, mac, and sort of windows. So it may work for you. We are currently ironing out details of how the two systems can work better with each other.

Hope this helps, we are happy to help you out if you reach out to us :-).

1 Like

@cnuernber Thanks for your comments. After reading the usage the section of julia-clj on github, it is a solution what I am looking for. Incanter is my personal favourite, Clojure’s primary statistical computing library, I like the syntax. Unfortunately, the library didn’t update over the last few years.

Opened an account in zulip and I try to install the library in mac. Thanks for your efforts to implement libjulia-clj.

@cnuernber libjulia-clj is installed for mac successfully.

Steps for OS X, hope it can help for installation.

  1. Install julia with brew.
    brew install julia
  2. A symlink to a Julia version (here 1.5) which has already been created by brew. We don’t need to create a symlink manually.
    ln -s /Applications/Julia-1.5.app/Contents/Resources/julia/bin/julia /usr/local/bin/julia
  3. Setting the path in ~/.bash_profile
    export JULIA_HOME=/Applications/Julia-1.5.app/Contents/Resources/julia
    export PATH=${PATH}:${JULIA_HOME}/bin
  4. Leiningen/Boot in project.clj
    [cnuernber/libjulia-clj "0.01"]
  5. Connecting to the REPL
    Visual studio code has been my main text editor, I did my Clojure development with extension nREPL.
    lein repl :connect
  6. In your repl, load the julia base namespace and initialize the system.
    libjulia-clj-play.core=> (require '[libjulia-clj.julia :as julia])
    nil
    libjulia-clj-play.core=>  (julia/initialize!)
    :ok
    
  7. Congratulation, libjulia-clj is installed successfully.

There is a great article to compare different programming language in scientific computing.

1 Like

Memoizing progress on the signaling front from Slack:

@cnuernber reported that preloading libjsig.so works if preloaded. This allows the Java Virtual Machine and Julia to cooperatively use signals so that both Java and Julia garbage collection runtimes can function properly.

Preloading from the libjsig documentation:

export LD_PRELOAD=libjvm.so-directory/libjsig.so; java_application

From the OpenJDK GPLv2 source code, documentation, and Java RFE 4381843 , this library interposes sigaction, signal (deprecated), and sigset (deprecated) so that the JVM signal handlers can still function along side other signal handlers. It must be loaded before libc, libthread, and before JVM installs its signal handlers since it looks for a flag that indicates the JVM is installing its signal handlers. If the signals are not relevant for the JVM, it will then relay the signal event to the signal handlers that were installed before or after libjsig was loaded. This applies on POSIX (Linux, Mac, etc.) only.

libjuila-clj usage:

user> (System/getenv "LD_PRELOAD")
"/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjsig.so"
user> (require '[libjulia-clj.julia :as julia])
nil
user> (julia/initialize! {:signals-enabled? true :n-threads -1})
10:52:43.045 [nRepl-session-a4fb6d18-0c4a-450d-8422-cc0cb547edbd] INFO libjulia-clj.impl.base - Attempting to initialize Julia at /home/chrisn/dev/cnuernber/libjulia-clj/julia-1.5.3/lib/libjulia.so
10:52:43.176 [nRepl-session-a4fb6d18-0c4a-450d-8422-cc0cb547edbd] INFO tech.v3.jna.base - Library /home/chrisn/dev/cnuernber/libjulia-clj/julia-1.5.3/lib/libjulia.so found at [:system "/home/chrisn/dev/cnuernber/libjulia-clj/julia-1.5.3/lib/libjulia.so"]
:ok
user> (System/gc)
nil
user> (System/gc)
nil
user> (System/gc)
nil
user> (System/gc)
nil
user> (System/gc)
nil
user> (System/gc)
nil
user> (def ignored (julia/eval-string "Threads.@threads for i in 1:1000; zeros(1024, 1024) .+ zeros(1024, 1024); end"))
#'user/ignored

Given these results, I will also explore loading libjsig.so in JavaCall.jl as well.

Release of libjulia-clj 0.02 is up with some documentation on the JVM and signals.

If you intend to use libjulia-clj with Julia and enable Julia’s threading mechanism then you probably should read this document :-). @mkitti and myself just did quite a deep dive into how the JVM and Julia deal with signals and figured out a middle path that allows both to coexist peacefully.

2 Likes

@mkitti - Simple demos work for diffeq lib:


user> (require '[libjulia-clj.julia :as julia])
nil
user> (julia/initialize! {:n-threads -1})
13:33:32.571 [nRepl-session-5d74e137-6c8c-4d00-bd4b-4ce1958fb7be] INFO libjulia-clj.impl.base - Attempting to initialize Julia at /home/chrisn/dev/cnuernber/libjulia-clj/julia-1.5.3/lib/libjulia.so
13:33:32.624 [nRepl-session-5d74e137-6c8c-4d00-bd4b-4ce1958fb7be] INFO tech.v3.jna.base - Library /home/chrisn/dev/cnuernber/libjulia-clj/julia-1.5.3/lib/libjulia.so found at [:system "/home/chrisn/dev/cnuernber/libjulia-clj/julia-1.5.3/lib/libjulia.so"]
13:33:32.640 [nRepl-session-5d74e137-6c8c-4d00-bd4b-4ce1958fb7be] INFO libjulia-clj.impl.jna - Julia startup options: n-threads -1, signals? true
:ok
user> (require '[libjulia-clj.modules.DifferentialEquations :as diffeq])
13:34:08.754 [nRepl-session-5d74e137-6c8c-4d00-bd4b-4ce1958fb7be] INFO libjulia-clj.impl.base - Attempting to initialize Julia at /home/chrisn/dev/cnuernber/libjulia-clj/julia-1.5.3/lib/libjulia.so
nil
user> (def f (julia/eval-string "f(u,p,t) = 0.98u"))
13:36:11.268 [nRepl-session-5d74e137-6c8c-4d00-bd4b-4ce1958fb7be] INFO libjulia-clj.impl.base - Rooting address  0x00007F413C2A36C0
#'user/f
user> (def u0 1.0)
#'user/u0
user> (def tspan (julia/tuple 0.0 1.0))
13:36:58.599 [nRepl-session-5d74e137-6c8c-4d00-bd4b-4ce1958fb7be] INFO libjulia-clj.impl.base - Rooting address  0x00007F41390CB310
#'user/tspan
user> tspan
(0.0, 1.0)
user> (def prob (diffeq/ODEProblem f u0 tspan))
13:37:40.532 [nRepl-session-5d74e137-6c8c-4d00-bd4b-4ce1958fb7be] INFO libjulia-clj.impl.base - Rooting address  0x00007F413D3A8A30
13:37:40.533 [nRepl-session-5d74e137-6c8c-4d00-bd4b-4ce1958fb7be] INFO libjulia-clj.impl.base - Rooting address  0x00007F413D3A8AC0
#'user/prob
user> prob
ODEProblem with uType Float64 and tType Float64. In-place: false
timespan: (0.0, 1.0)
u0: 1.0
user> (def sol (diffeq/solve prob))
13:38:20.848 [nRepl-session-5d74e137-6c8c-4d00-bd4b-4ce1958fb7be] INFO libjulia-clj.impl.base - Rooting address  0x00007F413D1EDF90
13:38:20.848 [nRepl-session-5d74e137-6c8c-4d00-bd4b-4ce1958fb7be] INFO libjulia-clj.impl.base - Rooting address  0x00007F413D1EE050
#'user/sol
user> sol
retcode: Success
Interpolation: automatic order switching interpolation
t: [0.0, 0.10042494449239292, 0.35218603951893646, 0.6934436028208104, 1.0]
u: [1.0, 1.1034222047865465, 1.4121908848175448, 1.9730384275622996, 2.664456142481451]

There is more to do here; I haven’t setup a way to get field values from a thing only call it as a function but, after waiting quite a while for diffeq to compile, it appears to work.

Speaking of which:

  • I should redirect logging to the normal logging pathways. Do you have an example of someone writing a custom logger? I had zero feedback during diffeq compilation and killed the process to try it from Julia then saw the log message.
  • Does Julia have an object-oriented way to override stderr/stdout or do they have to be C-files? In python I could implement an object and set that as std err and stdout and this really helps debugging things. Perhaps for Julia simply setting up logging correctly is fine; I just do not know.
2 Likes

On redirecting stderr/stdout, see

For Logging see

Stuck on general ccall wrapping:

user> (def fn-wrapper (julia/eval-string "function(fn-ptr::Ptr{Nothing}) return function(a...) return ccall(fn-ptr, Any, (Any,), a) end end"))
13:45:05.497 [nRepl-session-5d74e137-6c8c-4d00-bd4b-4ce1958fb7be] INFO libjulia-clj.impl.base - Rooting address  0x00007F413C2A3C28
#'user/fn-wrapper
user> (def jl-fn (fn-wrapper jl-voidp))
Execution error at libjulia-clj.impl.base/check-last-error (base.clj:131).
Julia error:
MethodError: no method matching -(::Ptr{Nothing})
Closest candidates are:
  -(::Any, !Matched::Ptr{Nothing}) at none:1
  -(::Any, !Matched::Any) at none:1

Help would be appreciated :-).

The problem with your code is that fn-ptr is not a valid identifier in Julia and is instead parsed as (-)(fn, ptr).

1 Like

Julia is lisp-y, but not to the point of allowing dashes in identifiers :stuck_out_tongue_winking_eye:

3 Likes