Release of libjulia-clj 0.02 is up with some documentation on the JVM and signals.
If you intend to use libjulia-clj with Julia and enable Julia’s threading mechanism then you probably should read this document :-). @mkitti and myself just did quite a deep dive into how the JVM and Julia deal with signals and figured out a middle path that allows both to coexist peacefully.
@mkitti - Simple demos work for diffeq lib:
user> (require '[libjulia-clj.julia :as julia])
nil
user> (julia/initialize! {:n-threads -1})
13:33:32.571 [nRepl-session-5d74e137-6c8c-4d00-bd4b-4ce1958fb7be] INFO libjulia-clj.impl.base - Attempting to initialize Julia at /home/chrisn/dev/cnuernber/libjulia-clj/julia-1.5.3/lib/libjulia.so
13:33:32.624 [nRepl-session-5d74e137-6c8c-4d00-bd4b-4ce1958fb7be] INFO tech.v3.jna.base - Library /home/chrisn/dev/cnuernber/libjulia-clj/julia-1.5.3/lib/libjulia.so found at [:system "/home/chrisn/dev/cnuernber/libjulia-clj/julia-1.5.3/lib/libjulia.so"]
13:33:32.640 [nRepl-session-5d74e137-6c8c-4d00-bd4b-4ce1958fb7be] INFO libjulia-clj.impl.jna - Julia startup options: n-threads -1, signals? true
:ok
user> (require '[libjulia-clj.modules.DifferentialEquations :as diffeq])
13:34:08.754 [nRepl-session-5d74e137-6c8c-4d00-bd4b-4ce1958fb7be] INFO libjulia-clj.impl.base - Attempting to initialize Julia at /home/chrisn/dev/cnuernber/libjulia-clj/julia-1.5.3/lib/libjulia.so
nil
user> (def f (julia/eval-string "f(u,p,t) = 0.98u"))
13:36:11.268 [nRepl-session-5d74e137-6c8c-4d00-bd4b-4ce1958fb7be] INFO libjulia-clj.impl.base - Rooting address 0x00007F413C2A36C0
#'user/f
user> (def u0 1.0)
#'user/u0
user> (def tspan (julia/tuple 0.0 1.0))
13:36:58.599 [nRepl-session-5d74e137-6c8c-4d00-bd4b-4ce1958fb7be] INFO libjulia-clj.impl.base - Rooting address 0x00007F41390CB310
#'user/tspan
user> tspan
(0.0, 1.0)
user> (def prob (diffeq/ODEProblem f u0 tspan))
13:37:40.532 [nRepl-session-5d74e137-6c8c-4d00-bd4b-4ce1958fb7be] INFO libjulia-clj.impl.base - Rooting address 0x00007F413D3A8A30
13:37:40.533 [nRepl-session-5d74e137-6c8c-4d00-bd4b-4ce1958fb7be] INFO libjulia-clj.impl.base - Rooting address 0x00007F413D3A8AC0
#'user/prob
user> prob
ODEProblem with uType Float64 and tType Float64. In-place: false
timespan: (0.0, 1.0)
u0: 1.0
user> (def sol (diffeq/solve prob))
13:38:20.848 [nRepl-session-5d74e137-6c8c-4d00-bd4b-4ce1958fb7be] INFO libjulia-clj.impl.base - Rooting address 0x00007F413D1EDF90
13:38:20.848 [nRepl-session-5d74e137-6c8c-4d00-bd4b-4ce1958fb7be] INFO libjulia-clj.impl.base - Rooting address 0x00007F413D1EE050
#'user/sol
user> sol
retcode: Success
Interpolation: automatic order switching interpolation
t: [0.0, 0.10042494449239292, 0.35218603951893646, 0.6934436028208104, 1.0]
u: [1.0, 1.1034222047865465, 1.4121908848175448, 1.9730384275622996, 2.664456142481451]
There is more to do here; I haven’t setup a way to get field values from a thing only call it as a function but, after waiting quite a while for diffeq to compile, it appears to work.
Speaking of which:
On redirecting stderr/stdout, see
https://docs.julialang.org/en/v1/base/io-network/#Base.redirect_stdout
For Logging see
https://docs.julialang.org/en/v1/stdlib/Logging/#Writing-log-events-to-a-file
Stuck on general ccall wrapping:
user> (def fn-wrapper (julia/eval-string "function(fn-ptr::Ptr{Nothing}) return function(a...) return ccall(fn-ptr, Any, (Any,), a) end end"))
13:45:05.497 [nRepl-session-5d74e137-6c8c-4d00-bd4b-4ce1958fb7be] INFO libjulia-clj.impl.base - Rooting address 0x00007F413C2A3C28
#'user/fn-wrapper
user> (def jl-fn (fn-wrapper jl-voidp))
Execution error at libjulia-clj.impl.base/check-last-error (base.clj:131).
Julia error:
MethodError: no method matching -(::Ptr{Nothing})
Closest candidates are:
-(::Any, !Matched::Ptr{Nothing}) at none:1
-(::Any, !Matched::Any) at none:1
Help would be appreciated :-).
The problem with your code is that fn-ptr is not a valid identifier in Julia and is instead parsed as (-)(fn, ptr).
Julia is lisp-y, but not to the point of allowing dashes in identifiers 
Haha, thanks for the help :-).
user> (def fn-wrapper (julia/eval-string "function(fn_ptr::Ptr{Nothing}) return function(a...) return ccall(fn_ptr, Any, (Any,), a) end end"))
15:30:01.547 [nRepl-session-5d74e137-6c8c-4d00-bd4b-4ce1958fb7be] INFO libjulia-clj.impl.base - Rooting address 0x00007F413C2A3CA0
#'user/fn-wrapper
user> (def jl-voidp (libjulia-clj.impl.base/fn->jl println))
15:30:06.034 [nRepl-session-5d74e137-6c8c-4d00-bd4b-4ce1958fb7be] INFO libjulia-clj.impl.base - Rooting address 0x00007F413CE0C950
#'user/jl-voidp
user> (def new-fn (fn-wrapper jl-voidp))
15:30:10.676 [nRepl-session-5d74e137-6c8c-4d00-bd4b-4ce1958fb7be] INFO libjulia-clj.impl.base - Rooting address 0x00007F413CE0D1E0
15:30:10.677 [nRepl-session-5d74e137-6c8c-4d00-bd4b-4ce1958fb7be] INFO libjulia-clj.impl.base - Rooting address 0x00007F413CE0D240
#'user/new-fn
user> (new-fn 1 2 3 "hey")
15:30:19.283 [nRepl-session-5d74e137-6c8c-4d00-bd4b-4ce1958fb7be] INFO libjulia-clj.impl.base - Rooting address 0x00007F413DCF33D0
1 2 3 hey
nil
That functionality survives a forced GC run of the JVM and julia so it is robust enough for now :-).
I found how pyjulia manages stdout and stderr. I have a question about references and GC. The python code maintains the new stdout, stderr pipes in global variables. But the code also closes over them like such:
# At file scope
const read_stdout = Ref{Base.PipeEndpoint}()
# At local scope
global readout_task
read_stdout[], = redirect_stdout()
readout_task = @async pipe_stream(read_stdout[], out_receiver)
I could see the readout task needing to be global else the GC could clean it up but I do not understand why the read_stdout needs to be global. It is closed over in pipe_stream.
Not a big issues as I can just copy the code and it was written by people with a lot more Julia knowledge than I have. Just trying to get a feel for some of the more subtle issues of Julia GC.
Nex issue: – tasks appear to be failing:
user> (def sync-fn (julia/eval-string "function(receiver) receiver(\"heyyou\") end"))
#'user/sync-fn
user> (def async-fn (julia/eval-string "function(receiver) @async receiver(\"heyyou\") end"))
#'user/async-fn
user> (sync-fn (fn [data] (println "data was" data) data))
data was heyyou
"heyyou"
user> (def task (async-fn (fn [data] (println "data was" data) data)))
#'user/task
user> (def started? (julia/eval-string "istaskstarted"))
#'user/started?
user> (def done? (julia/eval-string "istaskdone"))
#'user/done?
user> (started? task)
false
user> (done? task)
false
user> (julia/eval-string "yield()")
nil
user> (started? task)
true
user> (done? task)
true
user> ;;??? Nothing printed
user> (def fetch (julia/eval-string "fetch"))
#'user/fetch
user> (fetch task)
nil
I would like to use async tasks to redirect IO to Clojure but so far I cannot get any task to output anything to clojure at all.
It might be useful to note that Julia uses libuv to implement asynchronous I/O:
https://docs.julialang.org/en/v1.1/devdocs/stdio/#Libuv-wrappers-for-stdio-1
My guess is that this code was originally from IJulia which is the Julia kernel for Jupyter Notebooks.
One reason to expose a global is so that the variable is accessible. It’s like a public property of a class in Java.
The reason for the const Ref is actually a Julia idiom. globals can really slow things down in Julia because they mutate. A const Ref allows us to make sure that a global’s type stays consistent.
I think first I would like to figure out exactly what is going on with the task system. I can always read data out of the redirection pipe with readline or something like that. I tested this and it works.
But why the task never actually runs but reports that it did is something that I think is important. Perhaps it is a problem with my setup; not sure but I was hoping there was a clear answer or some global task error queue I could check.
@mkitti - I feel the task question deserves a new thread on discourse or on slack. What is the best way to elevate this question a bit? Or perhaps get a few more eyes on that specific question in this thread? I think it comes down to exactly how the @async macro is implemented as it feels to me like there is some interaction with ccall inside the macro. If the code is pure Julia code, everything works fine. As soon as I try to call back to clojure using a C fn ptr then async call fails.
A new Discourse post would be useful in either the Embedding or Internals topic. We could then chat about it in Slack in the corresponding channels.
A clear title would be useful.
Is this behavior dependent on JULIA_COPY_STACKS=1?
Great question. I do not set that environment variable anywhere so it is left to be the default. I assume the ccall convention has to work but I can set the variable to zero and see.
This is a nice write-up about Tasks and Threads in Julia:
The ALWAYS_COPY_STACKS variable eventually came to be controlled by the JULIA_COPY_STACKS environmental variable. What I have figured out is that the JVM does not like the the Task mode where where the stack is not copied.
Also if you are actually multithreading on the Julia side, then you might need to be very careful about which what JNIENV pointers you are using since you need a different one per thread. The problem is that Julia Tasks were purposely designed to abstract the relationship to Threads, so you don’t really know what Thread a Task might be running on.
Thanks, that is super helpful. Will have more test results soon.
JULIA_COPY_STACKS=1 allows tasks to at least run and callback into clojure. The IO redirection, however, is still failing.
Hmm, perhaps the best way to do redirection is to call redirect_xxx and then get the raw libuv handle from the pipe and install a read callback on it using libuv. This sidesteps the task system entirely.
Trying to use your LibJulia library - I’m having trouble with first line of
jna.clj as it utilizes julia_clj.JLOptions
From leinegen repl I’m using -
(require '[libjulia-clj.julia :as julia])
What I’ve done -
lein repl
At user prompt used:
user> (require '[libjulia-clj.julia :as julia])
I’m getting:
Syntax error (ClassNotFoundException) compiling at (libjulia_clj\impl\jna.clj:1:1).
julia_clj.JLOptions
Note - I found that Julia 1.5.4 does have the Options.jl library. My path for
julia_home is pointing to Julia 1.5.4 install directory. I have not checked for
later Julia install on my system. My julia_home is set to point to Julia 1.5.4
install directory.