Julia to other language communication

New to Julia
1

If I want to communicate between processes written in different languages on Linux, I typically use pipes. I couldn’t find any documentation on this. However, I did find examples of ZeroMQ.

2

Do you have a specific language or use case in mind?

3

I would like to communicate between Julia and Python. I have a data-science application which uses Julia, but I’d like to create a basic REST server with Flask. Although I am investigating writing the REST server with Julia, I’m passing off deployment to another team familiar with Python. Consequently, it would be ideal if Julia was abstracted away as much as possible. Previously, I tried using PyJulia, however I ran into threading issues.

4

You can use pipes perfectly fine with Julia (at least, if just using stdin and stdout). What kind of data are you sending/receiving? Arrays? Strings? Arbitrary nested structures? Etc.

5

Strings of arbitrary length, but probably not larger than 256 characters.

6

FYI, one way to workaround this is to make a “in-process service” by using PyJulia in a dedicated thread connected with other threads using (say) queue.Queue or collections.deque.

1 Like
7

I’d check these options in the following order:

  1. Try to resolve threading issues. Usually when deploying Flask application we use multiple worker processes, so I’m surprised threading appeared to be a problem.
  2. REST API in Julia, possibly wrapped into a richer API in Flask. HTTP is really a common ground - rich format, easy to implement, easy to use from any client.
  3. Pipes. Much fewer tools for data transfer than in HTTP, but if you are used to it, it might be a good starting point.
  4. ZeroMQ or any other protocol.
8

Ah, I misunderstood the Python GIL constraint and will keep this in mind.

9

It’s actually libjulia’s problem. Currently it’s not thread-safe: https://github.com/JuliaLang/julia/issues/17573

But GIL could play a role if you consider the throughput of the REST service. It means that the service would be blocked whenever Julia does some calculation.

10

Here is one similar question with different option: Which is the right way to integrate Julia program with Node.js

11

I’m partial to tcp sockets myself. They are also dead-simple with Python, but if you want to go even bare-bones-er you could make a pair of fifos, since you like working with pipes. As far as I know, Julia doesn’t come with a function for this (or a function for hardlinks, weirdly), but it’s easy to write one:

function mkfifo(pathname, mode=0o666)
    err = ccall(:mkfifo, Cint, (Cstring, Cuint), pathname, mode)
    systemerror("couldn't make fifo, $(repr(pathname))", err != 0)
    pathname
end

Alternatively, you could just use Python’s os.mkfifo. works (more or less) the same as the above.

When you make the fifo, you can just open it and read and write to it like a normal file, but it blocks until it’s been opened both for reading and writing simultaneously. I only ever use them for one-way stuff, and TCP sockets for two-way.

12

As far as I know IJulia also uses ZeroMQ for communicating with Jupyter.

13

Does making libjulia thread-safe fall under the “mutlithreading” compiler development priority?

14

This is very low-level part of Julia core for me so honestly, I don’t know. Having said that, my guess is that the parallel task runtime thing is unrelated. I thought “thread safety” in the list refers to thread safety of Julia code (i.e., usage of the core API), not about libjulia.