Data-exchange Julia-LabView

[martincornejo]

For an experiment I need for a Julia process to communicate with a LabView process and exchange data every few seconds (1-5 s). Since (I think) there is no package to directly bind Julia and LabView I was thinking of connecting both with a TCP-socket and exchange the data in a common format (maybe HDF5?).

Any suggestion is appreciated

[Eben60]

Another possibility could be using Python as middleware via PyJulia. LabVIEW 2018+ supports some Python communication natively (however I tried it and somehow coudn’t get numpy). There is also Python Intergartion Toolkit from Enthought. I didn’t try it, but they have a 30-day eval version.

[Eben60]

@martincornejo

(maybe HDF5?)

or JSON? - JSON is well supported by LV, I beleive.

[martincornejo]

Thanks! JSON might be the simplest solution, I do think it is well supported on LabView.

I also thought on the python as intermediator, but that would be over-complicating it too far…

[pdeffebach]

Take a look at TableView.jl, the lazy loading yields big performance gains that are non-trivial to work out. I think it works by writing JSON as an intermediary.

[Iulian.Cioarca]

How much data are you exchanging? Maybe a simple write/read to disk is enough, given the refresh rate in the order of seconds.

[Eben60]

I know next to nothing about ZeroMQ, but there exist Julia as well as LabVIEW binding for it.

[martincornejo]

That was kind of the initial implementation: reading and writing CSV files. I just wanted to try a more “elegant” solution if I had to increase the data exchange rate.

[Eben60]

Installed ZMQ.jl and labview-zmq without a glitch. Getting data from LabVIEW client to Julia ZMQ server and back turned out to be trivial. Launching and stopping Julia ZMQ server from LV already done, now I’m going to add the JSON part and then would publish on the Github.

[martincornejo]

Nice! Do publish the repository, maybe I can help with what I’ve learned on binding Julia and LabView with JSON.

[martincornejo]

This is a very minimal implementation of a Julia client to LabView server interaction

using Sockets
using JSON

# Establish connection
lv = connect(5050)    # TCP-Socket on port 5050 (localhost)

# Send data
data = Dict("A" => rand(5), "B" => rand(5))
msg = JSON.json(data)
write(lv, msg)

# Receive data
str = String(readavailable(lv))
data = JSON.parse(str; dicttype=Dict{String,Vector{Float64}})

I think the most important and tricky step to consider is implementing readavailable to receive the JSON-string from LabView, else Julia doesn’t know how many bytes to read from the buffer.

[Eben60]

Progress report:

What is done - see the block diagram below:

We start server from LV by launching a Julia script in a Terminal:

and stop it by sending it a command to stop.

The VI in the first diagram is a Functional Global Variable, saving the state of the client and server. Upon initialization it can be then called independently from different parts of the LV program (first come first served).

I have then specified the data exchange protocoll:

The Request (from LV) contains JSON string and an optional binary part. The JSON dict contains the name of the function to be called by Julia server and (optionally) keyword arguments to be passed to this function. The binary data, if there are any, will be passed as an argument, too.

The Response contains a JSON encoded dict corresponding to LV error cluster, optional JSON encoded data, and optional binary data.

But as this project has little to do with my current work, and the weekend happened to be just 2 days long, I didn’t manage to get it at least half-ready . Maybe next weekend

[Eben60]

This is a working example (on the LabVIEW side). The both loops access the Julia server concurrently. The roundtrip time for light loads is well below 1 ms on the average and below 20 ms max

For large binary data sent from LV to Julia the roundtrip time grows by approx. 4 ms/MB.

The project is undocumented yet and needs a thorough clean-up, but is already usable (see example3.vi and example4.vi).

How to use:

• Install ZMQ for LV and ZMQ.jl and JSON3.jl on Julia side.
• On the LV side, you define a cluster with field names corresponding to the arguments you want to pass to your Julia function, and an additional string field named “fun2call” for the name of your Julia function. You may additionaly create a byte array to be passed as binary data
• On the Julia side you define your functions with corresponding names, which must accept keyword arguments only, argument names see above. For binary data, if any, additional keyword argument “bin_data”. The result must be a named tuple or anything else, convertable to JSON.
• On the LV side you parse the JSON string to a cluster. Additionally it is your responsibility to parse binary data returned by Julia server (returning binary data not implemented yet).

TODO:

• Clean-up
• Documentation
• Error handling
• Returning binary data
• Utilities to encode/decode some common data types to binary data (arrays of different floats and integer types)
• Timeouts
• LabVIEW server and Julia client

Timescale undefined (and depends on the interest), but I hope to deal with the first two items of the list on a reasonable timescale.

The LabVIEW part is LV2018. If anybody interested, I could try to save the library for an earlier version.

[Eben60]

Progress report

• Clean-up
  • has been done once long ago
• Documentation
  • TODO (mostly)
• Error handling
  • On error, Julia server sends error message & stack trace (which are parsed by LabVIEW), and continues to run
  • In LabVIEW code, all checks I could think of
• Returning binary data
  • (see next item)
• Utilities to encode/decode some common data types to binary data (arrays of different floats and integer types)
  • Data sent as byte array, corresponding data descriptors withing JSON data
  • Arrays of most numeric types (namely intersecting set of LV and Julia supported types):
    • Float32, Float64 and respective Complex
    • Signed and unsigned integer 8, 16, 32, and 64 bit
    • Boolean (which is not a numeric type in LV)
    • 1, 2, and 3-dimensional arrays (those are different types in LV)
  • RGB Images (8 bit/ch.)
    • TODO grayscale images; stacks of images
• Timeouts
  • not sure what and how
• LabVIEW server and Julia client
  • TODO - shouldn’t be difficult now

On LabVIEW, usage looks like following:

Sending data (2 arrays, 2 scalar arguments):

Receiving processed data (2 arrays, one scalar value):

On Julia side, user provides a script file like following:

function foo(;arg1, arg2_string, arg3_arr_Cx32, arg4_arr_i16)

    # doing some stuff, e.g. check if we got the right data type
    @assert typeof(arg3_arr_Cx32) == Array{Complex{Float32},2}
    @assert typeof(arg4_arr_i16) == Vector{Int16}
    # some more stuff

    # get data to return, e.g.
    resp1 = rand()
    resp_arr_I32 = Int32.(vcat(arg4_arr_i16, [1,2,3]))
    resp_arr_F64 = arg1.*abs.(vec(arg3_arr_Cx32)) .+ sqrt(2)

    return (;resp1, bigarrs=(;resp_arr_I32, resp_arr_F64))
end

function bar(;kwargs...)
    # doing some stuff
end

fns = (;foo, bar)

Path to this script to be passed to LabVIEW, which starts Julia server and passes user functions to it.

P.S. LabVIEW is statically typed, which means a lot of boilerplate

P.P.S. Thanks a due to Covid-19 pandemic, which provided me with sufficient spare time.