I’m able to set up TCP and UDP to send some basic data back and forth, but I can’t figure out how to send the bits-type data from some simple immutable structs (short of sending each field separately). In C, I would just use a pointer to the_struct and send sizeof(the_struct) bytes. I thought it would be similar with Julia using the unsafe_convert/unsafe_wrap functions, but I can’t figure out how. The ultimate goal is to send structured data from Julia to C running on an embedded processor. Anyone have some pointers for me? Is there a better way to do this altogether?
2 Likes
Don’t use unsafe_convert and unsafe_wrap, they are unrelated and in particular unsafe_wrap is not allowed to be used on object pointers.
There are multiple ways to do it, the simplest way is basically replace C pointers with Ref
Server,
julia> sock = listen(ip"127.0.0.1", 9000)
Base.TCPServer(RawFD(17) active)
julia> client = accept(sock)
TCPSocket(RawFD(19) open, 0 bytes waiting)
julia> read!(client, Ref{Tuple{Int,Float64}}())
Base.RefValue{Tuple{Int64,Float64}}((1, 2.0))
client
julia> sock = connect(ip"127.0.0.1", 9000)
TCPSocket(RawFD(17) open, 0 bytes waiting)
julia> v = (1, 2.0)
(1, 2.0)
julia> write(sock, Ref(v))
16
2 Likes
@yuyichao, thanks! This is helpful.
Maybe there’s a version issue here (I’m v0.6): when I get to your read! line though, it can’t find a matching read!.
julia> read!(client, Ref{Tuple{Int,Float64}}())
ERROR: MethodError: no method matching read!(::TCPSocket, ::Base.RefValue{Tuple{Int64,Float64}})
Closest candidates are:
read!(::IO, ::BitArray) at bitarray.jl:2007
read!(::AbstractString, ::Any) at io.jl:161
read!(::IO, ::Array{UInt8,N} where N) at io.jl:387
...
I used z = (1, 1.); x = read(client, Ref(z)); payload = x[] to get the data.
Also, do you know how this would be different with UDP? UDP doesn’t seem to support read and write, but uses recv and send, with different interfaces that don’t quite map to the way this works.
We should probably try to add those if they are not there. In the mean time you should be able to use an array to send the data. (recv doesn’t really support that but you can copy the data)
I think the part I’m missing is how to map a struct to an array of UInt8s (and the reverse). If I had the UInt8s, then send and recv will work for me. I feel like I’m missing something really basic.
reinterpret for input and unsafe_copy! for output.
Thanks again, @yuyichao; I have it working. Dealing with pointers to immutables structs was very confusing to me as a new Julia user who’s familiar with C. For future readers, let me summarize what I got working. Perhaps others will know a cleaner way to do this!
Listener side:
sock = UDPSocket(); # Create the socket.
bind(sock, ip"127.0.0.1", 2000); # Listen on port 2000.
x = recv(sock); # Get an array of UInt8s from somewhere.
Sender side:
struct MySt; a::UInt32; b::Float64; end; # Create a super-simple immutable struct type.
m = MySt(1, 2.3); # Create an instance of the type.
sock = UDPSocket(); # Get ready to send some UDP.
x = reinterpret(UInt8, [m]); # Turn m into an array of UInt8, stored in x.
send(sock, ip"127.0.0.1", 2000, x); # Send it.
Note in the above that the call to reinterpret requires an array of the struct! I don’t really follow why. There are several long discussions about this around the internet; the longest actually currently ends with, “Wait, so why is it fine to convert an array of one immutable struct to bytes but not a scalar?” So, I don’t know, but we need to use an array as far as I can tell.
Back over to the listener, we now have x:
julia> x
16-element Array{UInt8,1}:
0x01
0x00
...
struct MySt; a::UInt32; b::Float64; end; # Define the same type on this end.
ma = [MySt(0, 0.)]; # Create an ARRAY to write to.
pma = reinterpret(Ptr{UInt8}, pointer(ma)); # Pretend that a pointer to ma is actually a pointer to UInt8.
unsafe_copy!(pma, pointer(x), 16); # Copy 16 units of whatever type (UInt8 in our case) to our pointer to the array from a pointer to the received data.
julia> m = ma[1] # Finally, pull out the data from our array.
MySt(0x00000001, 3.141592653589793)
It worked! Again, note that we can’t get a pointer to an instance of our struct; we have to get a pointer to the beginning of an array of our struct.
In C, the UDP part takes much more code, but the “just copy from this memory location to that one” part is far less confusing (recvfrom(sock, &m, 16, ...)). Does anyone see a good way to clean up my code above to make this more straightforward? I’d be happy to contribute something to the codebase, but my Julia-fu is clearly not where it needs to be yet.
[Despite that credit clearly goes to @yuyichao for the help, I’m marking this reply as the solution since it contains the working answer all in one post, and I’ll edit as necessary to reflect any new additions.]
1 Like
Fast forwarding to 2023 and I wonder what is the most appropriate way to do that. @yuyichao solution works nicely. But using Serialization appears a bit less complicated:
using Sockets, Serialization
using UUIDs # use case: transfer UUIDs
# server
servsock = listen(Sockets.localhost, 9000)
sock = accept(servsock)
servid = deserialize(sock)
# client
clisock = connect(Sockets.localhost, 9000)
id = UUIDs.uuid4()
serialize(clisock, id)
However, every now and then people are saying that serialization is very sensitive to the julia version. When doing such remote stuff it’s highly likely that the julia versions are different. However I don’t see how to avoid that, outside of making a customized protocol per composite type. So for me it looks like the Serialization solution would be the way to go but keeping in mind for julia version potential troubles.
1 Like