Using Julia Sockets to send message to an existing TCP Socket

Hi, I have a problem to send a simple message to an existing TCP server, using Julia Sockets module.

I used one of the “C” examples on the web to test if I can make a connection and send/receive messages to/from the server. This works fine. This C code (called client_c.c) is below:

#include<stdio.h> //printf
#include<string.h>    //strlen
#include <unistd.h>
#include<sys/socket.h>    //socket
#include<arpa/inet.h> //inet_addr

int main(int argc , char *argv[])
{
    int sock;
    int proto = 6; // TCP protocol
    struct sockaddr_in server;
    char message[100] , server_reply[1024];

    //Create socket
    sock = socket(PF_INET , SOCK_STREAM , IPPROTO_TCP);

    if (sock == -1)
    {
        printf("Could not create socket");
    }
    puts("Socket created");

    server.sin_addr.s_addr = inet_addr("198.214.229.46");
    server.sin_family = AF_INET;
    server.sin_port = htons( 22401 );

    //Connect to remote server
    if (connect(sock , (struct sockaddr *)&server , sizeof(server)) < 0)
    {
        perror("connect failed. Error");
        return 1;
    }

    puts("Connected\n");
    memcpy(message,"iraf", 4);

    if ( send(sock,message,100,0) < 0 )
    {
        puts("Failed to send message");
        return 1;
    }

    if ( recv(sock, server_reply, 1024, 0) < 0 )
    {
        puts("Failed to receive message");
    }
    printf("%s\n",server_reply);

    close(sock);
    return 0;
}

After compiled to an executable called client_c and upon running it results in the following:

shell> ../../client_c
Socket created
Connected

  2  6 Thu Feb 10 2022( 41) 18:15:35.1 0 0 0
 LST/HA +20:42:42.34  - 0: 0: 0.37   180.0  50.8  1.5792
 MEAN   +20:41:28.96  -20:11:12.6  2000.0   0.000    0.00
 APP    +20:42:42.99  -20: 6:36.9  2022.1
 REFRA  +20:42:42.71  -19:59:55.8  2022.1
 RAW    +20:42:42.71  -19:59:55.8  2022.1
 107    F    36974

The last 7 rows are the messages from this TCP server.

Now, I tried to do the same thing in Julia by the following:
(Please note that I am pretty new to this).

# Import Sockets module
using Sockets

# Establish socket connection to the TCP server
tcs = connect(TCPSockets(;delay=false), IPv4("192.214.229.46"), 22401)

# Create 100 byte Char array with command "iraf"
a = Array{Char,1}(undef,100)
a[1:4] = collect("iraf")

# Write the message to the socket
write(tcs,String(a))

# Read message back from the server
read(tcs)

close(tcs)

Please note that the TCP server expects a message from a client to be 100 byte long string and that is why I created char array “a” of length 100, put the message, and converting it to String when writing it to the socket.

When running this in REPL, write(tcs,String(a)) return 100 which is the length of the message, but it hangs at read(tcs) line. Upon re running it up to write(tcs,String(a)), I saw tcs having no bytes. I am supposing that this is why read() hangs, but not sure whether the way I am connecting to the server in Julia is identical to that in client_c.c (?).

Also note that this TCP server is quite old and is a complete black box…

I really wish experts can give a pointer to a possible resolution.
I appreciate any help.

1 Like

The fact that this is undef looks like a problem. The 100 Chars in this array could be anything which happens to be in memory (except for the first four set to "iraf"). This same bug occurs in the C code, but you might just be lucky there and the rest of the buffer might be set to zeros.

Note also that Char is any Unicode character, so is 32 bits in size. String(a) will construct a UTF-8 encoded string from that, which might happen to be 100 bytes long, but it could be longer. This is probably not what you want! Instead I’d try

buf = zeros(UInt8, 100)
buf[1:4] = codeunits("iraf")  # works because "iraf" is ASCII, so there's one byte per char here in codeunits.

write(tcs, buf)

This might not fix the problem, but it’ll be a step in the right direction.

2 Likes

Hello Chris, Thank you for pointing out this.
I tried it but without success unfortunately.

Out of curiosity, I used one of Python examples from the web.

The code is as follows:

import socket

# Create a TCP/IP socket
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

# Connect the socket to the port where the server is listening
server_address = ('198.214.229.46', 22401)
print ('connecting to %s port %s' % server_address)
sock.connect(server_address)

try:
    
    # Send data
    message = 'iraf                                                                                                '
    print ('sending "%s"' % message)
    sock.sendall(message.encode('ascii'))

    # Look for the response
    amount_received = 0
    amount_expected = len(message)
    
    
    data = sock.recv(1024)
    print ('received "%s"' % data)

finally:
    print ('closing socket')
    sock.close()

Note that message is 100 in length (bytes?) and gets encoded in ascii in sendall() method, but it worked the same even if it was encoded in the default ‘utf-8’. ‘utf-16’ didn’t work.
Upon running it, I get the expected response (like in the c example) back from the server as follows:

[lee@atlas2 test]$ python test2tcs.py
connecting to 198.214.229.46 port 22401
sending "iraf                                                                                                "
received "  2  6 Fri Feb 11 2022( 42) 18:40:41.9 0 0 0
 LST/HA +21:11:49.83  - 0: 0: 0.28   180.0  50.8  1.5792
 MEAN   +21:10:37.96  -20:11:54.3  2000.0   0.000    0.00
 APP    +21:11:51.06  -20: 6:39.0  2022.1
 REFRA  +21:11:50.10  -19:59:57.8  2022.1
 RAW    +21:11:50.10  -19:59:57.8  2022.1
 107    F    36616
"
closing socket

Note that Python 2.6.6 was used to run this.

For a novice like me, the only difference of the Julia script that I can tell from the C / Python codes is the way the TCP/IP socket is defined at the beginning. In both C and Python examples, the socket gets defined with parameter such as AF_INET and SOCK_STREAM. Changing SOCK_STREAM to SOCK_DGRAM resulted in connection refused. So, the server seems clearly operating in STREAM mode. It is unclear to me how Julia Sockets creates a TCP Socket when issuing connect() method. At least I could not find information how this is set up in Sockets.jl documentation.

I suppose I could write what I need in C and call it from Julia to get around this problem, but it seems like I am missing some basic thing in using Sockets.jl to do this.

Thanks for the help. appreciate it.

Update
I think I made it work.

I wasn’t paying attention at all to the asynchronous nature of Sockets.jl which appears based on libuv package.

The revised code is now,

using Sockets
using Observables

# Function to read from the server asynchronously and update Observable 'reply' when there is a response from the server
function readTCS(tcs::TCPSocket,reply)
    task = @async try
        while isopen(tcs)
            In = String(readavailable(tcs))
            if !isempty(In)
                println("Read from TCS, check reply.")
                reply[] = In
            end
        end
        println("readTCS task finished normally.")
    catch e
        println("readTCS task finished with $e.")
    end
    return task
end

# Connect to the server and display socket and peer details
inet_addr = Sockets.InetAddr(IPv4("198.214.229.46"),22401)
tcs = Sockets.connect(inet_addr)
sockname = getsockname(tcs)
peername = getpeername(tcs)
println("getsockname(tcs) returns : ip $(sockname[1]) port $(sockname[2]) : host")
println("getpeername(tcs) returns : ip $(peername[1]) port $(peername[2]) : remote")

# Set up an observable to monitor changes in the response from the server
reply = Observable("")
reply_change = observe_changes(reply)

# Run readTCS() task
readTask = readTCS(tcs,reply)

# Set up even handler when Observable reply changes and print out the changed value (String)
replyObs = on(reply_change) do drep
        println("reply[] ==> $drep")
end

# Set up an example command to the server
N = 100
cmd = "iraf"
ncmd = length(cmd)
buf = zeros(UInt8,N)
buf[1:ncmd] .= codeunit(cmd)

After running this portion of the script, buf can be written to tcs and the corresponding message (String) from the server is stored to reply and printed out as,

julia> write(tcs,buf);

julia> Read from TCS, check reply.
reply[] ==>   2  6 Fri Feb 11 2022( 42) 21:48:51.6 0 0 0
 LST/HA + 0:20:30.49  + 0: 0: 0.04   180.0  50.8  1.5792
 MEAN   + 0:19:25.86  -20:13:46.6  2000.0   0.000    0.00
 APP    + 0:20:31.28  -20: 6:40.7  2022.1
 REFRA  + 0:20:30.45  -20: 0: 0.0  2022.1
 RAW    + 0:20:30.45  -20: 0: 0.0  2022.1
 107    F    36616

julia> split(reply_change[],'\n')
8-element Vector{SubString{String}}:
 "  2  6 Fri Feb 11 2022( 42) 21:48:51.6 0 0 0"
 " LST/HA + 0:20:30.49  + 0: 0: 0.04   180.0  50.8  1.5792"
 " MEAN   + 0:19:25.86  -20:13:46.6  2000.0   0.000    0.00"
 " APP    + 0:20:31.28  -20: 6:40.7  2022.1"
 " REFRA  + 0:20:30.45  -20: 0: 0.0  2022.1"
 " RAW    + 0:20:30.45  -20: 0: 0.0  2022.1"
 " 107    F    36616"
 "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0" ⋯ 33 bytes ⋯ "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"

I suppose the reason that read() hanged in my first post was due to read() was blocking everything whereas in readTCS() above it is running asynchronously and hence non blocking.

1 Like

I don’t think you need the task-handling for the simple read() you’re doing. With a few lines of Python to mimic the server you’re using your original Julia code seems to work fine:

melis@juggle 12:25:~$ cat s.py 
import socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM, 0)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind(('', 22401))
s.listen(1)
t,a = s.accept()
r = t.recv(100)
print('got %d bytes: %s' % (len(r), r))
t.sendall('hello!'.encode('utf8'))

melis@juggle 12:25:~$ cat s.jl
# Import Sockets module
using Sockets

# Establish socket connection to the TCP server
tcs = connect(TCPSocket(;delay=false), IPv4("127.0.0.1"), 22401)

# Create 100 byte Char array with command "iraf"
a = Array{Char,1}(undef,100)
a[1:4] = collect("iraf")

# Write the message to the socket
write(tcs,String(a))

# Read message back from the server
r = read(tcs)
println("read '$(r)'")

close(tcs)

# terminal 1 
melis@juggle 12:27:~$ python s.py 
got 100 bytes: b'iraf\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'

# terminal 2
melis@juggle 12:27:~$ j s.jl 
read 'UInt8[0x68, 0x65, 0x6c, 0x6c, 0x6f, 0x21]'

After all, your Python of the client also used a blocking call to sock.recv(1024) and it worked, so I don’t see why the Julia equivalent would not work similarly.

Edit: one thing that is different in the Julia code is that the read() call doesn’t specify the amount of bytes to read. That might cause a longer wait before it returns. E.g. you can try read(tcs, 40) to read at most 40 bytes.

3 Likes

That is exactly what I thought Sockets.jl should be able to do. Thanks for doing a quick test with the python server and julia client. This is a great pointer for me.

I tried this on this server without the task handler, but with readavailable() method rather than read().
And it worked. In this case, I commented out the task handler readTCS() and the observable event handler.

read(tcs) still hangs, but as you said I tried it with the number of bytes specified (400 in this case) and voila! it worked.

julia> writeTCS(tcs,"iraf"); a = split(String(read(tcs,400)),'\n')
Request sent to tcs normally.
8-element Vector{SubString{String}}:
 "  2  6 Sat Feb 12 2022( 43) 18:25: 0.2 0 0 0"
 " LST/HA +21: 0: 2.14  - 0: 0: 0.69   180.0  50.8  1.5791"
 " MEAN   +20:58:50.30  -20:11:37.7  2000.0   0.000    0.00"
 " APP    +21: 0: 3.82  -20: 6:37.8  2022.1"
 " REFRA  +21: 0: 2.83  -19:59:59.1  2022.1"
 " RAW    +21: 0: 2.83  -19:59:59.1  2022.1"
 " 107    F    37306"
 "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0" ⋯ 33 bytes ⋯ "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"

Note that writeTCS() is a method I added to make it easy for me to parse a string command into 100-byte long message to the server as,

function writeTCS(tcs::TCPSocket,request::String)
   Lcmd = length(request)
   Lmax = 100
   try
        @assert 0 < Lcmd <= Lmax
        buf = zeros(UInt8,Lmax)
        buf[1:Lcmd] .= codeunits(request)
        @assert write(tcs,buf) == Lmax
        println("Request sent to tcs normally.")
   catch e
        println("Failed to send request: $e.")
   end
end

But when I tried to read more than 400 bytes after writeTCS(), read(tcs,N) where N > 400 hangs. So, it looks like the server response is 400-byte long for this particular request (a piece of new information about this server). I think this should do the job for me, but readavailable(tcs) seems working as well, in which case I don’t think I have to know the exact number of bytes of the server response. The size of server responses for other requests may be different (nothing is said about it in its manual). So, readavailable() would work safe for the time being.

I think my problem has been solved and I learned a lot from it. Thanks very much for all great suggestions!

Ah of course, this will certainly be the problem: read() will read bytes from the socket until it’s closed. Which will always block forever unless the remote side closes the connection.

readavailable() might work for you, but it’s not very reliable in general — it’ll only read “some” bytes which might be less than the number of bytes sent by the remote side. And if you call it when no data is available it can block until one byte is available.

Ideally it’s better to use read() with the exact message size so you know you got the whole packet. Well-designed protocols generally have some framing with a fixed size header containing the total size of the next message packet for this purpose. It sounds like this is all implicit for the black box you’re trying to talk to. My suggestion would be to figure out the response size for each different request type (hopefully it never varies!?) then hardcode that somewhere.

(For this awkward situation it would be best if Julia had an easy way to “read everything until a timeout”, but that’s not builtin. You can simulate it with a readavailable loop in a separate task and some Channel plumbing, etc, but it really should be easier.)

4 Likes

That is a great point Chris. And I now understand why read hangs in this case. I am going to follow your suggestion to find out the response sizes and hard code them in my client script. Thanks!

Sure, no problem, I hope the response size from the server is fixed!

(It would be possible for the server to be particularly badly behaved and give a variable response size with no message framing. If this was the case you’d be forced into heuristics with timeouts and message parsing. I don’t think I’ve seen a case of this in practice before, however.)

It has to signal that the message is done somehow. The options are basically:

  1. Close the connection.
  2. Convey the length in advance.
  3. Send an end of message marker inline.