Calling a Julia app from Python (communicating through stdin)

I am trying to run an app that takes a Python script and calls a Julia app. It should be sending JSON to the Julia app and getting back a text line.

The Python side has code like

from subprocess import Popen, PIPE
...
agent_process = Popen(["JuliaAgent/bin/LuxAI"], stdin=PIPE, stdout=PIPE, stderr=PIPE, cwd=cwd)
...
agent_process.stdin.write(message.encode())
agent_process.stdin.flush()
...
agent1res = (agent_process.stdout.readline()).decode()

I have tested the app JuliaAgent/bin/LuxAI with directly from the shell and it works fine. I generated it using PackageCompiler. However, when I try to run it through the Python script it doesn’t seem like it gets anything back from the agent (maybe something with the connection closing out)…

I tried debugging it with Julia with

open(`JuliaAgent/bin/LuxAI`) do io
    print(io, r"""{"updates": ["1", "16 16", "rp 0 0", "rp 1 0", "r uranium 0 14 321", "r uranium 0 15 341", "r uranium 1 15 329", "r wood 2 7 800", "r wood 2 8 800", "r wood 2 9 800", "r coal 2 10 351", "r wood 3 2 307", "r wood 3 4 301", "r coal 3 9 380", "r coal 3 10 364", "r wood 4 2 369", "r wood 4 3 366", "r wood 4 4 318", "r coal 4 9 367", "r wood 5 2 326", "r wood 5 3 376", "r wood 10 2 326", "r wood 10 3 376", "r wood 11 2 369", "r wood 11 3 366", "r wood 11 4 318", "r coal 11 9 367", "r wood 12 2 307", "r wood 12 4 301", "r coal 12 9 380", "r coal 12 10 364", "r wood 13 7 800", "r wood 13 8 800", "r wood 13 9 800", "r coal 13 10 351", "r uranium 14 15 329", "r uranium 15 14 321", "r uranium 15 15 341", "u 0 0 u_1 1 8 0 0 0 0", "u 0 1 u_2 14 8 0 0 0 0", "c 0 c_1 0 23", "c 1 c_2 0 23", "ct 0 c_1 1 8 0", "ct 1 c_2 14 8 0", "ccd 1 8 6", "ccd 14 8 6", "D_DONE"], "step": 0}""")
    txt = readline(io)
    @show txt
end

but it throws the following error

txt = "ArgumentError(\"invalid JSON at byte position 0 while parsing type Any: UnexpectedEOF\\n\\n\")"
fatal: error thrown and no exception handler available.
MethodError(f=Base.convert, args=(Int32, ArgumentError(msg="invalid JSON at byte position 0 while parsing type Any: UnexpectedEOF

I guess something is happening with EOF or how the text is being send/received. In the app the receiving part is

inputs = readline()
observation = JSON3.read(inputs)
...
print(join(actions, ", "))

Any thoughts on what I should investigate or if it is something obvious someone can point out from what I shared… Thanks!

Unfortunately I cannot directly answer your question, but have you considered using PyJulia/PyCall or a REST API for communication between Julia and Python?
Both approaches have the advantage of keeping the Julia session open so that there is no compile-lag after first usage. In addition, data transfer is more powerful/ cleaner and (at least for the PyJulia solution) much faster.

The submission for this has to be a Python file so I tried PyJulia, however I hit a few issues one being
https://github.com/JuliaPy/pyjulia/issues/450
which messes up with the API by polluting the API stdout. It also seems to take a few seconds at startup that are crucial for a 3-seconds time limit on responding per turn.

It wasn’t all bad since it forced me to learn a few new things like how to use PackagePrecompiler to create executable apps with Julia and custom system images. However, I am running out of things to try to fix this issue… Don’t know if it comes from some undocumented behavior that I am just not familiar with… maybe something like
https://github.com/JuliaLang/julia/issues/30790