Hi all
I am using PyCall to run a simple python script that prints out the ascii table.
#test_stdout_pycall.py
digit = 0
while digit<=255:
print("%d = %c" %(digit, chr(digit)))
digit += 1
print("done")
I want to read the output in a buffer in Julia
using PyCall
buf = IOBuffer()
pyimport("sys").stdout = buf
@pyinclude("test_stdout_pycall.py")
while buf != "done/n"
print( String(take!(buf)) )
end
BUT the Julia script only gets me the first 157 iterations ( I only show the 124:157) then just hangs on the "157 = " line. If I interrupt the script and print out buf it’s empty.
124 = |
125 = }
126 = ~
127 =
128 =
129 =
130 =
131 =
132 =
133 =
134 =
135 =
136 =
137 =
138 =
139 =
140 =
142 =
143 =
144 =
146 =
147 =
148 =
149 =
150 =
151 =
152 =
154 =
155 =
157 =
My expected behavior based on my read of the Julia documentation regarding take! ( see below) is that it would consume buf until some condition was met. In my case the bytes “done” which is sent from the python script after sending 256 lines comprising of the count and associated chr code. So what’s happening here. Here’s the doc on take!
Obtain the contents of an IOBuffer as an array. Afterwards, the IOBuffer is reset to its initial state.
I got the above from the official docs on take!
Maybe call sys.stdout.flush() at the end of your Python script to make sure Python isn’t buffering some output internally?
I’m also not sure why you have a while loop in Julia — the Python code isn’t executing asynchronously, it should be completely done by the time it leaves @pyinclude, so you should only take! the buffer once. (This will also cause your loop to not terminate — the buffer contains all the lines, not a single line, so it will never equal "done\n".)
On my machine, no flushing is required:
using PyCall
buf = IOBuffer()
pyimport("sys").stdout = buf
py"""
digit = 0
while digit<=255:
print("%d = %c" %(digit, chr(digit)))
digit += 1
print("done")
"""
print( String(take!(buf)) )
prints all of the lines including the done.
Be sure that your mental model is correct here. Julia is not running a separate python script and reading the output. PyCall is calling directly into libpython and executing the Python code within the Julia process, so each call to Python completes before returning.
I put the while in to see if there was some kind of default buffer length in the hope that take would flush the buf and get the next given the 157 line issue.
ALSO I ran your code ( thanks for that) and it did EXACTLY the same only printed 155 ( down from 157) and this time didn’t hang.
julia
_
_ _ _(_)_ | Documentation: https://docs.julialang.org
(_) | (_) (_) |
_ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help.
| | | | | | |/ _` | |
| | |_| | | | (_| | | Version 1.7.0 (2021-11-30)
_/ |\__'_|_|_|\__'_| | Official https://julialang.org/ release
|__/ |
julia> using PyCall
julia> buf = IOBuffer()
IOBuffer(data=UInt8[...], readable=true, writable=true, seekable=true, append=false, size=0, maxsize=Inf, ptr=1, mark=-1)
julia> pyimport("sys").stdout = buf
IOBuffer(data=UInt8[...], readable=true, writable=true, seekable=true, append=false, size=0, maxsize=Inf, ptr=1, mark=-1)
julia> py"""
digit = 0
while digit<=255:
print("%d = %c" %(digit, chr(digit)))
digit += 1
print("done")
"""
julia> print( String(take!(buf)) )
0 =
1 =
2 =
3 =
4 =
5 =
6 =
7 =
8 =
9 =
10 =
11 =
12 =
13 =
14 =
15 =
16 =
17 =
18 =
19 =
20 =
21 =
22 =
23 =
24 =
25 =
26 = �
27 =
8 =
29 =
30 =
31 =
32 =
33 = !
34 = "
35 = #
36 = $
37 = %
38 = &
39 = '
40 = (
41 = )
42 = *
43 = +
44 = ,
45 = -
46 = .
47 = /
48 = 0
49 = 1
50 = 2
51 = 3
52 = 4
53 = 5
54 = 6
55 = 7
56 = 8
57 = 9
58 = :
59 = ;
60 = <
61 = =
62 = >
63 = ?
64 = @
65 = A
66 = B
67 = C
68 = D
69 = E
70 = F
71 = G
72 = H
73 = I
74 = J
75 = K
76 = L
77 = M
78 = N
79 = O
80 = P
81 = Q
82 = R
83 = S
84 = T
85 = U
86 = V
87 = W
88 = X
89 = Y
90 = Z
91 = [
92 = \
93 = ]
94 = ^
95 = _
96 = `
97 = a
98 = b
99 = c
100 = d
101 = e
102 = f
103 = g
104 = h
105 = i
106 = j
107 = k
108 = l
109 = m
110 = n
111 = o
112 = p
113 = q
114 = r
115 = s
116 = t
117 = u
118 = v
119 = w
120 = x
121 = y
122 = z
123 = {
124 = |
125 = }
126 = ~
127 =
128 =
129 =
130 =
131 =
132 =
133 =
134 =
135 =
136 =
137 =
138 =
139 =
140 =
142 =
143 =
144 =
146 =
147 =
148 =
149 =
150 =
151 =
152 =
154 =
155 =
julia>
Try adding
import sys
sys.stdout.flush()
at the end of your Python code. (Or, equivalently, run pyimport("sys")."stdout".flush() in Julia before the take!.)
I was concerned that I wasn’t processing all of the buffer in one take, so that’s why I put the while loop in. I wanted to see if another take would “find” the rest of the buffer. When I read the docs I did see that the print would be completely sent in one buffer hence my bewilderment when it didn’t arrive. That’s why I put the while loop in.
Hi there
still doing the same thing. I removed the while loop and, as I want to concentrate on Julia, put in the buffer flush before the take!
I’m running Linux Mint 20.3 if that’s relevant and python 3.8
julia
_
_ _ _(_)_ | Documentation: https://docs.julialang.org
(_) | (_) (_) |
_ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help.
| | | | | | |/ _` | |
| | |_| | | | (_| | | Version 1.7.0 (2021-11-30)
_/ |\__'_|_|_|\__'_| | Official https://julialang.org/ release
|__/ |
julia> using PyCall
julia> buf = IOBuffer()
IOBuffer(data=UInt8[...], readable=true, writable=true, seekable=true, append=false, size=0, maxsize=Inf, ptr=1, mark=-1)
julia> pyimport("sys").stdout = buf
IOBuffer(data=UInt8[...], readable=true, writable=true, seekable=true, append=false, size=0, maxsize=Inf, ptr=1, mark=-1)
julia> @pyinclude("test_stdout_pycall.py")
julia> pyimport("sys")."stdout".flush()
julia> print( String(take!(buf)) )
0 =
1 =
2 =
3 =
4 =
5 =
6 =
7 =
8 =
9 =
10 =
11 =
12 =
13 =
14 =
15 =
16 =
17 =
18 =
19 =
20 =
21 =
22 =
23 =
24 =
25 =
26 = �
27 =
8 =
29 =
30 =
31 =
32 =
33 = !
34 = "
35 = #
36 = $
37 = %
38 = &
39 = '
40 = (
41 = )
42 = *
43 = +
44 = ,
45 = -
46 = .
47 = /
48 = 0
49 = 1
50 = 2
51 = 3
52 = 4
53 = 5
54 = 6
55 = 7
56 = 8
57 = 9
58 = :
59 = ;
60 = <
61 = =
62 = >
63 = ?
64 = @
65 = A
66 = B
67 = C
68 = D
69 = E
70 = F
71 = G
72 = H
73 = I
74 = J
75 = K
76 = L
77 = M
78 = N
79 = O
80 = P
81 = Q
82 = R
83 = S
84 = T
85 = U
86 = V
87 = W
88 = X
89 = Y
90 = Z
91 = [
92 = \
93 = ]
94 = ^
95 = _
96 = `
97 = a
98 = b
99 = c
100 = d
101 = e
102 = f
103 = g
104 = h
105 = i
106 = j
107 = k
108 = l
109 = m
110 = n
111 = o
112 = p
113 = q
114 = r
115 = s
116 = t
117 = u
118 = v
119 = w
120 = x
121 = y
122 = z
123 = {
124 = |
125 = }
126 = ~
127 =
128 =
129 =
130 =
131 =
132 =
133 =
134 =
135 =
136 =
137 =
138 =
139 =
140 =
142 =
143 =
144 =
146 =
147 =
148 =
149 =
150 =
151 =
152 =
154 =
155 =
julia>
It’s something to do with the print statement.
print( String(take!(buf)) )
This is what buf actually looks like if don’t print it out and just use
String(take!(buf))
the python code
digit = 0
while digit<=255:
print("%d = %c" %(digit, chr(digit)))
digit += 1
print("done")
the julia code
$julia
_
_ _ _(_)_ | Documentation: https://docs.julialang.org
(_) | (_) (_) |
_ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help.
| | | | | | |/ _` | |
| | |_| | | | (_| | | Version 1.7.0 (2021-11-30)
_/ |\__'_|_|_|\__'_| | Official https://julialang.org/ release
|__/ |
julia> using PyCall
julia> buf = IOBuffer()
IOBuffer(data=UInt8[...], readable=true, writable=true, seekable=true, append=false, size=0, maxsize=Inf, ptr=1, mark=-1)
julia> pyimport("sys").stdout = buf
IOBuffer(data=UInt8[...], readable=true, writable=true, seekable=true, append=false, size=0, maxsize=Inf, ptr=1, mark=-1)
julia> @pyinclude("test_stdout_pycall.py")
julia> pyimport("sys")."stdout".flush()
julia> String(take!(buf))
"0 = \0\n1 = \x01\n2 = \x02\n3 = \x03\n4 = \x04\n5 = \x05\n6 = \x06\n7 = \a\n8 = \b\n9 = \t\n10 = \n\n11 = \v\n12 = \f\n13 = \r\n14 = \x0e\n15 = \x0f\n16 = \x10\n17 = \x11\n18 = \x12\n19 = \x13\n20 = \x14\n21 = \x15\n22 = \x16\n23 = \x17\n24 = \x18\n25 = \x19\n26 = \x1a\n27 = \e\n28 = \x1c\n29 = \x1d\n30 = \x1e\n31 = \x1f\n32 = \n33 = !\n34 = \"\n35 = #\n36 = \$\n37 = %\n38 = &\n39 = '\n40 = (\n41 = )\n42 = *\n43 = +\n44 = ,\n45 = -\n46 = .\n47 = /\n48 = 0\n49 = 1\n50 = 2\n51 = 3\n52 = 4\n53 = 5\n54 = 6\n55 = 7\n56 = 8\n57 = 9\n58 = :\n59 = ;\n60 = <\n61 = =\n62 = >\n63 = ?\n64 = @\n65 = A\n66 = B\n67 = C\n68 = D\n69 = E\n70 = F\n71 = G\n72 = H\n73 = I\n74 = J\n75 = K\n76 = L\n77 = M\n78 = N\n79 = O\n80 = P\n81 = Q\n82 = R\n83 = S\n84 = T\n85 = U\n86 = V\n87 = W\n88 = X\n8" ⋯ 768 bytes ⋯ "180 = ´\n181 = µ\n182 = ¶\n183 = ·\n184 = ¸\n185 = ¹\n186 = º\n187 = »\n188 = ¼\n189 = ½\n190 = ¾\n191 = ¿\n192 = À\n193 = Á\n194 = Â\n195 = Ã\n196 = Ä\n197 = Å\n198 = Æ\n199 = Ç\n200 = È\n201 = É\n202 = Ê\n203 = Ë\n204 = Ì\n205 = Í\n206 = Î\n207 = Ï\n208 = Ð\n209 = Ñ\n210 = Ò\n211 = Ó\n212 = Ô\n213 = Õ\n214 = Ö\n215 = ×\n216 = Ø\n217 = Ù\n218 = Ú\n219 = Û\n220 = Ü\n221 = Ý\n222 = Þ\n223 = ß\n224 = à\n225 = á\n226 = â\n227 = ã\n228 = ä\n229 = å\n230 = æ\n231 = ç\n232 = è\n233 = é\n234 = ê\n235 = ë\n236 = ì\n237 = í\n238 = î\n239 = ï\n240 = ð\n241 = ñ\n242 = ò\n243 = ó\n244 = ô\n245 = õ\n246 = ö\n247 = ÷\n248 = ø\n249 = ù\n250 = ú\n251 = û\n252 = ü\n253 = ý\n254 = þ\n255 = ÿ\ndone\n"
julia>
Since I can’t reproduce, and I’m not sure what is going on, I’m not sure what else to say. Maybe someone else has some advice.
(On a separate note, printing to stdout and then reading it back in Julia is a pretty inefficient and awkward way to get data from Python to Julia. Much better to just call a Python function and get an array back directly.)
Ok thanks for trying. I wanted to examine the take! approach as the actual feed from the python script will be feeding the buffer asynchronously with a new buffer every second.
I wanted to feed the Julia program by just redirecting the stdout and consuming the data flow from the Python script. In that case I don’t see what object I could pass over?