# PyCall stdout buffer processing not doing what I expected

**URL:** <https://discourse.julialang.org/t/pycall-stdout-buffer-processing-not-doing-what-i-expected/75086>\
**Category:** Performance\
**Tags:** pycall\
**Created:** [January 23, 2022, 8:30pm UTC](https://discourse.julialang.org/t/pycall-stdout-buffer-processing-not-doing-what-i-expected/75086 "2022-01-23T20:30:42Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![anon69491625](https://avatars.discourse-cdn.com/v4/letter/a/aeb1de/32.png) [@anon69491625](https://discourse.julialang.org/u/anon69491625)\
**Post date:** [January 23, 2022, 8:30pm UTC](https://discourse.julialang.org/t/pycall-stdout-buffer-processing-not-doing-what-i-expected/75086/1 "2022-01-23T20:30:42Z")

</div>

Hi all

I am using PyCall to run a simple python script that prints out the ascii table.

```julia
#test_stdout_pycall.py

digit = 0

while digit<=255:
    print("%d = %c" %(digit, chr(digit)))
    digit += 1

print("done")

```

I want to read the output in a buffer in Julia

```julia
using PyCall

buf = IOBuffer()

pyimport("sys").stdout = buf

@pyinclude("test_stdout_pycall.py")

while buf != "done/n"
      print( String(take!(buf)) )
 end

```

BUT the Julia script only gets me the first 157 iterations ( I only show the 124:157) then just hangs on the "157 = " line. If I interrupt the script and print out buf it’s empty.

```julia
124 = |
125 = }
126 = ~
127 = 
128 = 
129 = 
130 = 
131 = 
132 = 

133 = 

134 = 
135 = 
136 = 
137 = 	
138 = 
139 = 
140 = 
142 = 
143 = 
144 = 
146 = 
147 = 
148 = 
149 = 
150 = 
151 = 
152 = 
154 = 
      155 = 

157 = 

```

My expected behavior based on my read of the Julia documentation regarding take! ( see below) is that it would consume buf until some condition was met. In my case the bytes “done” which is sent from the python script after sending 256 lines comprising of the count and associated chr code. So what’s happening here. Here’s the doc on take!

```julia
Obtain the contents of an IOBuffer as an array. Afterwards, the IOBuffer is reset to its initial state.

```

I got the above from [the official docs on take!](https://docs.julialang.org/en/v1/base/io-network/#Base.take!-Tuple%7BBase.GenericIOBuffer%7D)

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [January 23, 2022, 8:43pm UTC](https://discourse.julialang.org/t/pycall-stdout-buffer-processing-not-doing-what-i-expected/75086/2 "2022-01-23T20:43:08Z")

</div>

Maybe call `sys.stdout.flush()` at the end of your Python script to make sure Python isn’t buffering some output internally?

I’m also not sure why you have a `while` loop in Julia — the Python code isn’t executing asynchronously, it should be completely done by the time it leaves `@pyinclude`, so you should only `take!` the buffer once. (This will also cause your loop to not terminate — the buffer contains _all_ the lines, not a single line, so it will never equal `"done\n"`.)

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [January 23, 2022, 8:47pm UTC](https://discourse.julialang.org/t/pycall-stdout-buffer-processing-not-doing-what-i-expected/75086/3 "2022-01-23T20:47:49Z")

</div>

On my machine, no flushing is required:

```julia
using PyCall
buf = IOBuffer()
pyimport("sys").stdout = buf

py"""
digit = 0

while digit<=255:
    print("%d = %c" %(digit, chr(digit)))
    digit += 1

print("done")
"""

print( String(take!(buf)) )

```

prints all of the lines including the `done`.

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [January 23, 2022, 8:50pm UTC](https://discourse.julialang.org/t/pycall-stdout-buffer-processing-not-doing-what-i-expected/75086/4 "2022-01-23T20:50:04Z")

</div>

> [@stevengj](#):
>
> Python code isn’t executing asynchronously

Be sure that your mental model is correct here. Julia is _not_ running a separate `python` script and reading the output. PyCall is calling directly into `libpython` and executing the Python code _within_ the Julia process, so each call to Python completes before returning.

---

<div class="post-metadata">

**Author:** ![anon69491625](https://avatars.discourse-cdn.com/v4/letter/a/aeb1de/32.png) [@anon69491625](https://discourse.julialang.org/u/anon69491625)\
**Post date:** [January 23, 2022, 8:53pm UTC](https://discourse.julialang.org/t/pycall-stdout-buffer-processing-not-doing-what-i-expected/75086/5 "2022-01-23T20:53:32Z")

</div>

I put the while in to see if there was some kind of default buffer length in the hope that take would flush the buf and get the next given the 157 line issue.

ALSO I ran your code ( thanks for that) and it did EXACTLY the same only printed 155 ( down from 157) and this time didn’t hang.

```julia
julia
               _
   _ _ _(_)_ | Documentation: https://docs.julialang.org
  (_) | (_) (_) |
   _ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` | |
  | | |_| | | | (_| | | Version 1.7.0 (2021-11-30)
 _/ |\ __'_|_|_|\__'_| | Official https://julialang.org/ release
|__/ |

julia> using PyCall

julia> buf = IOBuffer()
IOBuffer(data=UInt8[...], readable=true, writable=true, seekable=true, append=false, size=0, maxsize=Inf, ptr=1, mark=-1)

julia> pyimport("sys").stdout = buf
IOBuffer(data=UInt8[...], readable=true, writable=true, seekable=true, append=false, size=0, maxsize=Inf, ptr=1, mark=-1)

julia> py"""
       digit = 0

       while digit<=255:
           print("%d = %c" %(digit, chr(digit)))
           digit += 1

       print("done")
       """

julia> print( String(take!(buf)) )
0 = 
1 = 
2 = 
3 = 
4 = 
5 = 
6 = 
7 = 
8 = 
9 = 	
10 = 

11 = 

12 = 

13 = 
14 = 
15 = 
16 = 
17 = 
18 = 
19 = 
20 = 
21 = 
22 = 
23 = 
24 = 
25 = 
26 = �
27 = 
8 = 
29 = 
30 = 
31 = 
32 =  
33 = !
34 = "
35 = #
36 = $
37 = %
38 = &
39 = '
40 = (
41 = )
42 = *
43 = +
44 = ,
45 = -
46 = .
47 = /
48 = 0
49 = 1
50 = 2
51 = 3
52 = 4
53 = 5
54 = 6
55 = 7
56 = 8
57 = 9
58 = :
59 = ;
60 = <
61 = =
62 = >
63 = ?
64 = @
65 = A
66 = B
67 = C
68 = D
69 = E
70 = F
71 = G
72 = H
73 = I
74 = J
75 = K
76 = L
77 = M
78 = N
79 = O
80 = P
81 = Q
82 = R
83 = S
84 = T
85 = U
86 = V
87 = W
88 = X
89 = Y
90 = Z
91 = [
92 = \
93 = ]
94 = ^
95 = _
96 = `
97 = a
98 = b
99 = c
100 = d
101 = e
102 = f
103 = g
104 = h
105 = i
106 = j
107 = k
108 = l
109 = m
110 = n
111 = o
112 = p
113 = q
114 = r
115 = s
116 = t
117 = u
118 = v
119 = w
120 = x
121 = y
122 = z
123 = {
124 = |
125 = }
126 = ~
127 = 
128 = 
129 = 
130 = 
131 = 
132 = 

133 = 

134 = 
135 = 
136 = 
137 = 	
138 = 
139 = 
140 = 
142 = 
143 = 
144 = 
146 = 
147 = 
148 = 
149 = 
150 = 
151 = 
152 = 
154 = 
      155 = 

julia> 

```

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [January 23, 2022, 8:54pm UTC](https://discourse.julialang.org/t/pycall-stdout-buffer-processing-not-doing-what-i-expected/75086/6 "2022-01-23T20:54:36Z")

</div>

> [@anon69491625](#):
>
> ALSO I ran your code ( thanks for that) and it did EXACTLY the same only printed 155 ( down from 157) and this time didn’t hang.

Try adding

```julia
import sys
sys.stdout.flush()

```

at the end of your Python code. (Or, equivalently, run `pyimport("sys")."stdout".flush()` in Julia before the `take!`.)

---

<div class="post-metadata">

**Author:** ![anon69491625](https://avatars.discourse-cdn.com/v4/letter/a/aeb1de/32.png) [@anon69491625](https://discourse.julialang.org/u/anon69491625)\
**Post date:** [January 23, 2022, 8:56pm UTC](https://discourse.julialang.org/t/pycall-stdout-buffer-processing-not-doing-what-i-expected/75086/7 "2022-01-23T20:56:32Z")

</div>

I was concerned that I wasn’t processing all of the buffer in one take, so that’s why I put the while loop in. I wanted to see if another take would “find” the rest of the buffer. When I read the docs I did see that the print would be completely sent in one buffer hence my bewilderment when it didn’t arrive. That’s why I put the while loop in.

---

<div class="post-metadata">

**Author:** ![anon69491625](https://avatars.discourse-cdn.com/v4/letter/a/aeb1de/32.png) [@anon69491625](https://discourse.julialang.org/u/anon69491625)\
**Post date:** [January 23, 2022, 9:05pm UTC](https://discourse.julialang.org/t/pycall-stdout-buffer-processing-not-doing-what-i-expected/75086/8 "2022-01-23T21:05:03Z")

</div>

Hi there

still doing the same thing. I removed the while loop and, as I want to concentrate on Julia, put in the buffer flush before the take!

I’m running Linux Mint 20.3 if that’s relevant and python 3.8

```julia
julia
               _
   _ _ _(_)_ | Documentation: https://docs.julialang.org
  (_) | (_) (_) |
   _ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` | |
  | | |_| | | | (_| | | Version 1.7.0 (2021-11-30)
 _/ |\ __'_|_|_|\__'_| | Official https://julialang.org/ release
|__/ |

julia> using PyCall

julia> buf = IOBuffer()
IOBuffer(data=UInt8[...], readable=true, writable=true, seekable=true, append=false, size=0, maxsize=Inf, ptr=1, mark=-1)

julia> pyimport("sys").stdout = buf
IOBuffer(data=UInt8[...], readable=true, writable=true, seekable=true, append=false, size=0, maxsize=Inf, ptr=1, mark=-1)

julia> @pyinclude("test_stdout_pycall.py")

julia> pyimport("sys")."stdout".flush()

julia> print( String(take!(buf)) )
0 = 
1 = 
2 = 
3 = 
4 = 
5 = 
6 = 
7 = 
8 = 
9 = 	
10 = 

11 = 

12 = 

13 = 
14 = 
15 = 
16 = 
17 = 
18 = 
19 = 
20 = 
21 = 
22 = 
23 = 
24 = 
25 = 
26 = �
27 = 
8 = 
29 = 
30 = 
31 = 
32 =  
33 = !
34 = "
35 = #
36 = $
37 = %
38 = &
39 = '
40 = (
41 = )
42 = *
43 = +
44 = ,
45 = -
46 = .
47 = /
48 = 0
49 = 1
50 = 2
51 = 3
52 = 4
53 = 5
54 = 6
55 = 7
56 = 8
57 = 9
58 = :
59 = ;
60 = <
61 = =
62 = >
63 = ?
64 = @
65 = A
66 = B
67 = C
68 = D
69 = E
70 = F
71 = G
72 = H
73 = I
74 = J
75 = K
76 = L
77 = M
78 = N
79 = O
80 = P
81 = Q
82 = R
83 = S
84 = T
85 = U
86 = V
87 = W
88 = X
89 = Y
90 = Z
91 = [
92 = \
93 = ]
94 = ^
95 = _
96 = `
97 = a
98 = b
99 = c
100 = d
101 = e
102 = f
103 = g
104 = h
105 = i
106 = j
107 = k
108 = l
109 = m
110 = n
111 = o
112 = p
113 = q
114 = r
115 = s
116 = t
117 = u
118 = v
119 = w
120 = x
121 = y
122 = z
123 = {
124 = |
125 = }
126 = ~
127 = 
128 = 
129 = 
130 = 
131 = 
132 = 

133 = 

134 = 
135 = 
136 = 
137 = 	
138 = 
139 = 
140 = 
142 = 
143 = 
144 = 
146 = 
147 = 
148 = 
149 = 
150 = 
151 = 
152 = 
154 = 
      155 = 

julia> 

```

---

<div class="post-metadata">

**Author:** ![anon69491625](https://avatars.discourse-cdn.com/v4/letter/a/aeb1de/32.png) [@anon69491625](https://discourse.julialang.org/u/anon69491625)\
**Post date:** [January 23, 2022, 9:16pm UTC](https://discourse.julialang.org/t/pycall-stdout-buffer-processing-not-doing-what-i-expected/75086/9 "2022-01-23T21:16:10Z")

</div>

It’s something to do with the print statement.

```julia
print( String(take!(buf)) )

```

This is what buf actually looks like if don’t print it out and just use

```julia
String(take!(buf))

```

the python code

```julia
digit = 0

while digit<=255:
    print("%d = %c" %(digit, chr(digit)))
    digit += 1

print("done")

```

the julia code

```julia
$julia
               _
   _ _ _(_)_ | Documentation: https://docs.julialang.org
  (_) | (_) (_) |
   _ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` | |
  | | |_| | | | (_| | | Version 1.7.0 (2021-11-30)
 _/ |\ __'_|_|_|\__'_| | Official https://julialang.org/ release
|__/ |

julia> using PyCall

julia> buf = IOBuffer()
IOBuffer(data=UInt8[...], readable=true, writable=true, seekable=true, append=false, size=0, maxsize=Inf, ptr=1, mark=-1)

julia> pyimport("sys").stdout = buf
IOBuffer(data=UInt8[...], readable=true, writable=true, seekable=true, append=false, size=0, maxsize=Inf, ptr=1, mark=-1)

julia> @pyinclude("test_stdout_pycall.py")

julia> pyimport("sys")."stdout".flush()

julia> String(take!(buf))
"0 = \0\n1 = \x01\n2 = \x02\n3 = \x03\n4 = \x04\n5 = \x05\n6 = \x06\n7 = \a\n8 = \b\n9 = \t\n10 = \n\n11 = \v\n12 = \f\n13 = \r\n14 = \x0e\n15 = \x0f\n16 = \x10\n17 = \x11\n18 = \x12\n19 = \x13\n20 = \x14\n21 = \x15\n22 = \x16\n23 = \x17\n24 = \x18\n25 = \x19\n26 = \x1a\n27 = \e\n28 = \x1c\n29 = \x1d\n30 = \x1e\n31 = \x1f\n32 = \n33 = !\n34 = \"\n35 = #\n36 = \$\n37 = %\n38 = &\n39 = '\n40 = (\n41 = )\n42 = *\n43 = +\n44 = ,\n45 = -\n46 = .\n47 = /\n48 = 0\n49 = 1\n50 = 2\n51 = 3\n52 = 4\n53 = 5\n54 = 6\n55 = 7\n56 = 8\n57 = 9\n58 = :\n59 = ;\n60 = <\n61 = =\n62 = >\n63 = ?\n64 = @\n65 = A\n66 = B\n67 = C\n68 = D\n69 = E\n70 = F\n71 = G\n72 = H\n73 = I\n74 = J\n75 = K\n76 = L\n77 = M\n78 = N\n79 = O\n80 = P\n81 = Q\n82 = R\n83 = S\n84 = T\n85 = U\n86 = V\n87 = W\n88 = X\n8" ⋯ 768 bytes ⋯ "180 = ´\n181 = µ\n182 = ¶\n183 = ·\n184 = ¸\n185 = ¹\n186 = º\n187 = »\n188 = ¼\n189 = ½\n190 = ¾\n191 = ¿\n192 = À\n193 = Á\n194 = Â\n195 = Ã\n196 = Ä\n197 = Å\n198 = Æ\n199 = Ç\n200 = È\n201 = É\n202 = Ê\n203 = Ë\n204 = Ì\n205 = Í\n206 = Î\n207 = Ï\n208 = Ð\n209 = Ñ\n210 = Ò\n211 = Ó\n212 = Ô\n213 = Õ\n214 = Ö\n215 = ×\n216 = Ø\n217 = Ù\n218 = Ú\n219 = Û\n220 = Ü\n221 = Ý\n222 = Þ\n223 = ß\n224 = à\n225 = á\n226 = â\n227 = ã\n228 = ä\n229 = å\n230 = æ\n231 = ç\n232 = è\n233 = é\n234 = ê\n235 = ë\n236 = ì\n237 = í\n238 = î\n239 = ï\n240 = ð\n241 = ñ\n242 = ò\n243 = ó\n244 = ô\n245 = õ\n246 = ö\n247 = ÷\n248 = ø\n249 = ù\n250 = ú\n251 = û\n252 = ü\n253 = ý\n254 = þ\n255 = ÿ\ndone\n"

julia> 

```

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [January 23, 2022, 9:31pm UTC](https://discourse.julialang.org/t/pycall-stdout-buffer-processing-not-doing-what-i-expected/75086/10 "2022-01-23T21:31:03Z")

</div>

> [@anon69491625](#):
>
> still doing the same thing. I removed the while loop and, as I want to concentrate on Julia, put in the buffer flush before the take!

Since I can’t reproduce, and I’m not sure what is going on, I’m not sure what else to say. Maybe someone else has some advice.

(On a separate note, printing to `stdout` and then reading it back in Julia is a pretty inefficient and awkward way to get data from Python to Julia. Much better to just call a Python function and get an array back directly.)

---

<div class="post-metadata">

**Author:** ![anon69491625](https://avatars.discourse-cdn.com/v4/letter/a/aeb1de/32.png) [@anon69491625](https://discourse.julialang.org/u/anon69491625)\
**Post date:** [January 23, 2022, 9:37pm UTC](https://discourse.julialang.org/t/pycall-stdout-buffer-processing-not-doing-what-i-expected/75086/11 "2022-01-23T21:37:57Z")

</div>

Ok thanks for trying. I wanted to examine the take! approach as the actual feed from the python script will be feeding the buffer asynchronously with a new buffer every second.

I wanted to feed the Julia program by just redirecting the stdout and consuming the data flow from the Python script. In that case I don’t see what object I could pass over?