Channel gets emptied and stays that way

General Usage
,
1

I want to use a @async task to put! video frames (from VideoIO) in a Channel and use that channel as a buffer of frames to update a Makie.Scene with. The idea is that while the user plays with the slider to explore a video the buffer will have all the frames the user needs and as the user uses them the task will put! more frames in the buffer in the background.

My problem is that the channel always gets empty, even if I leave the slider alone. It’s like some background task never yields to allow that @asynced task to fill back the buffer.

Any idea what’s going on?

Here is a bare-bone MWE (ignore the scaling and rotation of the image, just trying to keep it super simple):

using Makie, VideoIO
avf = VideoIO.testvideo("annie_oakley")
f = VideoIO.openvideo(avf)
seek(f, 5.0) # skip the beginning
img = Node(read(f))
buff = Channel{typeof(img[])}(500) # have a buffer with 500 frames
task = @async begin # this is the task that fills that buffer
    while !eof(f)
        put!(buff, read(f))
    end
end
scene = Makie.image(img)
slider_h = slider(1:500, raw = true, camera = campixel!, start = 2)
old_slider_pos = Node(1)
lift(slider_h[end][:value]) do frame
    if old_slider_pos[] ≤ frame
        for i in old_slider_pos[]:frame
            img[] = take!(buff)
        end
        old_slider_pos[] = frame
    else
        println("can't go back in time! Yet…")
    end
end
hbox(slider_h, scene)

Now slide the slider a little to the right and then check the buff variable, it should show something like this:

julia> buff
Channel{PermutedDimsArray{ColorTypes.RGB{FixedPointNumbers.Normed{UInt8,8}},2,(2, 1),(2, 1),Array{ColorTypes.RGB{FixedPointNumbers.Normed{UInt8,8}},2}}}(sz_max:500,sz_curr:500)

Nice, the channel is still full, it has 500 frames ready.

OK, now be nasty and pull the slider to the right more vigorously. Check buff again:

julia> buff
Channel{PermutedDimsArray{ColorTypes.RGB{FixedPointNumbers.Normed{UInt8,8}},2,(2, 1),(2, 1),Array{ColorTypes.RGB{FixedPointNumbers.Normed{UInt8,8}},2}}}(sz_max:500,sz_curr:128)

Now it has lost tons of frames. It will stay like that no matter what.

Why? And how can I get it to do what I want: fill up the buffer in the background (like when the user isn’t playing with the slider)?

2

Could be a put! without a corresponding take!? You might have to empty!…

This is just what I got from reading the docstring, YMMV.

3

Yea, that’s what’s happening, but why? Why is the put! task halted? Why does it never catch up with all the take!s?

Empty what? The buffer? That’s exactly the opposite of what I’m looking for. The buffer is emptied no matter what, which is the problem.

4

It seems you consumed the frames. Isn’t eof(f) reached very soon? ~580 frames is the total

1 Like
5

Nope, the video isn’t close to its end. Try it, you’ll see!

6

It is closed as soon as you move to ~80 the slider. And then you have a problem at the end of the slider because of a 1-off error on the take loop, it should be:

        for i in old_slider_pos[]+1:frame
            img[] = take!(buff)
        end

so there where more take!s than put!s

1 Like
7

Pretty sure the problem is that the lift function might be called concurrently, and the slider value might be updated multiple times – that means that multiple tasks might try take!ing stuff out of the buffer at the same time (not really at the same time, but in an undefined order).

Your code works fine if you set the slider value programmatically with e.g. slider_h[end][:value] = 500, or adjust your code as follows:

lift(slider_h[end][:value]) do frame
    while old_slider_pos[] ≤ frame
        img[] = take!(buff)
        old_slider_pos[] += 1
    end
end

A better option would be to debounce the invocation of the function called by lift.

Edit: Or the above is completly wrong and Makie already debounces the slider output. @smldis solution works just fine :wink:

1 Like
8

Definitely the while solution has a better style and avoids the 1-off bug in the for loop.

1 Like
9

this worked! Awesome!

What…? How would I do that?

10

If the called function is expensive you could try to avoid to run it too often, this gets trickier soon and I think you would need to schedule it based on time, not only on the change of the slider value.
This is not your case, since the function is not that expensive, but you might want to insert a yield() inside the while loop in order to avoid to block on all the put!s needed to fill the channel.

Another issue I found is that the called function errors if you don’t return nothing. I am not familiar with the slider API so I cannot explain what’s happening.
The final code is (eventually put the while loop solution for better style):

using Makie, VideoIO
avf = VideoIO.testvideo("annie_oakley")
f = VideoIO.openvideo(avf)
seek(f, 5.0) # skip the beginning
img = Node(read(f))
buff = Channel{typeof(img[])}(500) # have a buffer with 500 frames
task = @async begin # this is the task that fills that buffer
    i=0
    while !eof(f)
        put!(buff, read(f))
        yield()
    end
end
scene = Makie.image(img)
slider_h = slider(1:500, raw = true, camera = campixel!, start = 2)
old_slider_pos = Node(1)
lift(slider_h[end][:value]) do frame
    @show frame
    if old_slider_pos[] < frame
        for i in old_slider_pos[]+1:frame
            img[] = take!(buff)
        end
        old_slider_pos[] = frame
    else
        if old_slider_pos[] > frame
            println("can't go back in time! Yet…")
        end
    end
    nothing
end
hbox(slider_h, scene)
2 Likes
11

So the yield in that async while loop is great (Simon recommended the very same thing on slack). Thanks!

Not sure about that spurious i in that async while loop, is that doing anything?

12

Sorry, delete it, I had a counter there for debugging

2 Likes