In the following MWE (requires a connected webcam to work), I experience a substantial lag and a surprisingly low frame-rate:
using VideoIO, WGLMakie, JSServe, Observables
cam = VideoIO.opencamera() # open the webcam
img = read(cam)
o = Observable(img)
p = @async while true # update img with a new frame from the camera
read!(cam, img)
o[] = img
sleep(0.01)
end
App() do session::Session # get the app going
fig, ax, cplot = image(o)
fig
end
So nothing special is happening here (I think), but I get a lag of about ~10 seconds and fps of ~5 frames per second.
Can you try this with really small resolution, just to make sure that the data transfer to javascript is not the bottleneck? Which is what it sounds like to me if you say GLMakie works well.
Hmm next thing I would check is to use the same async logic, but update a small line plot or something like that. Something which should really give you the lowest delay and highest frame rate. I don’t know if the way you programmed this with @async and sleep(0.01) is the best way to implement this, although I don’t have a better alternative for you right now. Maybe Simon knows more
Line-plot alone plots very rapidly.
Line-plot with the frame plots similarly to how the frame plots alone.
I’m just surprised by the performance, the smaller version of the frame is just 216 x 384 RGB pixels large… So not that big, and still, the lag and fps are noticeably “there”.
Here is an even more compact MWE (doesn’t require anything). Seems like (using my eyes only) we get lag and dropped frames with images that are larger than 200 x 200 pixels at 30 fps. If that is “normal”, then great. If I’m doing something wrong, then I’d love to fix it:
using WGLMakie, JSServe
Base.throwto(p, InterruptException()) # stops the previous async loop (will error if this is the first run, but that's fine)
n = 200 # frame dimensions
fps = 30 # frames per second
img = Node(rand(UInt8, n, n))
p = @async while true
img[] = rand(UInt8, n, n)
sleep(1/fps)
end
App() do session::Session
image(img)
end
One thing that is definitely suboptimal is sleeping for 1/fps, because that only gives you a framerate of fps if you do exactly nothing in between. In any other case, you drop frames. The simplest way to avoid this is to use a non-drifting update loop, where you sleep only the difference between the elapsed time and the necessary inter-frame interval.
I just tested it a bit as well, and on my macbook the 200x200 px heatmap took around 30ms to update, so that would fit your observation. That is not great, but if you then also sleep 33ms you halve your framerate unnecessarily
You are of course right. That sleep call was just for illustration, I don’t plan to include that in my final code.
I timed updating the frame with:
using WGLMakie, JSServe
n = 200 # frame dimensions
img = Node(rand(UInt8, n, n))
function f()
img[] = rand(UInt8, n, n)
end
App() do session::Session
image(img)
end
using BenchmarkTools
@btime f();
and got:
# n = 100 -> 4.893 ms (10690 allocations: 586.30 KiB)
# n = 200 -> 20.830 ms (40695 allocations: 2.07 MiB)
# n = 400 -> 87.734 ms (160697 allocations: 8.10 MiB)
# n = 800 -> 353.295 ms (640699 allocations: 28.87 MiB)
I also found out that the timings do not improve even if I use a function like this one:
function f(img, n)
for i in eachindex(img[])
img[][i] = rand(UInt8)
end
img[] = img[]
end
Which reduces allocations. So most of the time is spent on data transfer? @sdanisch is this performance (about 200 x 200 pixels at 30 fps) right? Or am I missing something?