Lag with WGLMakie

In the following MWE (requires a connected webcam to work), I experience a substantial lag and a surprisingly low frame-rate:

using VideoIO, WGLMakie, JSServe, Observables

cam = VideoIO.opencamera() # open the webcam
img = read(cam)
o = Observable(img)
p = @async while true # update img with a new frame from the camera
    read!(cam, img)
    o[] = img

App() do session::Session # get the app going
    fig, ax, cplot = image(o)

So nothing special is happening here (I think), but I get a lag of about ~10 seconds and fps of ~5 frames per second.

I’m on:

julia> versioninfo()
Julia Version 1.6.0-beta1
Commit b84990e1ac (2021-01-08 12:42 UTC)
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: AMD Ryzen Threadripper 2950X 16-Core Processor
  LIBM: libopenlibm
  LLVM: libLLVM-11.0.0 (ORCJIT, znver1)


pkg> st
Status `~/ffmpegpipe/Project.toml`
  [3da002f7] ColorTypes v0.10.9
  [5789e2e9] FileIO v1.4.5
  [6218d12a] ImageMagick v1.1.6
  [824d6782] JSServe v1.2.0
  [510215fc] Observables v0.3.3
  [d6d074c3] VideoIO v0.8.4
  [276b4fcb] WGLMakie v0.3.2

The camera is a decent Logitech.

What am I doing wrong?

I should add that when I replace WGLMakie with GLMakie:

using VideoIO, GLMakie, JSServe, Observables

# ...
# same as above
# ...

fig, ax, cplot = image(o)

the lag is gone (unnoticeable at least) and the frame-rate is maximal (for the camera).

Can you try this with really small resolution, just to make sure that the data transfer to javascript is not the bottleneck? Which is what it sounds like to me if you say GLMakie works well.

1 Like

Interesting, I used img[1:5:end, 1:5:end] and the lag almost disappeared and the frame rate increased (but was still lower than 25 fps I think).

Hmm next thing I would check is to use the same async logic, but update a small line plot or something like that. Something which should really give you the lowest delay and highest frame rate. I don’t know if the way you programmed this with @async and sleep(0.01) is the best way to implement this, although I don’t have a better alternative for you right now. Maybe Simon knows more

1 Like

Line-plot alone plots very rapidly.
Line-plot with the frame plots similarly to how the frame plots alone.
I’m just surprised by the performance, the smaller version of the frame is just 216 x 384 RGB pixels large… So not that big, and still, the lag and fps are noticeably “there”.

Here is an even more compact MWE (doesn’t require anything). Seems like (using my eyes only) we get lag and dropped frames with images that are larger than 200 x 200 pixels at 30 fps. If that is “normal”, then great. If I’m doing something wrong, then I’d love to fix it:

using WGLMakie, JSServe
Base.throwto(p, InterruptException()) # stops the previous async loop (will error if this is the first run, but that's fine)
n = 200 # frame dimensions
fps = 30 # frames per second
img = Node(rand(UInt8, n, n))
p = @async while true
    img[] = rand(UInt8, n, n)
App() do session::Session

One thing that is definitely suboptimal is sleeping for 1/fps, because that only gives you a framerate of fps if you do exactly nothing in between. In any other case, you drop frames. The simplest way to avoid this is to use a non-drifting update loop, where you sleep only the difference between the elapsed time and the necessary inter-frame interval.

I just tested it a bit as well, and on my macbook the 200x200 px heatmap took around 30ms to update, so that would fit your observation. That is not great, but if you then also sleep 33ms you halve your framerate unnecessarily

You are of course right. That sleep call was just for illustration, I don’t plan to include that in my final code.

I timed updating the frame with:

using WGLMakie, JSServe
n = 200 # frame dimensions
img = Node(rand(UInt8, n, n))
function f() 
    img[] = rand(UInt8, n, n)
App() do session::Session

using BenchmarkTools
@btime f();

and got:

  # n = 100 -> 4.893 ms (10690 allocations: 586.30 KiB)
  # n = 200 -> 20.830 ms (40695 allocations: 2.07 MiB)
  # n = 400 -> 87.734 ms (160697 allocations: 8.10 MiB)
  # n = 800 -> 353.295 ms (640699 allocations: 28.87 MiB)

I also found out that the timings do not improve even if I use a function like this one:

function f(img, n) 
    for i in eachindex(img[])
        img[][i] = rand(UInt8)
    img[] = img[]

Which reduces allocations. So most of the time is spent on data transfer? @sdanisch is this performance (about 200 x 200 pixels at 30 fps) right? Or am I missing something?

I expect this to be pretty slow… Besides that, I wouldn’t be surprised if there’s a performance bug somewhere which doubles the work.


Well then! That’s good enough for me, I’ll work with that then.