# Lag with WGLMakie

**URL:** <https://discourse.julialang.org/t/lag-with-wglmakie/55399>\
**Category:** Visualization\
**Tags:** makie\
**Created:** [February 16, 2021, 2:55pm UTC](https://discourse.julialang.org/t/lag-with-wglmakie/55399 "2021-02-16T14:55:25Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![yakir12](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yakir12/32/297_2.png) [@yakir12](https://discourse.julialang.org/u/yakir12)\
**Post date:** [February 16, 2021, 2:55pm UTC](https://discourse.julialang.org/t/lag-with-wglmakie/55399/1 "2021-02-16T14:55:25Z")

</div>

In the following MWE (requires a connected webcam to work), I experience a substantial lag and a surprisingly low frame-rate:

```julia
using VideoIO, WGLMakie, JSServe, Observables

cam = VideoIO.opencamera() # open the webcam
img = read(cam)
o = Observable(img)
p = @async while true # update img with a new frame from the camera
    read!(cam, img)
    o[] = img
    sleep(0.01)
end

App() do session::Session # get the app going
    fig, ax, cplot = image(o)
    fig
end

```

So nothing special is happening here (I think), but I get a lag of about ~10 seconds and fps of ~5 frames per second.

I’m on:

```julia
julia> versioninfo()
Julia Version 1.6.0-beta1
Commit b84990e1ac (2021-01-08 12:42 UTC)
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: AMD Ryzen Threadripper 2950X 16-Core Processor
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-11.0.0 (ORCJIT, znver1)
Environment:
  JULIA_NUM_THREADS = 16

```

with:

```julia
pkg> st
Status `~/ffmpegpipe/Project.toml`
  [3da002f7] ColorTypes v0.10.9
  [5789e2e9] FileIO v1.4.5
  [6218d12a] ImageMagick v1.1.6
  [824d6782] JSServe v1.2.0
  [510215fc] Observables v0.3.3
  [d6d074c3] VideoIO v0.8.4
  [276b4fcb] WGLMakie v0.3.2

```

The camera is a decent Logitech.

What am I doing wrong?

I should add that when I replace `WGLMakie` with `GLMakie`:

```julia
using VideoIO, GLMakie, JSServe, Observables

# ...
# same as above
# ...

fig, ax, cplot = image(o)
fig

```

the lag is gone (unnoticeable at least) and the frame-rate is maximal (for the camera).

---

<div class="post-metadata">

**Author:** ![jules](https://avatars.discourse-cdn.com/v4/letter/j/41988e/32.png) [@jules](https://discourse.julialang.org/u/jules)\
**Post date:** [February 16, 2021, 4:38pm UTC](https://discourse.julialang.org/t/lag-with-wglmakie/55399/2 "2021-02-16T16:38:12Z")

</div>

Can you try this with really small resolution, just to make sure that the data transfer to javascript is not the bottleneck? Which is what it sounds like to me if you say GLMakie works well.

---

<div class="post-metadata">

**Author:** ![yakir12](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yakir12/32/297_2.png) [@yakir12](https://discourse.julialang.org/u/yakir12)\
**Post date:** [February 16, 2021, 5:01pm UTC](https://discourse.julialang.org/t/lag-with-wglmakie/55399/3 "2021-02-16T17:01:15Z")

</div>

Interesting, I used `img[1:5:end, 1:5:end]` and the lag almost disappeared and the frame rate increased (but was still lower than 25 fps I think).

---

<div class="post-metadata">

**Author:** ![jules](https://avatars.discourse-cdn.com/v4/letter/j/41988e/32.png) [@jules](https://discourse.julialang.org/u/jules)\
**Post date:** [February 16, 2021, 5:23pm UTC](https://discourse.julialang.org/t/lag-with-wglmakie/55399/4 "2021-02-16T17:23:20Z")

</div>

Hmm next thing I would check is to use the same `async` logic, but update a small line plot or something like that. Something which should really give you the lowest delay and highest frame rate. I don’t know if the way you programmed this with `@async` and `sleep(0.01)` is the best way to implement this, although I don’t have a better alternative for you right now. Maybe Simon knows more

---

<div class="post-metadata">

**Author:** ![yakir12](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yakir12/32/297_2.png) [@yakir12](https://discourse.julialang.org/u/yakir12)\
**Post date:** [February 16, 2021, 6:25pm UTC](https://discourse.julialang.org/t/lag-with-wglmakie/55399/5 "2021-02-16T18:25:47Z")

</div>

Line-plot alone plots very rapidly.  
Line-plot with the frame plots similarly to how the frame plots alone.  
I’m just surprised by the performance, the smaller version of the frame is just `216 x 384` RGB pixels large… So not _that_ big, and still, the lag and fps are noticeably “there”.

---

<div class="post-metadata">

**Author:** ![yakir12](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yakir12/32/297_2.png) [@yakir12](https://discourse.julialang.org/u/yakir12)\
**Post date:** [February 17, 2021, 8:51am UTC](https://discourse.julialang.org/t/lag-with-wglmakie/55399/6 "2021-02-17T08:51:24Z")

</div>

Here is an even more compact MWE (doesn’t require anything). _Seems like_ (using my eyes only) we get lag and dropped frames with images that are larger than 200 x 200 pixels at 30 fps. If that is “normal”, then great. If I’m doing something wrong, then I’d love to fix it:

```julia
using WGLMakie, JSServe
Base.throwto(p, InterruptException()) # stops the previous async loop (will error if this is the first run, but that's fine)
n = 200 # frame dimensions
fps = 30 # frames per second
img = Node(rand(UInt8, n, n))
p = @async while true
    img[] = rand(UInt8, n, n)
    sleep(1/fps)
end
App() do session::Session
    image(img)
end

```

---

<div class="post-metadata">

**Author:** ![jules](https://avatars.discourse-cdn.com/v4/letter/j/41988e/32.png) [@jules](https://discourse.julialang.org/u/jules)\
**Post date:** [February 17, 2021, 9:24am UTC](https://discourse.julialang.org/t/lag-with-wglmakie/55399/7 "2021-02-17T09:24:54Z")

</div>

One thing that is definitely suboptimal is sleeping for 1/fps, because that only gives you a framerate of `fps` if you do exactly nothing in between. In any other case, you drop frames. The simplest way to avoid this is to use a non-drifting update loop, where you sleep only the difference between the elapsed time and the necessary inter-frame interval.

I just tested it a bit as well, and on my macbook the 200x200 px heatmap took around 30ms to update, so that would fit your observation. That is not great, but if you then also sleep 33ms you halve your framerate unnecessarily

---

<div class="post-metadata">

**Author:** ![yakir12](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yakir12/32/297_2.png) [@yakir12](https://discourse.julialang.org/u/yakir12)\
**Post date:** [February 17, 2021, 9:57am UTC](https://discourse.julialang.org/t/lag-with-wglmakie/55399/8 "2021-02-17T09:57:55Z")

</div>

You are of course right. That sleep call was just for illustration, I don’t plan to include that in my final code.

I timed updating the frame with:

```julia
using WGLMakie, JSServe
n = 200 # frame dimensions
img = Node(rand(UInt8, n, n))
function f() 
    img[] = rand(UInt8, n, n)
end
App() do session::Session
    image(img)
end

using BenchmarkTools
@btime f();

```

and got:

```julia
  # n = 100 -> 4.893 ms (10690 allocations: 586.30 KiB)
  # n = 200 -> 20.830 ms (40695 allocations: 2.07 MiB)
  # n = 400 -> 87.734 ms (160697 allocations: 8.10 MiB)
  # n = 800 -> 353.295 ms (640699 allocations: 28.87 MiB)

```

I also found out that the timings do not improve even if I use a function like this one:

```julia
function f(img, n) 
    for i in eachindex(img[])
        img[][i] = rand(UInt8)
    end
    img[] = img[]
end

```

Which reduces allocations. So most of the time is spent on data transfer? @sdanisch is this performance (about 200 x 200 pixels at 30 fps) right? Or am I missing something?

---

<div class="post-metadata">

**Author:** ![sdanisch](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sdanisch/32/1406_2.png) [@sdanisch](https://discourse.julialang.org/u/sdanisch)\
**Post date:** [February 17, 2021, 10:37am UTC](https://discourse.julialang.org/t/lag-with-wglmakie/55399/9 "2021-02-17T10:37:54Z")

</div>

I expect this to be pretty slow… Besides that, I wouldn’t be surprised if there’s a performance bug somewhere which doubles the work.

---

<div class="post-metadata">

**Author:** ![yakir12](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yakir12/32/297_2.png) [@yakir12](https://discourse.julialang.org/u/yakir12)\
**Post date:** [February 17, 2021, 10:51am UTC](https://discourse.julialang.org/t/lag-with-wglmakie/55399/10 "2021-02-17T10:51:20Z")

</div>

Well then! That’s good enough for me, I’ll work with that then.