# Makie recipe with plots being added on the fly and why does the recipe run so long

**URL:** <https://discourse.julialang.org/t/makie-recipe-with-plots-being-added-on-the-fly-and-why-does-the-recipe-run-so-long/80539>\
**Category:** Performance\
**Tags:** distributed, makie\
**Created:** [May 5, 2022, 12:54pm UTC](https://discourse.julialang.org/t/makie-recipe-with-plots-being-added-on-the-fly-and-why-does-the-recipe-run-so-long/80539 "2022-05-05T12:54:34Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![kunzaatko](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kunzaatko/32/25153_2.png) [@kunzaatko](https://discourse.julialang.org/u/kunzaatko)\
**Post date:** [May 5, 2022, 12:54pm UTC](https://discourse.julialang.org/t/makie-recipe-with-plots-being-added-on-the-fly-and-why-does-the-recipe-run-so-long/80539/1 "2022-05-05T12:54:34Z")

</div>

I am trying to make a `Makie` recipe for a sequence of images with the current image frame from the sequence being an `Observable` (`frame`). I would like to have the separate frames of the image load on demand, when the `frame` changes.

# Attempt 1

My first attempt was to create the desired image plot when the frame changes. Here it is:

```julia
using DataStructures, Images
using Makie
import Makie: convert_arguments
import MakieCore: argument_names

@recipe(ImageSequence) do scene
    Attributes(
        image_attributes = (; interpolate = false, inspectable = false)
    )
end

argument_names(::Type{<:ImageSequence}, numargs::Integer) = numargs == 2 && (:frame, :sequence)

function Makie.plot!(
    im_sequence::ImageSequence{<:Tuple{Integer,AbstractVector{<:AbstractMatrix{<:Colorant}}}})

imageframes = fill(nothing, length(im_sequence[:sequence][])) |> Vector{Union{Nothing, AbstractPlot}}

    visitedframes = CircularBuffer(3)
    on(im_sequence[:frame]) do frame
        foreach(f -> begin # make the previous plots invisible
                # NOTE: It is possible to notify frame without changing it
                if f != frame
                    imageframes[f].visible[] = false
                end
            end, visitedframes)
    if isnothing(imageframes[frame])
            imageframes[frame] = image!(im_sequence, im_sequence[:sequence][][frame]; im_sequence.image_attributes..., visible = true)
        end
        push!(visitedframes, frame)
    end
    notify(im_sequence[:frame])

    return im_sequence
end

```

And then I call it

```julia-auto
l = imagesequence(1, img_vec)

```

Which shows the first frame correctly,

 ![imagesequence_with_no_cache_first](https://global.discourse-cdn.com/julialang/original/3X/a/3/a38a6df1e61c2cfd74704d3973f979ad76399fc2.png)

but on an attempt to change the frame,

```repl
l.plot[:frame][] = 2

```

an empty axis appears.

 ![imagesequence_with_no_cache](https://global.discourse-cdn.com/julialang/original/3X/7/6/76c20cba676f3525e752cf6538eaa5b29536e1e0.png)

But the plot of image frame `2` is included…

```julia-auto
julia> l.plot.plots
> 2-element Vector{AbstractPlot}:
 Image{Tuple{StepRangeLen{Float32, Float64, Float64, Int64}, StepRangeLen{Float32, Float64, Float64, Int64}, Matrix{RGB
A{Float32}}}}
 Image{Tuple{StepRangeLen{Float32, Float64, Float64, Int64}, StepRangeLen{Float32, Float64, Float64, Int64}, Matrix{RGB
A{Float32}}}}

```

And `l.plot.plots[2].visible[] == true` so there is no issue on that front.

I thought that to associate the plot with the given recipe it has to be created in the `Makie.plot!` function, so I tried to create them using `@async`

# Attempt 2

```julia-auto
using Distributed
using DataStructures, Images
using Makie
import Makie: convert_arguments
import MakieCore: argument_names

@recipe(ImageSequence) do scene
    Attributes(
        image_attributes = (; interpolate = false, inspectable = false)
    )
end

argument_names(::Type{<:ImageSequence}, numargs::Integer) = numargs == 2 && (:frame, :sequence)

function Makie.plot!(
    im_sequence::ImageSequence{<:Tuple{Integer,AbstractVector{<:AbstractMatrix{<:Colorant}}}})

    imageframes = fill(nothing, length(im_sequence[:sequence][])) |> Vector{Union{Nothing,AbstractPlot}}
    imageframetasks = Vector(undef, length(im_sequence[:sequence][]))
    for i in 1:length(imageframes)
        imageframetasks[i] = @async imageframes[i] = image!(im_sequence, im_sequence[:sequence][][i]; im_sequence.image_attributes..., visible = false)
    end

    visitedframes = CircularBuffer(3)
    on(im_sequence[:frame]) do frame
        wait(imageframetasks[frame])
        foreach(f -> begin # make the previous plots invisible
                # NOTE: It is possible to notify frame without changing it
                if f != frame
                    imageframes[f].visible[] = false
                end
            end, visitedframes)
        imageframes[frame].visible[] = true
        push!(visitedframes, frame)
    end
    notify(im_sequence[:frame])

    return im_sequence
end

```

which works correctly when using the frame, but it takes the same time as my original synchronous version for some reason.

```julia-auto
julia> @time l = ANT.Recipes.imagesequence(1, img_vec)
 44.776909 seconds (1.51 G allocations: 86.338 GiB, 46.70% gc time, 1.48% compilation time)

```

## Synchronous version

```julia-auto
using DataStructures, Images
using Makie
import Makie: convert_arguments
import MakieCore: argument_names

@recipe(ImageSequence) do scene
    Attributes(
        # Attributes to be used for the image plot. Including visible would break the mechanism for
        # showing the correct frame, so do not include `visible = true`.
        image_attributes = (; interpolate = false, inspectable = false)
    )
end

argument_names(::Type{<:ImageSequence}, numargs::Integer) = numargs == 2 && (:frame, :sequence)

function Makie.plot!(
    im_sequence::ImageSequence{<:Tuple{Integer,AbstractVector{<:AbstractMatrix{<:Colorant}}}})

    imageframes = Vector(undef, length(im_sequence[:sequence][]))
    for i in 1:length(imageframes)
         imageframes[i] = image!(im_sequence, im_sequence[:sequence][][i]; im_sequence.image_attributes..., visible = false)
    end

    visitedframes = CircularBuffer(3)
    on(im_sequence[:frame]) do frame
        foreach(f -> begin # make the previous plots invisible
                # NOTE: It is possible to notify frame without changing it
                if f != frame
                    imageframes[f].visible[] = false
                end
            end, visitedframes)
        imageframes[frame].visible[] = true
        push!(visitedframes, frame)
    end
    notify(im_sequence[:frame])

    return im_sequence
end

```

Which works as expected, but takes a long time (same as _Attempt 2_)

```julia-auto
julia> @time l = imagesequence(1, img_vec)
 41.936290 seconds (1.51 G allocations: 86.258 GiB, 46.83% gc time, 0.09% compilation time)

```

* * *

Also, do you have an idea, why there are so much allocations? The original image sequence size is

```julia-auto
.rw-r--r-- 252M kunzaatko 21 Mar 2021 TIRF488_cam1_0_L.tiff

```

and when only showing one frame from the total of 60 the allocation size is much smaller:

```julia-auto
julia> @time image(img[:,:,1])
  0.026145 seconds (45.76 k allocations: 38.600 MiB)

```

(And also it is a mystery to me as to why the loading of the images takes so long, when

```julia-auto
julia> 60 * 0.026145
> 1.5687

```

so why doesn’t it take only one this time to load them all?)

---

<div class="post-metadata">

**Author:** ![jules](https://avatars.discourse-cdn.com/v4/letter/j/41988e/32.png) [@jules](https://discourse.julialang.org/u/jules)\
**Post date:** [May 5, 2022, 5:01pm UTC](https://discourse.julialang.org/t/makie-recipe-with-plots-being-added-on-the-fly-and-why-does-the-recipe-run-so-long/80539/2 "2022-05-05T17:01:44Z")

</div>

> [@kunzaatko](#):
>
> recipe for a sequence of images with the current image frame from the sequence being an `Observable` ( `frame` ). I would like to have the separate frames of the image load on demand, when the `frame` changes.

I’m just going to go off of this, as there’s a lot of code there and maybe a much simpler solution will suffice.

A normal `image` plot can take an Observable that you can update with new images whenever you want. You don’t need to create a recipe for that. Where that data comes from doesn’t really matter, you can of course use something like a frame number to take it from somewhere else. Like this:

```julia
images = [rand(50,50) for _ in 1:30] # thirty images stored in a vector, could be any other format as well
current_frame = Observable(1)
current_frame_data = @lift(images[$current_frame])

fig, ax, im = image(current_frame_data)
display(fig)

# now you can change the image data by selecting a different frame with the observable
current_frame[] = 2

```

---

<div class="post-metadata">

**Author:** ![kunzaatko](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kunzaatko/32/25153_2.png) [@kunzaatko](https://discourse.julialang.org/u/kunzaatko)\
**Post date:** [May 5, 2022, 8:44pm UTC](https://discourse.julialang.org/t/makie-recipe-with-plots-being-added-on-the-fly-and-why-does-the-recipe-run-so-long/80539/3 "2022-05-05T20:44:17Z")

</div>

This works great! Why didn’t I think of that… Thank you!

---

<div class="post-metadata">

**Author:** ![stillyslalom](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stillyslalom/32/45687_2.png) [@stillyslalom](https://discourse.julialang.org/u/stillyslalom)\
**Post date:** [May 5, 2022, 9:17pm UTC](https://discourse.julialang.org/t/makie-recipe-with-plots-being-added-on-the-fly-and-why-does-the-recipe-run-so-long/80539/4 "2022-05-05T21:17:10Z")

</div>

For image sequence display, I like to use something like this:

```julia
images = [rand(Float32, 512, 512) for _ = 1:50]
f = Figure(resolution = (700, 900))

framerange = eachindex(images)
idxpad = maximum(length ∘ string, framerange)
ls = labelslider!(f, "Index", framerange, format = s -> lpad(s, idxpad))
ls_idx = ls.slider.value
f.layout[2, 1] = ls.layout

frame = @lift(images[$ls_idx])
ax = GLMakie.Axis(f[1, :], aspect=DataAspect())
image!(ax, frame)

on(events(f).keyboardbutton) do event # keyboard navigation
    if event.action in (Keyboard.press, Keyboard.repeat)
        event.key == Keyboard.left && (ls.slider.selected_index[] = clamp(ls.slider.selected_index[] - 1, eachindex(ls.slider.range[])))
        event.key == Keyboard.right && (ls.slider.selected_index[] = clamp(ls.slider.selected_index[] + 1, eachindex(ls.slider.range[])))
    end
end
```