Makie recipe with plots being added on the fly and why does the recipe run so long

General Usage Performance
,
1

I am trying to make a Makie recipe for a sequence of images with the current image frame from the sequence being an Observable (frame). I would like to have the separate frames of the image load on demand, when the frame changes.

Attempt 1

My first attempt was to create the desired image plot when the frame changes. Here it is:

using DataStructures, Images
using Makie
import Makie: convert_arguments
import MakieCore: argument_names

@recipe(ImageSequence) do scene
    Attributes(
        image_attributes = (; interpolate = false, inspectable = false)
    )
end

argument_names(::Type{<:ImageSequence}, numargs::Integer) = numargs == 2 && (:frame, :sequence)

function Makie.plot!(
    im_sequence::ImageSequence{<:Tuple{Integer,AbstractVector{<:AbstractMatrix{<:Colorant}}}})

imageframes = fill(nothing, length(im_sequence[:sequence][])) |> Vector{Union{Nothing, AbstractPlot}}

    visitedframes = CircularBuffer(3)
    on(im_sequence[:frame]) do frame
        foreach(f -> begin # make the previous plots invisible
                # NOTE: It is possible to notify frame without changing it
                if f != frame
                    imageframes[f].visible[] = false
                end
            end, visitedframes)
    if isnothing(imageframes[frame])
            imageframes[frame] = image!(im_sequence, im_sequence[:sequence][][frame]; im_sequence.image_attributes..., visible = true)
        end
        push!(visitedframes, frame)
    end
    notify(im_sequence[:frame])

    return im_sequence
end

And then I call it

l = imagesequence(1, img_vec)

Which shows the first frame correctly,

imagesequence_with_no_cache_first
imagesequence_with_no_cache_first757×1031 282 KB

but on an attempt to change the frame,

l.plot[:frame][] = 2

an empty axis appears.

imagesequence_with_no_cache
imagesequence_with_no_cache757×1031 4.42 KB

But the plot of image frame 2 is included…

julia> l.plot.plots
> 2-element Vector{AbstractPlot}:
 Image{Tuple{StepRangeLen{Float32, Float64, Float64, Int64}, StepRangeLen{Float32, Float64, Float64, Int64}, Matrix{RGB
A{Float32}}}}
 Image{Tuple{StepRangeLen{Float32, Float64, Float64, Int64}, StepRangeLen{Float32, Float64, Float64, Int64}, Matrix{RGB
A{Float32}}}}

And l.plot.plots[2].visible[] == true so there is no issue on that front.

I thought that to associate the plot with the given recipe it has to be created in the Makie.plot! function, so I tried to create them using @async

Attempt 2

using Distributed
using DataStructures, Images
using Makie
import Makie: convert_arguments
import MakieCore: argument_names

@recipe(ImageSequence) do scene
    Attributes(
        image_attributes = (; interpolate = false, inspectable = false)
    )
end

argument_names(::Type{<:ImageSequence}, numargs::Integer) = numargs == 2 && (:frame, :sequence)

function Makie.plot!(
    im_sequence::ImageSequence{<:Tuple{Integer,AbstractVector{<:AbstractMatrix{<:Colorant}}}})

    imageframes = fill(nothing, length(im_sequence[:sequence][])) |> Vector{Union{Nothing,AbstractPlot}}
    imageframetasks = Vector(undef, length(im_sequence[:sequence][]))
    for i in 1:length(imageframes)
        imageframetasks[i] = @async imageframes[i] = image!(im_sequence, im_sequence[:sequence][][i]; im_sequence.image_attributes..., visible = false)
    end

    visitedframes = CircularBuffer(3)
    on(im_sequence[:frame]) do frame
        wait(imageframetasks[frame])
        foreach(f -> begin # make the previous plots invisible
                # NOTE: It is possible to notify frame without changing it
                if f != frame
                    imageframes[f].visible[] = false
                end
            end, visitedframes)
        imageframes[frame].visible[] = true
        push!(visitedframes, frame)
    end
    notify(im_sequence[:frame])

    return im_sequence
end

which works correctly when using the frame, but it takes the same time as my original synchronous version for some reason.

julia> @time l = ANT.Recipes.imagesequence(1, img_vec)
 44.776909 seconds (1.51 G allocations: 86.338 GiB, 46.70% gc time, 1.48% compilation time)

Synchronous version

using DataStructures, Images
using Makie
import Makie: convert_arguments
import MakieCore: argument_names

@recipe(ImageSequence) do scene
    Attributes(
        # Attributes to be used for the image plot. Including visible would break the mechanism for
        # showing the correct frame, so do not include `visible = true`.
        image_attributes = (; interpolate = false, inspectable = false)
    )
end

argument_names(::Type{<:ImageSequence}, numargs::Integer) = numargs == 2 && (:frame, :sequence)

function Makie.plot!(
    im_sequence::ImageSequence{<:Tuple{Integer,AbstractVector{<:AbstractMatrix{<:Colorant}}}})

    imageframes = Vector(undef, length(im_sequence[:sequence][]))
    for i in 1:length(imageframes)
         imageframes[i] = image!(im_sequence, im_sequence[:sequence][][i]; im_sequence.image_attributes..., visible = false)
    end

    visitedframes = CircularBuffer(3)
    on(im_sequence[:frame]) do frame
        foreach(f -> begin # make the previous plots invisible
                # NOTE: It is possible to notify frame without changing it
                if f != frame
                    imageframes[f].visible[] = false
                end
            end, visitedframes)
        imageframes[frame].visible[] = true
        push!(visitedframes, frame)
    end
    notify(im_sequence[:frame])

    return im_sequence
end

Which works as expected, but takes a long time (same as Attempt 2)

julia> @time l = imagesequence(1, img_vec)
 41.936290 seconds (1.51 G allocations: 86.258 GiB, 46.83% gc time, 0.09% compilation time)

Also, do you have an idea, why there are so much allocations? The original image sequence size is

.rw-r--r-- 252M kunzaatko 21 Mar  2021 TIRF488_cam1_0_L.tiff

and when only showing one frame from the total of 60 the allocation size is much smaller:

julia> @time image(img[:,:,1])
  0.026145 seconds (45.76 k allocations: 38.600 MiB)

(And also it is a mystery to me as to why the loading of the images takes so long, when

julia> 60 * 0.026145
> 1.5687

so why doesn’t it take only one this time to load them all?)

2

I’m just going to go off of this, as there’s a lot of code there and maybe a much simpler solution will suffice.

A normal image plot can take an Observable that you can update with new images whenever you want. You don’t need to create a recipe for that. Where that data comes from doesn’t really matter, you can of course use something like a frame number to take it from somewhere else. Like this:

images = [rand(50,50) for _ in 1:30] # thirty images stored in a vector, could be any other format as well
current_frame = Observable(1)
current_frame_data = @lift(images[$current_frame])

fig, ax, im = image(current_frame_data)
display(fig)

# now you can change the image data by selecting a different frame with the observable
current_frame[] = 2
1 Like
3

This works great! Why didn’t I think of that… Thank you!

4

For image sequence display, I like to use something like this:

images = [rand(Float32, 512, 512) for _ = 1:50]
f = Figure(resolution = (700, 900))

framerange = eachindex(images)
idxpad = maximum(length ∘ string, framerange)
ls = labelslider!(f, "Index", framerange, format = s -> lpad(s, idxpad))
ls_idx = ls.slider.value
f.layout[2, 1] = ls.layout

frame = @lift(images[$ls_idx])
ax = GLMakie.Axis(f[1, :], aspect=DataAspect())
image!(ax, frame)

on(events(f).keyboardbutton) do event # keyboard navigation
    if event.action in (Keyboard.press, Keyboard.repeat)
        event.key == Keyboard.left && (ls.slider.selected_index[] = clamp(ls.slider.selected_index[] - 1, eachindex(ls.slider.range[])))
        event.key == Keyboard.right && (ls.slider.selected_index[] = clamp(ls.slider.selected_index[] + 1, eachindex(ls.slider.range[])))
    end
end
1 Like