Fast display of an array as an image

I’m playing around with cellular automata and related systems, and I’m looking for a fast way to display them as they run.

Currently I’m using ImageView.imshow, which works, but it’s very, very slow - I can’t even resize the window to a convenient size without the frame rate dropping, so I have to view them in a tiny little corner of my screen.

So I’m looking for something faster. Is there another package that can display an array of RGB values as an image on the screen and then dynamically update that at a decent speed? I don’t need any fancy features beyond that, and speed is the only thing I really care about. (In particular, I don’t need to display the images on an axis with tick marks - just a window with the image in it would be fine.)

For example, can Makie.jl accomplish this task, and if so, is there some example code somewhere? I found this for displaying an image, but nothing about dynamically updating it.

In an ideal world I’d like to display each pixel of my image as an n x n block of screen pixels, so that they’ll all be perfectly square. If there’s something out there that can do that it would be amazing.

You’ll have to adapt the loop here to your own case, but the idea is the same: create an image plot of Observable data, then update the observable in a loop:

julia> img = Observable(rand(RGB{N0f8}, 512, 512));

julia> imgplot = image(@lift(rotr90($img)), axis = (aspect=DataAspect(),))

julia> t0 = time(); while time() - t0 < 10
           img[] = rand(RGB{N0f8}, 512, 512)
           sleep(1/30)
       end

With global variables and lazy coding (and without sleep), this delivers around 210 fps on my machine, which should be plenty for real-time updates.

edit: for a square image with one pixel per array element, try

julia> imgplot = image(@lift(rotr90($img)),
                      axis = (aspect=DataAspect(),),
                      figure = (figure_padding=0, resolution=size(img[])))

julia> hidedecorations!(imgplot.axis)
3 Likes

Thank you so much, that’s great!

…but I can only get it to work in the REPL. If I put the following in a file and run that instead, then no window ever appears. What am I missing?


using GLMakie, Images

println("finished loading packages - in theory I'm running now")

img = Observable(rand(RGB{N0f8}, 512, 512));

imgplot = image(@lift(rotr90($img)),
                      axis = (aspect=DataAspect(),),
                      figure = (figure_padding=0, resolution=size(img[])))

hidedecorations!(imgplot.axis)

t0 = time(); while time() - t0 < 10
    img[] = rand(RGB{N0f8}, 512, 512)
    sleep(1/30)
end

Ok, I figured that one out - I need a display(imgplot) if I’m not in a REPL. So here’s a complete working example.


println("loading packages (this may take a while)")

using GLMakie, Images

println("opening a plot window (this may take a while)")

img = Observable(rand(RGB{N0f8}, 512, 512));

imgplot = image(@lift(rotr90($img)),
                axis = (aspect=DataAspect(),),
                figure = (figure_padding=0, resolution=size(img[])))

hidedecorations!(imgplot.axis)

display(imgplot)

println("running...")

nframes = 0
tmax = 10
t0 = time();
while time() - t0 < tmax
    global nframes
    img[] = rand(RGB{N0f8}, 512, 512)
    sleep(0)
    nframes += 1
end

println("achived a fps of $(nframes/tmax)")

I’m getting a theoretical framerate of 178, but it looks a bit jerky, so I’m guessing it doesn’t actually display every frame. The time-to-first-plot is really really long unfortunately :frowning:

2 Likes

Yeah, Makie’s TTFP is infamously bad, but the flexibility, customizability, and time to nth plot are excellent. Hopefully the native code serialization efforts in Julia v1.9 will bring significant improvements in the TTFP department, but in the meantime, I’d recommend trying the VSCode Julia extension’s auto-sysimage tooling to reduce latency.

Assuming you aren’t doing much image processing beyond displaying your images, you can also reduce loading overhead by only pulling in ImageCore.jl instead of all of Images.jl.

You need to put your code in a function. You are running with non-const global variables, which is the number one thing to avoid in Julia.

In general: don’t loop in global scope. Use functions, and read: Performance Tips · The Julia Language

(If your code ever includes the keyword global, it is a big, flaming, red flag. Both for performance and ‘good programming practice’ reasons.)

2 Likes

Sure, of course I’ll do that in any ‘real’ code - the point here was only to get @stillyslalom’s code running in a file instead of the REPL.

I commented because you complained about performance, which seems odd if you were aware of the performance pitfalls…

Here it is in a function if you insist


println("loading packages (this may take a while)")

using GLMakie, ImageCore

function main()

    println("opening a plot window (this may take a while)")

    img = Observable(rand(RGB{N0f8}, 512, 512));

    imgplot = image(@lift(rotr90($img)),
                axis = (aspect=DataAspect(),),
                figure = (figure_padding=0, resolution=size(img[])))

    hidedecorations!(imgplot.axis)

    display(imgplot)

    println("running...")

    nframes = 0
    tmax = 10
    t0 = time();
    while time() - t0 < tmax
        img[] = rand(RGB{N0f8}, 512, 512)
        sleep(0)
        nframes += 1
    end

    println("achived a fps of $(nframes/tmax)")
end

main()

Framerate and TTFP are not significantly different, which is what I would expect - the overhead from globals should be trivial in this case, since most of the work is in array operations, presumably.

1 Like

I wasn’t really complaining about the framerate, by the way, sorry if it came across that way. 150-200 is respectable, and much better than ImageView was offering me. (It is a bit annoying if it’s only displaying some of the frames as I suspect, but that’s a different issue.)

I didn’t insist. I’m just explaining why I commented.

I often find that overhead very difficult to predict.

Fair enough. I don’t mean to come across as antagonistic or anything either. I appreciate the comment and feedback.

1 Like

I think you’re seeing frames dropped because the Julia-side display pipeline isn’t yielding to the OpenGL side often enough when running flat-out. That’s the value of sleep(1/30) (or 1/60 or some other reasonable frame rate) - it leaves time for graphics display. Ideally you’d be plotting asynchronously from the data-generating process, but I’m not sure how much time you want to invest in software engineering versus just playing around with cellular automata.

In Matlab I always call the drawnow function to flush the graphics queue and make sure everything displays on-screen. Isn’t there something similar in Makie?

Calling sleep(specific_number) means you have to make some sort of judgement call about how long to wait.

1 Like