Fastest way to blit images to a window


#1

Hi, I’m in the process of writing a laser beam profiler software with Julia. Fo this I am grabbing greyscale images from a scientific camera, that can be up to 4000x3000 pixels, stored as wither UInt8 or UInt16.

I now want to show these on the screen as fast as possible. So far I have tried Plots. jl using:

using Plots, Vimba # Vimba is the camera library (will be released as a package soon)
glvisualize()

# this returns a closure get() that grabs the next image
get, close = make_get()

plt = heatmap(get())
gui(plt)

# get() calls the c library using @threadcall so can be used async
function myrun()
    while true
        heatmap!(plt, get())
        gui(plt)
    end
end

@async myrun()

But even using glvisualize this is not super fast, definitely showing the images more slowly than my camera.
I have a rough code working in python using https://github.com/pyqtgraph/pyqtgraph which is much faster, but that uses the Qt scenegraph, so I thought glvisualize might be faster. And I would much prefer to use Julia.

So my question is: what is the fastest way to show a colormapped image of a large 2d UInt8 or UInt16 array? I’m happy to use any graphics backend necessary. (As long as compatible with 64 bit Windows 10)

I have found this discussion https://groups.google.com/d/topic/julia-users/rshpENHvHao/discussion, so could try Gtk.jl as suggested by @tim.holy but wondered if maybe GLVisualize code from @sdanisch might be better.

In the end I will also need some controls (sliders, numeric input, buttons) and also some simple line plots too.

Thanks for any help!


#2

I would expect that GLVisualize to be the fastest, but that’s what Plots should be using (though perhaps there might be extra overhead). If you want to try Gtk, perhaps the easiest approach would be to use GtkUtilities.jl, where you can just copy! the image onto a Canvas. Example:

using GtkUtilities, Gtk.ShortNames, Colors
win = @Window()
c = @Canvas()
push!(win, c)
showall(win)
fill!(c, colorant"red")
# you should have a red window now
using Images, TestImages
img = testimage("mandrill");
copy!(c, img)
# you should see the mandrill now

#3

In such a case, I wouldn’t be surprised if Plots.jl is the bottleneck.
Also, is it possible to not allocate a new image every frame?
e.g. using get!(buffer), or will this happen internally anyways?
try this code:

using GLVisualize, Colors
w = glscreen()

function main(w, img)
    while isopen(w)
        rand!(img)# get[!](img)
        GLAbstraction.set_arg!(obj, :image, reinterpret(Gray{U8}, img))
        GLWindow.render_frame(w)
        GLWindow.swapbuffers(w)
        GLWindow.poll_glfw()
        yield()
    end
end

img = rand(UInt8, 4000, 4000)
imgi = reinterpret(Gray{U8}, img)
obj = visualize(imgi); _view(obj)
main(w, img)
GLWindow.destroy!(w)

This runs very smooth on my PC. If not, please open an issue!
Sliders, buttons etc can be brought in with GLPlot/GLVisualize!
E.g like in: https://github.com/JuliaGL/GLVisualize.jl/blob/master/examples/gui/buttons.jl

You can also use Signals with a much easier setup (without GLWindow.render/swapbuffers etc) which I believe will almost be as fast, but to be sure I gave you the version with as little overhead as possible :wink:


#4

Overlooked that!
It’s very similar, but will only work with floating point for now, so you’d need something like that:

using GLVisualize, Colors, GeometryTypes
w = glscreen()

function main(w, img)
    while isopen(w)
        rand!(img)# get[!](img)
        GLAbstraction.set_arg!(obj, :intensity, map(GLVisualize.Intensity{1, Float32}, img))
        GLWindow.render_frame(w)
        GLWindow.swapbuffers(w)
        GLWindow.poll_glfw()
        yield()
    end
end

img = rand(UInt8, 4000, 4000)
imgi = map(GLVisualize.Intensity{1, Float32}, img)
obj = visualize(
    imgi,
    color_map = [RGBA{Float32}(1, 0, 0), RGBA{Float32}(0, 1, 0)], # any vector of colors will do
    color_norm = Vec2f0(0, 255) # map intensity between 0 and 255 befor color lookup in [0-1]
)
_view(obj)
main(w, img)
GLWindow.destroy!(w)

Intensity only works for Float32 right now… If that introduces an overhead for you, please open an issue.
Should be straight forward to fix it to work with UInt8!


#5

It actually just seems to be this line, that doesn’t work with arbitrary element types:


If you’re supplying the value for color_norm so that it calls const_lift(extrema2f0, main) it might just work.


#6

@sdanisch Awesome! That works extremely well! Thank you very much.

I need to somehow work this into a proper interface, so I’ll probably need more help at some point, but this displays the images perfectly.

One question: is there any documentation for things like visualize and its various arguments? I can’t find much and this seems to be quite a magic function!

Also, I’ll look through all the examples and code etc. but can you point me in the right direction for getting things like mouse dragging and scroll wheel working so that I can do panning etc. of the display. (I will work it out, but I just need some examples of catching the events etc.)

Thanks for an amazing package!


#7

Great!

There is:

arg1 = [Intensity{1, Float32}(1) for i=1:2, j=1:2]
visualize(arg1) -> visualize command 
GLVisualize.get_docs(arg1) -> keyword args for above command with some docs.
GLVisualize.all_docs() ->  Displays all the possible visualize args with a short description of the resulting visualization

Also, try Pkg.test("GLVisualize")! It’ll walk you through the examples with some guidance.

You should be able to drag and zoom with ctrl pressed + left mousebutton or scroll wheel…I guess this should be documented somewhere more prominent!


#8

the code you posted for showing the camera image is great for spectrogram viewing, except I’m often dealing with really wide or tall aspect ratios. Is there an easy way to tell visualize to scale the aspect ratio to the window?


#9

Yes, with the primitive = SimpleRectangle(...) keyword argument:


using GLVisualize, Colors, GeometryTypes
w = glscreen()
img = rand(UInt8, 1000, 4000) #<---- different ratio
imgi = map(GLVisualize.Intensity{1, Float32}, img)
obj = visualize(
    imgi,
    primitive = SimpleRectangle(0, 0, 500, 500),
    color_map = [RGBA{Float32}(1, 0, 0), RGBA{Float32}(0, 1, 0)], # any vector of colors will do
    color_norm = Vec2f0(0, 255) # map intensity between 0 and 255 befor color lookup in [0-1]
)
_view(obj)
main(w, img)
GLWindow.destroy!(w)

Admittedly that is a bit low level…
Also for surface, you control the same property with ranges = ((0, 100), (0, 100)), which is inconsistent!


#10

Hmm, that doesn’t seem to quite work for me. I’m invoking visualize with:

obj = visualize(
        reinterpret(Gray{Float32}, data),
        primitive = SimpleRectangle(0, 0, 500, 500))

Where data is a 1406x257 array of Float32, and then updating with GLAbstraction.set_arg!(obj, :image, reinterpret(Gray{Float32}, data)). What I get is below, which is cutting off the top of the image and showing white space on the right side.

Screen Shot 2017-01-11 at 10.04.43 AM.png


#11

Sorry, I should answer more carefully :wink:
Of course for the window aspect ratio, you need to insert the window size :stuck_out_tongue:
so you need:

 primitive = SimpleRectangle(0, 0, widths(window)...)

and if you want it to stay there, use:

_view(obj, camera = :fixed_pixel)

#12

Awesome, that’s much better, thanks! My mental model of how the various GL* things work together is pretty weak.


#13

@ssfrr, btw, what frame rates did you finally achieve?


#14

@jtravs have you made any progress with the Vimba library? I have been trying to wrap the Vimba library and some other camera APIs using Clang.jl but have been running into some problems. Would be great to see a julia package for Vimba!