Is there a fast way to turn a plot (from either Plots.jl or Makie) into an array of pixel values? Basically, I would like to be able to draw a few lines, and then turn that drawing into an Array of UInt8 or similar, that I can use for further calculations.
(I could get the data that I need by saving a .png, then loading the image, but I need to do this hundreds of millions of times, so speed matters.)
Right away, GLMakie.scene2image(scene) should give you what you want.
But that’s not super fast - even faster would be:
screen = display(your_scene)
image = colorbuffer(screen)
If you tell a bit about your use case, there might be even faster ways 
Thanks. This seems to be exactly what I was looking for.
My use case is generating a “movie” (time series of images) from a stochastic model. The movie is then used as training input for a kind of neural network. If there were a way to create movie frames in parallel on the GPU, that would be ideal, as the training takes place there.
If all you want to do is draw lines (or other primitive shapes), and you don’t need any fancy plotting functionality, you can probably save a ton of time by implementing your own line drawer onto an UInt8 matrix. It’s not terribly difficult, and there’s plenty of examples online you can use as a starting point.
Yes. This is exactly what I have been doing so far. But I think I will be using increasingly complex shapes, and I’d rather spend my time and energy on the neural network model than writing a new graphics library from scratch…
Can you point to an example. I am not sure what to google for exactly. I have the same problem that I’d like to create a movie of an agent-based model evolving through time.
Sure. For lines, a simple and efficient algorithm is Bresenham’s. There’s pseudo code on that page, and you can find thousands of implementations online if you google for “Bresenham”, which can be trivially ported to Julia, or probably you can find Julia implementations already written. Here’s an example:
function line!(A, x0, y0, x1, y1)
dx = abs(x1 - x0)
sx = x0 < x1 ? 1 : -1
dy = -abs(y1 - y0)
sy = y0 < y1 ? 1 : -1
err = dx + dy
while true
A[y0, x0] = 1
x0 == x1 && y0 == y1 && break
e2 = 2 * err
e2 >= dy && (err += dy; x0 += sx)
e2 <= dx && (err += dx; y0 += sy)
end
end
Usage:
julia> A = zeros(UInt8, 8, 8);
julia> line!(A, 2, 2, 7, 6);
julia> @. Char(A*56)+32
8Ă—8 Array{Char,2}:
' ' ' ' ' ' ' ' ' ' ' ' ' ' ' '
' ' 'X' ' ' ' ' ' ' ' ' ' ' ' '
' ' ' ' 'X' ' ' ' ' ' ' ' ' ' '
' ' ' ' ' ' 'X' 'X' ' ' ' ' ' '
' ' ' ' ' ' ' ' ' ' 'X' ' ' ' '
' ' ' ' ' ' ' ' ' ' ' ' 'X' ' '
' ' ' ' ' ' ' ' ' ' ' ' ' ' ' '
' ' ' ' ' ' ' ' ' ' ' ' ' ' ' '
Of course implementing one’s own drawing code sounds like a terrible idea and a last resort, but presumably it can be made significantly faster than going through an existing plotting library and rasterizing the result at the end.
It will depend a bit on the quality you want. The classic anti-aliasing algorithm is the “Bresenham algorithm”. A search for that should get you somewhere.
I doubt that you’ll be faster than Makie in the end, especially for many drawing operations!
If I understood the original post correctly, he wanted to draw a few lines, then rasterize the result to an UInt8 matrix, and repeat that whole process hundreds of millions of times. I haven’t benchmarked it, but I would suspect that this can be done considerably faster on pure CPU writing straight to the UInt8 matrix (~1 μs per line), than on GPU and transfer the results to system memory for each frame. Don’t you think?
If he wants to draw more complex shapes, and/or more than a few lines, that’s a different situation, but then it probably needs to be looked at in a broader perspective, not just what’s the fastest way to rasterize an image regardless of plotting library, but the fastest overall to draw the desired shapes and get the resulting image.
I guess it was a bit ballsy, considering how little information there are
Really depends on the size of the matrix and the number of shapes!
Transfer costs for a 960x540 color targets is ~70μs, which will be the constant overhead you’ll need to pay - but the drawing should be much faster than on the CPU!
But you could actually directly render to a uint8 rendertarget and make a cuda buffer out of it, which you could directly use with CuArrays + Flux That should be quite unbeatable!
Yes, like I said, I will do further processing on the GPU in any case, so for me it would be an advantage that the drawing happens there, if I can then get the image directly as a CuArray.
Is there a way to do that?
It’s not super straight forward… I tried it a couple of times and it was always a hassle to get this right.
You can take a look at the last message in:
using Makie, GLMakie
scene = scatter(rand(5))
screen = display(scene)
color_texture = screen.framebuffer.color;
This gives you the texture which should hold all the right handles
Thanks for the info! Very likely AbstractPlotting.colorbuffer(screen) will already be fast enough in relation to the rest of my code, so if it’s too much of a hassle to avoid the return trip to CPU memory, then I’ll just let it be.
Btw, if you use colorbuffer, you can also disable the renderloop to do a bit less duplicate work:
GLMakie.opengl_renderloop[] = (screen) -> nothing
you could also PyCall.jl out to Module: draw — skimage v0.20.0.dev0 docs if ImagesDraw.jl is not enough.
ImageDraw looks like it might be quite useful. Is there documentation for it anywhere?
use the tests Luke
Hi all,
I am looking for a way to turn plots(Plots.jl) into image pixel color value 2D array too. Then using each image to generate each video frame (as described in Writing Videos · VideoIO.jl )
Any updates on a solution for this?
I can use png() to get iobuffer then convert that(using Images.jl) back to an image array, but that’s two extra steps.
The Makie method as suggested by @sdanisch works well. But I am facing memory leak(Makie memory leak? · Issue #795 · MakieOrg/Makie.jl · GitHub) on each plot and julia crashes before I can finish generating all my frames. I am also thinking Plots.jl might be more lightweight compared to using GLMakie.
Welcome for any suggestions! 