[GLMakie] Overlay plots in animated 3D scene?

I am now back at an old problem which I abandoned due to lack of time: displaying overlay plots (2D) in an animated 3D scene :wink:

I started a GH discussion back then and I got it almost working with the help of Simon (still could not fix the layering issue, the plot was always behind the 3D scene) but things have apparently changed now and I cannot even reproduce the old layout.

Just a quick reminder, this is what I am looking for (roughly):

The proposed code last year was

    subwindow = Scene(scene, px_area=Observable(Rect(100, 100, 200, 200)), clear=true, backgroundcolor=:green)
    subwindow.clear = true
    meshscatter!(subwindow, rand(Point3f, 10), color=:gray)
    plot!(subwindow, [1, 2, 3], rand(3))

which kind of worked and produced in the old application an overlay like this:

When I however try the same with the current version of GLMakie (v0.10.5) I get a full overlay of the plot and it’s still somehow behind the 3D scene, or at least some parts, it’s a bit weird:

Two questions:

  1. How do I adjust the size and position of the overlay plot correctly?
  2. How do I put the plot in the foreground so that the 3D scene is entirely behind the plot?

Any hints are highly appreciated, I tried so many things already :see_no_evil:

Here is a MWE which can be executed by running:

julia> include("scripts/uioverlayplot.jl")
eventloop (generic function with 1 method)

julia> app = initialize()

It’s a rough skeleton of the main application which the mentioned code block (four lines) which adds the subwindow:

using Makie
using GLMakie
using GLFW
using GeometryBasics


@kwdef mutable struct App
    scene::Scene = Scene(backgroundcolor=RGBf(0.9))
    cam::Makie.Camera3D = cam3d!(scene, rotation_center = :lookat)
    fps::Int = 60
end

"""
Draws a grid on the XY-plane with an optional `center` point, `span`, grid-`spacing` and
styling options.
"""
function basegrid!(app::App; center=(0, 0, 0), span=(-10, 10), spacing=1, linewidth=1, color=(:grey, 0.3))
    scene = app.scene
    min, max = span
    center = Point3d(center)
    for q ∈ range(min, max; step=spacing)
        lines!(scene, [Point3d(q, min, 0) + center, Point3d(q, max, 0) + center], color=color, linewidth=linewidth)
        lines!(scene, [Point3d(min, q, 0) + center, Point3d(max, q, 0) + center], color=color, linewidth=linewidth)
    end
    app
end



"""

Initialise the app, enter the event loop and return the app instance to
the REPL.

"""
function initialize()
    app = App()
    #center!(app.scene)
    update_cam!(app.scene, app.cam, Vec3f(5), Vec3f(0, 0, 0), Vec3f(0, 0, 1))

    n = 10

    meshscatter!(
        app.scene,
        rand(Vec3f, n),
        markersize = [rand()/10 for _ in 1:n]
    )

    lines!(app.scene, [rand(Vec3f) for _ in 1:n]; color=:red, linewidth=4)

    basegrid!(app)

    Threads.@spawn :interactive eventloop(app)
    app
end


function eventloop(app::App)
    screen = display(GLMakie.Screen(start_renderloop=false, focus_on_show=true, title="RainbowAlga"), app.scene)
    glw = screen.glscreen
    GLMakie.GLFW.SetWindowPos(glw, 100, 100)
    GLMakie.GLFW.SetWindowSize(glw, 400, 400)

    scene = app.scene

    # Adding the overlay plot
    subwindow = Scene(scene, px_area=Observable(Rect(100, 100, 200, 200)), clear=true, backgroundcolor=:green)
    subwindow.clear = true
    meshscatter!(subwindow, rand(Point3f, 10), color=:gray)
    plot!(subwindow, [1, 2, 3], rand(3))

    while isopen(screen)
        frame_start = time()
        rotate_cam!(scene, Vec3f(0, 0.005, 0))

        GLMakie.pollevents(screen)
        GLMakie.render_frame(screen)
        GLMakie.GLFW.SwapBuffers(GLMakie.to_native(screen))

        yield()

        sleep_time = 1.0/app.fps - (time() - frame_start)
        sleep_time > 0 && sleep(sleep_time)
    end

    GLMakie.destroy!(screen)
end
1 Like

px_area is called viewport now, does that help?

1 Like

Yes, now the position is correct again :smiley:

The missing piece is how to put things in foreground…

I mean, I would assume that (z?) layering is following the order of things added to the scene and the 2D overlay is the last I add.
Maybe the GLMakie.GLFW.SwapBuffers is causing the issue in the eventloop (isopen(screen)) while loop?

You could try setting depth_shift=1 (or 0, not sure which end of the scale) for the Scene of the overlay, or the plots within that Scene. Alternatively you can try simply translate!(subwindow, 0, 0, 9_999) or something which should get the subscene to be as close to the camera as is allowed…

depth_shift = 0.0 — adjusts the depth value of a plot after all other transformations, i.e. in clip space, where 0 <= depth <= 1. This only applies to
  GLMakie and WGLMakie and can be used to adjust render order (like a tunable overdraw).
1 Like

None of them work, unfortunately, I tried different settings and orders…

If I use translate!(subwindow, 0, 0, 9_999), the dots in the plot disappear, but I can still see the scene behind it :confused:

I still have no success, so I am happy for any kind of ideas :see_no_evil:

Just a small update: I started to investigate the depth buffer and this is what I see (by adding using the code blow, after displaying a few frames).


According to the depth buffer, the dots of the overlay map have a lower value, so they are in the near field compared to the rest, which is fine, they are also in front in the actual 3D scene. However, I can’t see the background of the subwindow in the depth buffer :laughing: I assume it is not rendered in the scene as such? I am really lost. According to the docs, the clear=true gets rid of the depth issue for overlapping scenes (A primer on Makies scene graph | Makie) I have no idea.

Let me desperately ping @sdanisch :see_no_evil:

        depth_color = GLMakie.depthbuffer(screen)
        close(screen)
        f, ax, pl = heatmap(depth_color)
        Colorbar(f[1, 2], pl)
        display(f)

Hm, I think there doesn’t seem to be a perfect solution right now, without a bit of a GLMakie refactor :frowning:
But some variants seem to be working reasonably well from what I can tell:

using GLMakie

# Should work best to avoid any overlap, but isn't interactive
# For slow interactions, might be possible to keep second screen 
# open and update the image for every frame
function inset_plot_cbuff!(plot_func, parent_figure, rect_px)
    fig = Figure(; size=widths(rect_px))
    plot_func(fig)
    screen = display(GLMakie.Screen(visible=false), fig)
    img = colorbuffer(screen)
    ip = mesh!(parent_figure.scene, rect_px, color=reverse(img; dims=1), shading=Makie.NoShading, fxaa=false)
    translate!(ip, 0, 0, 1000)
end

# works ok'ish, but only with scene 
function inset_plot_scene!(plot_func, parent_figure, rect_px)
    scene = Scene(parent_figure.scene; viewport=rect_px, clear=true, backgroundcolor=:gray)
    cam2d!(scene)
    mesh!(scene, Rect2f(0, 0, widths(rect_px)...), space=:pixel, color=:gray)
    translate!(scene, 0, 0, 10000)
    plot_func(scene)
end

# Works suprisingly well, Axis can be swapped with Axis3/LScene
# Although I think for lscene it's a bit more complicated
# and needs some form of mesh!(scene, Rect2f(0, 0, widths(rect_px)...), space=:pixel, color=:gray)
function inset_plot_ax!(plot_func, parent_figure, rect_px)
    ax = Axis(parent_figure; bbox=rect_px, backgroundcolor=:gray)
    ax.scene.clear = true
    translate!(ax.scene, 0, 0, 10000)
    plot_func(ax)
end

f, ax, pl = meshscatter(rand(Point3f, 100))

inset_plot_ax!(f, Rect2f(100, 100, 200, 200)) do ax
    lp = lines!(ax, cumsum(randn(1000)), color=:black, fxaa=false)
end;

1 Like

Ah, thanks Simon, that’s already good enough I think, I will play around with it today :smiley:
Not sure if I could have figured this out by myself :laughing: I already started to read the low-level GLFW docs :wink: