Plots.jl Plot Coordinates - An Adventure

So I am on an adventure… To turn Plots.jl into something that can be extended to interactivity via GTK, or really anything hosting a plot. So in order to do that, I think there are a lot of ways, but one of the most fun for me is to convert UI coordinates to plot space. This should be a bit of an adventure, because who knows how Plots.jl actually renders plots and things? I really don’t, but this could be a fun way to explore it.

Lemme start off with some boiler plate to show what I got so far:

using Gtk, Plots, Cairo

minimaxi(x) = reduce( min, x ), reduce( max, x )
function minimaxi(x, padby)
    mi,ma = minimaxi(x)
    delta = (ma - mi)
    return mi - (delta * padby), ma + (delta * padby)
end

random_stuff = randn(300,5) .* 10.0
a = Plots.scatter(random_stuff, fmt = :png, title = "wow",
            ylim = minimaxi(random_stuff, 0.05));
#save the plot to png
png("/tmp/wut.png")
#load in the png to cairo
cairo_img = read_from_png("/tmp/wut.png"); # should be create_from_png
w, h = cairo_img.width, cairo_img.height;
println("Image Dimensions: $w by $h")
c = CairoRGBSurface(w,h);
cr = CairoContext(c);
set_source_surface(cr, cairo_img, 0, 0);

gtkcvs = @GtkCanvas(w+25,h+25)

cairo_img = read_from_png("/tmp/wut.png");
set_source_surface(cr, cairo_img, 0, 0);

@guarded draw(gtkcvs) do widget
    ctx = getgc(gtkcvs)
    h,w = height(gtkcvs), width(gtkcvs)
    set_source_surface(ctx, cairo_img, 0, 0);
    paint(ctx)
    fill(ctx)
end

gtkcvs.mouse.button1press = @guarded (widget, event) -> begin
    ctx = getgc(widget)
    set_source_rgb(ctx, 0, 0, 0)
    arc(ctx, event.x, event.y, 5, 0, 2pi)
    Gtk.stroke(ctx)
    reveal(widget)
    #Event coords are pixel space
    println((event.x, event.y))
    #Where the adventure begins
end

g = GtkGrid()
g[1,1] = GtkLabel("Look a plot!")
g[2,2] = gtkcvs

w = GtkWindow(g, "Plot Viewer", 700, 500);
Gtk.showall(w)

Basically we have a plot we can click on and draw circles ontop of. Real basic stuff. What I want is to know what those dots positions are in the plot space. Now, this is really a matter of algebra with the right constants. The question is finding those constants!

So the real meat of what happens in the plot, at least for GR seems to happen here: https://github.com/JuliaPlots/Plots.jl/blob/2022aebb07ce7822fc223cf118484fd66ba1ffaa/src/backends/gr.jl#L670

That’s where the adventure starts :slight_smile:

3 Likes

So I made a bit of the journey, I mean I have a thing that works pretty well! But, as always when you go exploring you end up with more questions then answers:

using Gtk, Plots, Cairo, Measures

minimaxi(x) = reduce( min, x ), reduce( max, x )
function minimaxi(x, padby)
    mi,ma = minimaxi(x)
    delta = ma - mi
    return mi - (delta * padby), ma + (delta * padby)
end

random_stuff = randn(3,5) .* 10.0
a = Plots.scatter(random_stuff, fmt = :png, title = "wow",
            ylim = minimaxi(random_stuff, 0.05));

function replaceBiggest(a,b)
    selinds = abs.(a) .< abs.(b)
    a[ selinds ] .= b[ selinds ]
    return a
end

#Get display
display_w_m, display_h_m, display_w_px, display_h_px = GR.inqdspsize()

#Find most extreme axis limits in all subplots
xplotlim, yplotlim = zeros(2), zeros(2)
for subplot in a.subplots
    global xplotlim = replaceBiggest(xplotlim, Plots.axis_limits(subplot, :x))
    global yplotlim = replaceBiggest(yplotlim, Plots.axis_limits(subplot, :y))
end
# w_px_per_mm = display_w_px / (display_w_m * 1000)
# h_px_per_mm = display_h_px / (display_h_m * 1000)
plot_max = maximum(Plots.gr_plot_size)
vp_plot_area_perc = Plots.viewport_plotarea
vp_plot_area_px = vp_plot_area_perc .* plot_max

#save the plot to png
png("/tmp/wut.png")
#load in the png to cairo
cairo_img = read_from_png("/tmp/wut.png"); # should be create_from_png
pltwidth, pltheight = cairo_img.width, cairo_img.height;
println("Image Dimensions: $pltwidth by $pltheight")
c = CairoRGBSurface(pltwidth,pltheight);
cr = CairoContext(c);
set_source_surface(cr, cairo_img, 0, 0);

gtkcvs = @GtkCanvas(pltwidth+25,pltheight+25)

cairo_img = read_from_png("/tmp/wut.png");
set_source_surface(cr, cairo_img, 0, 0);

@guarded draw(gtkcvs) do widget
    ctx = getgc(gtkcvs)
    h,w = height(gtkcvs), width(gtkcvs)
    set_source_surface(ctx, cairo_img, 0, 0);
    paint(ctx)
    fill(ctx)
end

gtkcvs.mouse.button1press = @guarded (widget, event) -> begin
    #Check bounds to ensure we are clicking inside the plot
    if (vp_plot_area_px[1] < event.x < vp_plot_area_px[2]) &&
            (vp_plot_area_px[3] < event.y < vp_plot_area_px[4])
        ctx = getgc(widget)
        set_source_rgb(ctx, 0, 0, 0)
        arc(ctx, event.x, event.y, 5, 0, 2pi)
        Gtk.stroke(ctx)
        reveal(widget)
        #Event coords are pixel space
        println("\t Pixels space: ", (event.x, event.y))
        npx = [ (event.x - vp_plot_area_px[1]) / (vp_plot_area_px[2] - vp_plot_area_px[1]),
                (event.y - vp_plot_area_px[3]) / (vp_plot_area_px[4] - vp_plot_area_px[3])]
        plotspace = [ xplotlim[1] + ( npx[1] * (xplotlim[2] - xplotlim[1]) ),
                      yplotlim[2] - ( npx[2] * (yplotlim[2] - yplotlim[1]) )
                     ]
        println( "\t Plot space: ", plotspace)
    end
end

g = GtkGrid()
g[1,1] = GtkLabel("Look a plot!")
g[2,2] = gtkcvs

w = GtkWindow(g, "Plot Viewer", 700, 500);
Gtk.showall(w)

My biggest question is probably, why does GR normalize it’s internal representation of the data, a so called NDC space, by the largest plot dimension? Usually each axis’ are considered independent from one another and thus individually normalized by their own max.

plot_max = maximum(Plots.gr_plot_size)
vp_plot_area_perc = Plots.viewport_plotarea
vp_plot_area_px = vp_plot_area_perc .* plot_max

Doesn’t that code just look wrong? I mean I did it sloppily but viewport_plotarea(the magic command for getting where the bounds of the plot are), returns (xmin,xmax,ymin,ymax) yet we normalize by the overall largest plot dimensions. It’s screwy, but that’s whats done?

Next biggest question is: How robust is this? Will it explode with different DPIs? I think not. What about if someone flips an axis(ooo maybe).

1 Like

here’s something a little cleaner… Latency doesn’t feel toooo bad. Would like some input, should probably refine a lot of things

using Gtk, Plots, Cairo, Measures
using DataStructures

function minimaxi(x, padby)
    mi,ma = extrema(x)
    return mi - ((ma - mi) * padby), ma + ((ma - mi) * padby)
end

function replaceBiggest!(a,b)
    selinds = abs.(a) .< abs.(b)
    a[ selinds ] .= b[ selinds ]
end

random_stuff = sin.( 0.0 : 0.01 : ( 10pi ) )
a = Plots.plot(random_stuff, fmt = :png, title = "", legend = false,
            ylim = minimaxi(random_stuff, 0.05));

mutable struct ClickablePlot
    plot::Plots.Plot
    canvas::GtkCanvas
    plotarea::Vector{Float64}
    cairoimg::Cairo.CairoSurfaceBase{UInt32}
    xplotlim::Vector{Float64}
    yplotlim::Vector{Float64}
    clickstate::UInt8
    uicoords::Queue{ Tuple{ Float64,Float64 } }
    plotcoords::Queue{ Tuple{ Float64,Float64 } }
end

function ClickablePlot( plt::Plots.Plot )
    #Find most extreme axis limits in all subplots
    xplotlim, yplotlim = zeros(2), zeros(2)
    for subplot in a.subplots
        replaceBiggest!(xplotlim, Plots.axis_limits(subplot, :x))
        replaceBiggest!(yplotlim, Plots.axis_limits(subplot, :y))
    end
    plot_max = maximum(Plots.gr_plot_size)
    vp_plot_area_px = Plots.viewport_plotarea .* plot_max
    png("/tmp/wut.png")                           #save the plot to png
    cairo_img = read_from_png("/tmp/wut.png")     #load in the png to cairo
    pltwidth, pltheight = cairo_img.width, cairo_img.height

    c = CairoRGBSurface( pltwidth, pltheight );
    cr = CairoContext( c );
    set_source_surface( cr, cairo_img, 0, 0 );
    gtkcvs = @GtkCanvas( pltwidth + 25, pltheight + 25 )
    return ClickablePlot(   plt,
                            gtkcvs, vp_plot_area_px,
                            cairo_img, xplotlim, yplotlim,
                            UInt8( 0 ),
                            Queue{ Tuple{ Float64,Float64 } }(),
                            Queue{ Tuple{ Float64,Float64 } }()
                        )
end

function make_selection_plot(cp::ClickablePlot)
    x = cp.plot.series_list[1].plotattributes[:x]
    y = cp.plot.series_list[1].plotattributes[:y]
    mini, maxi = reduce(min, [ f for (f,l) in cp.plotcoords]), reduce(max, [ f for (f,l) in cp.plotcoords ] )
    selection = mini .< x .< maxi
    if length( cp.plot.series_list ) > 1
        deleteat!(cp.plot.series_list, 2)
    end
    newplt = plot!(deepcopy(cp.plot), x[ selection ], y[ selection ],
                fillrange = [ zeros( sum( selection ) ), y[ selection ] ] )
    png(newplt,"/tmp/wut2.png")                           #save the plot to png
    cairo_img = read_from_png("/tmp/wut2.png")     #load in the png to cairo
    pltwidth, pltheight = cairo_img.width, cairo_img.height
    c = CairoRGBSurface( pltwidth, pltheight );
    cr = CairoContext( c );
    set_source_surface( cr, cairo_img, 0, 0 );
    cp.cairoimg = cairo_img
end

function render(ctx::CairoContext, cp::ClickablePlot)
    set_source_surface(ctx, cp.cairoimg, 0, 0);
    paint(ctx); fill(ctx)
    return nothing
end

function render_line(ctx::CairoContext, start, finish ; dash = nothing, RGB = (0,0,0) )
    @assert(length(start) == 2); @assert(length(finish) == 2); @assert(length(RGB) == 3)
    set_source_rgb( ctx, RGB... );
    if !isa( dash, Nothing )
        set_dash( ctx, dash )
    end
    move_to( ctx, start... );
    line_to( ctx, finish... );
    Gtk.stroke(ctx)
    return nothing
end

function attach_draw( cp::ClickablePlot )
    #Attach drawing routine
    @guarded draw( cp.canvas ) do widget
        ctx = getgc( cp.canvas )
        render(ctx, cp)
        for line in cp.uicoords
            render_line(ctx, [ line[1], 0 ], [ line[1], Gtk.height( cp.cairoimg )] ;
                        dash = [ 8.0, 8.0 ], RGB = ( 0, 0, 0 ) )
        end
    end
end

#Attach mouse events
function attach_mouse_down( cp::ClickablePlot )
    cp.canvas.mouse.button1press = @guarded (widget, event) -> begin
        #Check bounds to ensure we are clicking inside the plot
        if ( cp.plotarea[ 1 ] < event.x < cp.plotarea[ 2 ] ) &&
                ( cp.plotarea[ 3 ] < event.y < cp.plotarea[ 4 ] )
            ctx = getgc( cp.canvas )
            render(ctx, cp)
            render_line(ctx, [ event.x, 0 ], [event.x, Gtk.height( cp.cairoimg )] ;
                        dash = [ 8.0, 8.0 ], RGB = ( 0, 0, 0 ) )
            reveal(widget)
            #Event coords are pixel space
            npx = [ (event.x - cp.plotarea[1]) / (cp.plotarea[2] - cp.plotarea[1]),
                    (event.y - cp.plotarea[3]) / (cp.plotarea[4] - cp.plotarea[3]) ]
            plotspace = ( cp.xplotlim[1] + ( npx[1] * (cp.xplotlim[2] - cp.xplotlim[1]) ),
                          cp.yplotlim[2] - ( npx[2] * (cp.yplotlim[2] - cp.yplotlim[1]) ) )

            cp.uicoords = Queue{ Tuple{ Float64,Float64 } }()
            cp.plotcoords = Queue{ Tuple{ Float64,Float64 } }()
            enqueue!(cp.uicoords, ( event.x , event.y ) )
            enqueue!(cp.plotcoords, plotspace )
            #println( "\t Plot space: ", plotspace )
            cp.clickstate = 1
            reveal(widget)
        else #reset clickstate someone clicked elsewhere
            cp.clickstate = 0
        end
    end
    #handle dragging event
    cp.canvas.mouse.button1motion = @guarded (widget, event) -> begin
        #Check bounds to ensure we are clicking inside the plot
        if ( cp.plotarea[ 1 ] < event.x < cp.plotarea[ 2 ] ) &&
                ( cp.plotarea[ 3 ] < event.y < cp.plotarea[ 4 ] )
            if cp.clickstate == 1
                ctx = getgc( cp.canvas )
                render(ctx, cp)
                for line in cp.uicoords
                    render_line(ctx, [ line[1], 0 ], [ line[1], Gtk.height( cp.cairoimg )] ;
                                dash = [ 8.0, 8.0 ], RGB = ( 0, 0, 0 ) )
                end
                render_line(ctx, [ event.x, 0 ], [event.x, Gtk.height( cp.cairoimg )] ;
                            dash = [ 8.0, 8.0 ], RGB = ( 0.5, 0.5, 0.5 ) )
                reveal(widget)
            end
        end
    end
end

function attach_mouse_up( cp::ClickablePlot )
    cp.canvas.mouse.button1release = @guarded (widget, event) -> begin
        #Check bounds to ensure we are clicking inside the plot
        if ( cp.plotarea[ 1 ] < event.x < cp.plotarea[ 2 ] ) &&
                ( cp.plotarea[ 3 ] < event.y < cp.plotarea[ 4 ] )
            if cp.clickstate == 1
                #Event coords are pixel space
                npx = [ (event.x - cp.plotarea[1]) / (cp.plotarea[2] - cp.plotarea[1]),
                        (event.y - cp.plotarea[3]) / (cp.plotarea[4] - cp.plotarea[3]) ]
                plotspace = ( cp.xplotlim[1] + ( npx[1] * (cp.xplotlim[2] - cp.xplotlim[1]) ),
                              cp.yplotlim[2] - ( npx[2] * (cp.yplotlim[2] - cp.yplotlim[1]) ) )

                enqueue!(cp.uicoords, ( event.x , event.y ) )
                enqueue!(cp.plotcoords, plotspace )
                #println( "\t Plot space: ", cp.uicoords )
                make_selection_plot(cp)

                ctx = getgc( cp.canvas )
                render(ctx, cp)
                for line in cp.uicoords
                    render_line(ctx, [ line[1], 0 ], [ line[1], Gtk.height( cp.cairoimg )] ;
                                dash = [ 8.0, 8.0 ], RGB = ( 0, 0, 0 ) )
                end
                reveal(widget)
            end
        else
            cp.uicoords = Queue{ Tuple{ Float64,Float64 } }()
            cp.plotcoords = Queue{ Tuple{ Float64,Float64 } }()
            cp.clickstate = 0
        end

    end
end

cp = ClickablePlot( a )
attach_draw( cp )
attach_mouse_down( cp )
attach_mouse_up( cp )

g = GtkGrid()
g[ 1, 1 ] = GtkLabel( "Look a plot!" )
g[ 2, 2 ] = cp.canvas

w = GtkWindow( g, "Plot Viewer", 700, 500 );
Gtk.showall( w )
1 Like

I can’t really help you with the plotting, but I think you could use extrema(x) instead of minimaxi(x).

One thing I found that could help as well:
https://gr-framework.org/tutorials/windows_and_viewports.html

1 Like

Thanks for the extrema tip! I updated the code to include that.

Interesting find on windows/viewports. I’m still unsure how to use anything in that tutorial as it applies to Plots jl. But, there’s probably a way.

I think the functionality I want is implemented, but could use some guinea pigs to run the last bit of code in this thread and let me know if they feel it’s horribly slow. I am deep copying a plot, which is just bad news so I should fix that.

Okay so i did some revamping to try and make the solution more performant… Can anyone give this a whirl and give any feed-back? Still laggy? Would you prefer an area fill, or is a highlighted trace sufficient?

deleted code to put on GIST https://gist.github.com/caseykneale/49e447f41427cfdcc1efbd681c8f6833

As i mentioned in https://github.com/JuliaGraphics/Gtk.jl/issues/474, running longer examples from discourse is a little bit user-unfriendly … please put the code into a gist (or package or PR to a package).

2 Likes

sorry, never used gist before:
https://gist.github.com/caseykneale/49e447f41427cfdcc1efbd681c8f6833

1 Like

Hey @anon92994695, I’ve been catching up on your posts, it appears I’m interested in the same things as you. I would like an approach like this one to work, too.

Anyway, I’ve tried to run your gist above, and on my setup (Julia 1.5.2, macOS), it fails on line 191
a = Plots.plot(random_stuff, fmt = :png, title = "", legend = false)
with

2020-10-07 14:53:59.369 julia[56399:12904144] *** Terminating app due to uncaught exception 'GKSTermHasDiedException', reason: 'The connection to GKSTerm has timed out.'
*** First throw call stack:
(
)
libc++abi.dylib: terminating with uncaught exception of type NSException

signal (6): Abort trap: 6
in expression starting at none:0
__pthread_kill at /usr/lib/system/libsystem_kernel.dylib (unknown line)
Allocations: 42675722 (Pool: 42662350; Big: 13372); GC: 39
2 Likes

Well - I never tested the code on Mac, and I don’t know if Plots still supports this command :(. It might require the GR backend if it still works?

Maybe @mkborregaard would know?

I wish I could tell you the solution I came up with was practical - but I had to abandon making plots clickable. It did work at some point but I foresaw instabilities, and lots of other things kind of like this… This kind of behaviour is something Plots or some other package has to do or else it will always be some kind of “hack”…