Makie.jl Mouse Interaction Question/Discussion

standarddeviant · September 1, 2018, 5:19am

I recently got Julia 1.0.0 up and running with my favorite packages and tried out Makie and interactive plotting just worked!

I have a question and potential suggestion about the mouse interaction model and curious to see other peoples’ views.

I noticed that the default mouse interactions are

pan = right-click + drag
XY-zoom = scroll wheel

The default interaction behavior of pyqtgraph is something special in the world of visualization tools. Its mouse interactions are

pan = left-click + drag
X-zoom = right-click + drag left/right
Y-zoom = right-click + drag up/down

I found that model to be extremely useful in ‘navigating’ long time-series of multi-channel data.

I’m wondering…

How difficult would it be to achieve this behavior with Makie.jl? Can I just call a few functions and get this behavior?
Are others interested in this kind of interaction model and have others used pyqtgraph or its ‘zooming’ model?

lobingera · September 1, 2018, 12:28pm

I have no details, but i’d assume that current interaction is hard coded. As the code is largely uncommented, you should follow event processing in the backends.

sdanisch · September 3, 2018, 7:26am

you should follow event processing in the backends.

What do you mean by that?

Well the good news is, that it’s not hard coded.

github.com

JuliaPlots/AbstractPlotting.jl/blob/master/src/camera/camera2d.jl#L15


      
              panbutton::Node{Union{ButtonTypes, Vector{ButtonTypes}}}
              padding::Node{Float32}
              last_area::Node{Vec{2, Int}}
              update_limits::Node{Bool}
          end
          
          """
              cam2d!(scene::SceneLike, kwargs...)
          
          Creates a 2D camera for the given Scene.
          """
          function cam2d!(scene::SceneLike; kw_args...)
              cam_attributes = merged_get!(:cam2d, scene, Attributes(kw_args)) do
                  Attributes(
                      area = Node(FRect(0, 0, 1, 1)),
                      zoomspeed = 0.10f0,
                      zoombutton = nothing,
                      panbutton = Mouse.right,
                      selectionbutton = (Keyboard.space, Mouse.left),
                      padding = 0.001,
                      last_area = Vec(size(scene)),

So you can just do e.g.:

scene = Scene()
cam = cam2D!(scene) # you can also get the cam from an existing scene
cam[:panbutton] = Mouse.left

Changing zoom is a bit more involved, since it’s a whole different way of zooming.
But the camera is composable, so you can take the camera code here:

github.com

JuliaPlots/AbstractPlotting.jl/blob/master/src/camera/camera2d.jl#L19


      
          end
          
          """
              cam2d!(scene::SceneLike, kwargs...)
          
          Creates a 2D camera for the given Scene.
          """
          function cam2d!(scene::SceneLike; kw_args...)
              cam_attributes = merged_get!(:cam2d, scene, Attributes(kw_args)) do
                  Attributes(
                      area = Node(FRect(0, 0, 1, 1)),
                      zoomspeed = 0.10f0,
                      zoombutton = nothing,
                      panbutton = Mouse.right,
                      selectionbutton = (Keyboard.space, Mouse.left),
                      padding = 0.001,
                      last_area = Vec(size(scene)),
                      update_limits = false,
                  )
              end
              cam = from_dict(Camera2D, cam_attributes)

and change it to something like:

function mycam2d!(scene::SceneLike; kw_args...)
    cam_attributes, rest = merged_get!(:cam2d, scene, Attributes(kw_args)) do
        Theme(
            area = Signal(FRect(0, 0, 1, 1), name = "area"),
            zoomspeed = 0.10f0,
            zoombutton = nothing,
            panbutton = Mouse.right,
            selectionbutton = (Keyboard.space, Mouse.left),
            padding = 0.001
        )
    end
    cam = from_dict(Camera2D, cam_attributes)
    # remove previously connected camera
    disconnect!(camera(scene))
    my_zoom!(scene, cam) # add your own zoom
    add_pan!(scene, cam)
    correct_ratio!(scene, cam)
    selection_rect!(scene, cam, cam_attributes[:selectionbutton])
    cameracontrols!(scene, cam)
    cam
end

I hope the code in the file will help to write my_zoom

piever · September 3, 2018, 8:34am

I’m not sure how one would do this, but another common (and IMO useful) zoom for 2D plots is the “box zoom”: the user draws a rectangular box around the region of interest (by left clicking and dragging) and that region expands to take the whole space. Double-clicking brings it back to the original state: see Plotly for an example implementation. Not sure how easy/hard this is to implement and whether it should be a opt-in or default behavior (I tend to think this kind of things should be easy to access / default, as only advanced users will write code to customize mouse interaction).

sdanisch · September 3, 2018, 8:39am

That’s already implemented:

selectionbutton = (Keyboard.space, Mouse.left)

and I do hope I wrote it in a way that it’s easily customizable… After all the camera is basically also just a recipe, which can be swapped out with your own code building up on the existing camera building blocks!

piever · September 3, 2018, 8:53am

Cool! Just two remarks:

I think the “box zoom” code still needs some updating, I’m on master of AbstractPlotting and I get:

MethodError: no method matching ntuple(::getfield(AbstractPlotting, Symbol("##191#192")){Vec{2,Float32}}, ::Type{Val{2}})
Closest candidates are:
  ntuple(::F, ::Integer) where F at tuple.jl:133
  ntuple(::Any, ::Val{0}) at tuple.jl:155
  ntuple(::Any, ::Val{1}) at tuple.jl:156
  ...
absrect(::GeometryTypes.HyperRectangle{2,Float32}) at /home/pietro/.julia/packages/AbstractPlotting/ZHerD/src/camera/camera2d.jl:143 	31 	2.8k 	3d
Julia for live control of USB3.0 camera and DAC/ADC boards, with GUI? 1 	Usage 	11 	208 	5d
Too many ways to do the same thing in Julia 1 	Usage 	16 	643 	9d
Is it possible to override keyword arguments for specific types? 2
question
	Usage 	7 	158 	10d
There are 95 unread and 14 new topics remaining, or browse other topics in Visualization
sdanisch

Cool! Just two remarks:

    I think the “box zoom” code still needs some updating, I’m on master of AbstractPlotting and I get:

MethodError: no method matching ntuple(::getfield(AbstractPlotting, Symbol("##191#192")){Vec{2,Float32}}, ::Type{Val{2}})
Closest candidates are:
  ntuple(::F, ::Integer) where F at tuple.jl:133
  ntuple(::Any, ::Val{0}) at tuple.jl:155
  ntuple(::Any, ::Val{1}) at tuple.jl:156
  ...
absrect(::GeometryTypes.HyperRectangle{2,Float32}) at /home/pietro/.julia/packages/AbstractPlotting/ZHerD/src/camera/camera2d.jl:143

I think it’s important to have this easily customizable, esp. in the case of touch pad users. On many laptops there is a mechanism to block touch pad when typing so Space + Mouse Left is effectively disabled

sdanisch · September 3, 2018, 8:56am

I think it’s important to have this easily customizable, esp. in the case of touch pad users. On many laptops there is a mechanism to block touch pad when typing so Space + Mouse Left is effectively disabled

That’s why you can just change it It’s even themeable, so you can change it globally!

piever · September 3, 2018, 9:07am

This is really cool, I’m slowly wrapping my head around it. Here is the PR to fix the behavior on Julia 1.0.

I tried scatter(rand(100), rand(100), selectionbutton=Mouse.left) but it doesn’t work, what’s the correct way to do this?

sdanisch · September 3, 2018, 9:13am

ah, I don’t pass through the attributes:

https://github.com/JuliaPlots/AbstractPlotting.jl/blob/master/src/basic_recipes/basic_recipes.jl#L400
But otherwise, this should be how it works:

scatter(rand(100), rand(100), cam2d = NT(selectionbutton=Mouse.left))

I should remove the NT and just use NamedTuples on 1.0!

piever · September 3, 2018, 9:25am

Mhm, still doesn’t work:

julia> p = scatter(rand(100), rand(100), cam2d = NT(selectionbutton=Mouse.left))

julia> p.theme[:cam2d].attributes
Dict{Symbol,Reactive.Signal} with 6 entries:
  :padding         => 5422: "padding" = 0.001 Any 
  :selectionbutton => 5421: "selectionbutton" = (space::Button = 32, left::Button = 0) Any 
  :zoomspeed       => 5418: "zoomspeed" = 0.1 Any 
  :panbutton       => 5420: "panbutton" = right Any 
  :area            => 5417: "area" = GeometryTypes.HyperRectangle{2,Float32}(Float32[0.0, 0.0], Float32[1.0, 1.0]) Any 
  :zoombutton      => 5419: "zoombutton" = nothing Any

But if I change it manually it works:

p.theme[:cam2d][:selectionbutton] = Mouse.left

But I think as soon as this is figured out, it definitely deserves a section in the docs, it’s really nice to be able to define custom interactive behavior at the single plot level or globally with a theme.

piever · September 3, 2018, 9:29am

Oh, and a last thing: did you also implement a “reset button” to go back to the original plot size?

Sorry for the many questions but I’m getting really excited about the possibilities Makie offers.

sdanisch · September 3, 2018, 9:30am

Me:

ah, I don’t pass through the attributes:

What I meant is, that scatter(rand(100), rand(100), cam2d = NT(selectionbutton=Mouse.left)) currently doesn’t work!

"reset button” to go back to the original plot size?

Ah no, that is missing but should be easy to implement!