Any touchscreen APIs for Julia?

Benny · August 27, 2020, 7:00am

Are there any packages that measure touchscreen gestures? I want to make something like this virtual piano (https://www.apronus.com/music/flashpiano.htm) but in an offline interactive window. So far I’m imagining that I just want to measure if and where there are touches at regular intervals, not anything like swipes or double-taps.

Palli · August 27, 2020, 8:14am

I just looked up for one of the cross-platform GUI toolkits:

https://developer.gnome.org/gtk3/stable/chap-input-handling.html

The most basic input devices that every computer user has interacted with are keyboards and mice; beyond these, GTK+ supports touchpads, touchscreens and more exotic input devices such as graphics tablets.

Qt or QML may also have this, and are maybe better options in general (I believe many think so, for windows/mouse GUIs). That said, I’m not sure the Julia wrappers for those, Gtkl.jl, QML.jl and Qt.jl support. I’m even skeptical of that, but please look into that (and extend if not, or file an issue).

The web APIs may also support, not sure you want to use program that way with Julia. And Android support for Julia is I think not at that level yet.

Benny · August 31, 2020, 12:16am

Yeah I’m not sure, either. I read on stackoverflow that multitouch support was added to Gtk some 7+ years ago for version 3.4, so I’m guessing Gtk.jl is using a later version than that. However, I can’t find anything in the Gtk.jl docs explicitly about touchscreen events. Gtk apparently has a GdkEventTouch struct for touchscreen actions, but I can’t find it in Gtk.jl’s github repo (though I should mention I never use github). I did find mentions of the GDK_TOUCH_xxxx event types, though. I’ll try to read up on how Gtk and Gtk.jl work, but boy do I wish I had some ready-made multitouch examples.

Palli · August 31, 2020, 12:16pm

I did find QTouchEvent in https://doc.qt.io/qt-5/qtouchevent.html

but it doesn’t seem to be supported in QML.jl, I didn’t see it exported. @barche might know if easy to support, it seems you’re out of luck with all the Julia wrappers for GUI toolkits. You might want to look into Kivy with Python, or some Qt library with Python or other lanuage. I mention Python as you should be able to use Python libraries with Julia, even frameworks, but I’ve not seen anyone actually use Python (web) frameworks (maybe in the past GUI frameworks), while I know of no one using touch related with Julia (or possibly in conjunction with Python).

Palli · August 31, 2020, 12:24pm

I did actually find “touch device” mentioned in this GUI wrapper:

github.com

Gnimuc/CImGui.jl/blob/e43324e0fd36aeb7709dc653f2252d065ee341fa/src/wrapper.jl#L284



!!! note
    1. You should always pass your mouse/keyboard inputs to imgui, even when the
        `ImGuiIO.WantCaptureXXX` flag are set false. This is because imgui needs to detect that you
        clicked in the void to unfocus its own windows.
    2. The `ImGuiIO.WantCaptureMouse` is more accurate that any attempt to "check if the mouse
        is hovering a window" (don't do that!). It handle mouse dragging correctly (both dragging
        that started over your application or over an imgui window) and handle e.g. modal windows
        blocking inputs. Those flags are updated by [`NewFrame`](@ref). Preferably read the flags
        after calling [`NewFrame`](@ref) if you can afford it, but reading them before is also
        perfectly fine, as the bool toggle fairly rarely. If you have on a touch device, you
        might find use for an early call to `UpdateHoveredWindowAndCaptureFlags()`.
    3. Text input widget releases focus on "Return KeyDown", so the subsequent "Return KeyUp"
        event that your application receive will typically have `ImGuiIO.WantCaptureKeyboard=false`.
        Depending on your application logic it may or not be inconvenient. You might want to
        track which key-downs were targeted for Dear ImGui, e.g. with an array of bool, and
        filter out the corresponding key-ups.)
"""
IsWindowHovered(flags=0) = igIsWindowHovered(flags)

@deprecate IsAnyWindowHovered() IsWindowHovered(ImGuiHoveredFlags_AnyWindow)

barche · August 31, 2020, 12:30pm

I have never tried it (due to the lack of a Julia-running touchscreen device), but I think this might work for touch input:

https://doc.qt.io/qt-5/qml-qtquick-multipointtoucharea.html

Conceptually, all these things are handled on the QML side, so it should all just work.

Palli · August 31, 2020, 12:35pm

Also for CImGui, another package I just found that might be usuful (or not, as searching for touch got me nothing, but at least might be helpful for non-touch GUI, maybe simplifying a lot): JuliaHub

Palli · August 31, 2020, 12:44pm

Nuklear has touch support, and seems the wrapper may have too: https://github.com/Gnimuc/Nuklear.jl/search?q=touch&unscoped_q=touch

barche · August 31, 2020, 12:51pm

QML.jl doesn’t have any ready-made examples for this either, but a possible starting point could be the board example:

Julia part:https://github.com/barche/QML.jl/blob/master/example/board.jl
QML part: https://github.com/barche/QML.jl/blob/master/example/qml/board.qml

In the QML part, you can try to replace the MouseArea with a MultiPointTouchArea and follow the QML examples from the Qt documentation from there: Qt Quick Examples - Touch Interaction | Qt Quick 5.15.7

Gnimuc · August 31, 2020, 3:28pm

I never tried Dear ImGui with a touch screen device, but it looks like the config flags have a IsTouchScreen entry:

github.com

ocornut/imgui/blob/13f718337ac4df2c66727e85174d48e9a91eabcf/imgui.h#L1121


    ImGuiConfigFlags_None                   = 0,
    ImGuiConfigFlags_NavEnableKeyboard      = 1 << 0,   // Master keyboard navigation enable flag. NewFrame() will automatically fill io.NavInputs[] based on io.KeysDown[].
    ImGuiConfigFlags_NavEnableGamepad       = 1 << 1,   // Master gamepad navigation enable flag. This is mostly to instruct your imgui back-end to fill io.NavInputs[]. Back-end also needs to set ImGuiBackendFlags_HasGamepad.
    ImGuiConfigFlags_NavEnableSetMousePos   = 1 << 2,   // Instruct navigation to move the mouse cursor. May be useful on TV/console systems where moving a virtual mouse is awkward. Will update io.MousePos and set io.WantSetMousePos=true. If enabled you MUST honor io.WantSetMousePos requests in your binding, otherwise ImGui will react as if the mouse is jumping around back and forth.
    ImGuiConfigFlags_NavNoCaptureKeyboard   = 1 << 3,   // Instruct navigation to not set the io.WantCaptureKeyboard flag when io.NavActive is set.
    ImGuiConfigFlags_NoMouse                = 1 << 4,   // Instruct imgui to clear mouse position/buttons in NewFrame(). This allows ignoring the mouse information set by the back-end.
    ImGuiConfigFlags_NoMouseCursorChange    = 1 << 5,   // Instruct back-end to not alter mouse cursor shape and visibility. Use if the back-end cursor changes are interfering with yours and you don't want to use SetMouseCursor() to change mouse cursor. You may want to honor requests from imgui by reading GetMouseCursor() yourself instead.

    // User storage (to allow your back-end/engine to communicate to code that may be shared between multiple projects. Those flags are not used by core Dear ImGui)
    ImGuiConfigFlags_IsSRGB                 = 1 << 20,  // Application is SRGB-aware.
    ImGuiConfigFlags_IsTouchScreen          = 1 << 21   // Application is using a touch screen instead of a mouse.
};

// Back-end capabilities flags stored in io.BackendFlags. Set by imgui_impl_xxx or custom back-end.
enum ImGuiBackendFlags_
{
    ImGuiBackendFlags_None                  = 0,
    ImGuiBackendFlags_HasGamepad            = 1 << 0,   // Back-end Platform supports gamepad and currently has one connected.
    ImGuiBackendFlags_HasMouseCursors       = 1 << 1,   // Back-end Platform supports honoring GetMouseCursor() value to change the OS cursor shape.
    ImGuiBackendFlags_HasSetMousePos        = 1 << 2,   // Back-end Platform supports io.WantSetMousePos requests to reposition the OS mouse position (only used if ImGuiConfigFlags_NavEnableSetMousePos is set).
    ImGuiBackendFlags_RendererHasVtxOffset  = 1 << 3    // Back-end Renderer supports ImDrawCmd::VtxOffset. This enables output of large meshes (64K+ vertices) while still using 16-bit indices.