Watch folder and plot new data in Makie

[garrek]

I have an experiment where a detector collects data and then writes a file. I want to watch a folder and when a new file appears, I want to load the data and plot it using GLMakie.

I am almost there but I’m hitting one major problem. My strategy is to read a directory readdir() and load existing files into a menu, very similar to the Menu example on the Makie website. I am successfully able to read text files, load the data, select the data from a menu, and plot them.

The problem is updating the menu. I am using the standard FileWatching watch_folder and trying to update my list of data and then update the menu, but I can’t simultaneously watch the folder and update the plot. I can successfully see when a file (with the appropriate extension) has been added, and I can add this to a list (files).

I think what I’m trying to do is stretching what GLMakie is designed for, but I’m curious if anyone has suggestions. Maybe I have to design an entire GUI, but it would be nice to do something simple.

Here’s my function, which I run from the REPL. The my_load_file_function function loads the files into a list of DataFrames and the columns are Observables.

There are several problems with this code, but the main point is that I don’t know how to update the menu when a new file has been added.

function run(datadir::String, extension::String=".lvm")

    fig = Figure()
    sc = display(fig)

    files = readdir(datadir)
    funcs = my_load_files_function(datadir)

    menu = Menu(fig, options = zip(sort!(["square root", "square", "sine"]), funcs))

    fig[1, 1] = vgrid!(
            Label(fig, "Functions", width = nothing), menu;
            tellheight = false, width = 200
    )

    ax = Axis(fig[1, 2])

    df_obs = Observable{Any}(funcs[1])
    col1 = Observable(:A)
    col2 = Observable(:B)
    data = @lift(Point2f.($df_obs[:, $col1], $df_obs[:, $col2]))

    scat = scatter!(ax, data)
    
    on(menu.selection) do s
        df_obs[] = s
        autolimits!(ax)
    end

    while true
        (file, event) = watch_folder(datadir)
        if endswith(file, extension)

            if !(file in files)
                println("New file: ", file)
                push!(files, file)
                println(files)
            end
        end
    end

end

[garrek]

Nevermind, I figured out a solution that works and it’s pretty cool.

Here’s my solution and below is a screenshot of it working!

I’ll write up a post on how to do it, since I haven’t seen anyone do anything similar yet.

function run(datadir::String, extension::String)

    fig = Figure()
    sc = display(fig)

    
    xdata = [rand(10)]
    ydata = [rand(10)]
    xs = Observable(xdata[1])
    ys = Observable(ydata[1])
    plotnames = Observable(["nothing"])
    plotname = Observable(plotnames[][1])

    menu = Menu(fig, options = plotnames)
    fig[1, 1] = vgrid!(
        Label(fig, "Data", width = nothing), menu;
        tellheight = false, width = 200
        )
    
    ax = Axis(fig[1, 2], title = plotname)
    line = lines!(ax, xs, ys)

    on(menu.selection) do s
        i = to_value(menu.i_selected)
        xs.val = xdata[i]
        ys[] = ydata[i]
        autolimits!(ax)
        ax.title = plotnames[][i]
    end

    while true
        (file, event) = watch_folder(datadir)

        if endswith(file, extension)

            x, y, name = loadfile(datadir, file, extension)

            if !(name in plotnames[])
                xs.val = x
                ys[] = y
                autolimits!(ax)
                ax.title = name

                push!(xdata, x)
                push!(ydata, y)
                plotnames[] = push!(plotnames[], name)
            end
        end
    end

end