Optimization of the use of observables and globals in Makie (from script to @main function)

General Usage Performance
, , ,
1

I am trying to make my Makie GUI run faster and allocate less. I have just upgraded to Julia 1.11.2 and thought it was a good opportunity to use function (@main) in other words, wrap all my global scope code into a function.

As an interesting observation: just going from Julia 1.10.5 to 1.11.2, I expected allocations to become worse, but it turned out the opposite. One function allocated 3.4 GB in 1.10 to just 736 MB in 1.11, another went from 7.36 GB down to 5.34 GB.

That probably does not sound good to you yet. My issue is likely that I am still using my main mutable struct and other arrays as typed global variables, as an earlier attempt to make one main function resulted in interactivity not working.

My program structure, current and planned:

# currently

using GLMakie, Shapefile, # etc
include("otherfunctions_datatypes.jl")  

userdirectory = "/data" # user defined folders

function example(axes,maindata:MyData, myarray::BitVector)
   # does sometimes something with globals or observables not explicitly imported into the function

# initialize
global (maindata,output2, etc) = load_data(userdirectory, somesettings)
begin 
   global somenames = somevalues # etc
   selection = Observable(falses(length(maindata.x))) #initialize 
   inside_timeinterval = Observable(trues(length(maindata.x))) #initialize 
   inside_area = Observable(trues(length(maindata.x))) #initialize 
   # set up the plotting figure, axes, buttons, toggles
   ....
end

@lift(selection[] = $inside_timeinterval .&& $inside_area .&& $somefilter)  
#  I found this to work, slightly different from the way the manual suggested it

on(selection) do sel
   empty(axes)
   if animate[] == true
       # splitting sel into steps, and then:
            for loop that includes calls to scatter( maindata.x[step] etc] )
    else
       scatter(  maindata.x[sel] etc)
    end
end

on(zooms) # trigger selection via inside_timeinterval[] = indices, etc
on(clicks) 
on(button1) 
     global (maindata, otherarrays) = load_data(...etc) 
     # sets other global settings or observables
end
on(toggles) # makes an observable change, but triggers immediate plotting by changing something tiny about the area limits to go into the on(selection)

Plan:

# with @main:

using GLMakie, Shapefile, # etc
include("otherfunctions_datatypes.jl")  

function (@main)(ARGS)
   userdirectory = "/data" # user defined folders, probably it is a better idea to make the program load a user setting file which can stay the same across program updates.
   maindata,output2, etc = load_data(userdirectory, somesettings)
   somenames = somevalues #  now has to include all global declarations including previously first defined in on(button) blocks
   selection = Observable(falses(length(maindata.x))) #initialize 
   inside_timeinterval = Observable(trues(length(maindata.x))) #initialize 
   inside_area = Observable(trues(length(maindata.x))) #initialize 
   # set up the plotting figure, axes, buttons, toggles
   ....
  @lift(selection[] = $inside_timeinterval .&& $inside_area .&& $sfilter)  
   
   on(selection) do sel
   on(zooms) # trigger selection via inside_timeinterval[] = indices, etc
   on(clicks) # trigger selection via inside_timeinterval[] = indices, etc 
   on(button1) 
     maindata, otherarrays = load_data(...etc) 
     # I suppose all the "global" statements can now be removed, but all have to be defined in main function scope before to make them visible outside the on block.
   end
   on(toggles)  
end # main function

function example(axes,maindata:MyData,observable1, etc)
   # all globals and observable settings imported explicitly

# to start, issue:  main(nothing)  in REPL, or julia -t auto thisprogram.jl

Do you agree or have other suggestions?

2

I usually have a global struct where all members are concrete types. This should be sufficient, I think. If you use dicts make sure the members of the dict also have concrete types and are not any.

And just check the allocations of your functions one by one, starting with the inner-most functions, using @alloc or @time or @btime.

3

You need to profile which allocations slow you down, which interactive code actually matters. Usually the type instabilities only matter if you don’t have function barriers and directly do data processing on the global variables. If you just call functions on them that do the heavy lifting, you’ll be type stable inside those (unless there are other problems there)

4

Thanks for your help, yes, this is exactly how it currently is working. Processing of the struct fields (arrays of typically 100,000 to 2,000,000 elements) happens within functions, taking care of their concrete types. So, according to your responses, the main function would likely offer no further performance benefit. That is useful to know, to prioritize my steps.

I tried to follow advice from the Performance section of the manual, and also from OhMyThreads.jl. I guess that many benefits of the mostly computational examples cannot apply when I need to update values inside a struct based on conditions or clustered elements. My code likely breaks the advice of accessing arrays consecutively instead of randomly, probably false sharing occurs in my parallel loops. Just the act of indexing one array by another inside a loop is a big allocation source I am aware of now, but cannot always avoid.
Examples of writing to a temporary thread-local memory tend not to be so useful to me, as I need the results written to specific locations in my existing arrays.

It also is rare if my CPU uses more than 30-50% of its capability, indicating some likely bottlenecks in data access.

Is this an alarming number of lock conflicts?
3.354939 seconds (11.97 M allocations: 5.403 GiB, 28.02% gc time, 79896 lock conflicts)