I have a program where I have an input feed (via ZMQ) that constantly updates an Array to maintain current state. I have other processes (either separate Tasks via @async or maybe on other threads via multithreading, still a design choice in flux) which will need to periodically access the most current state of this array.
My current design simply maintains this State Array as a global variable, and copies it for each task that is generated to process state. I was reading through the performance tips about not using global variables, but I am not seeing any other way to structure the program. This seems like it must be a pretty common design pattern, though, so I was wondering if there is a better approach to solving this problem in more performant way? Performance, in this context, means doing the calculations with the most current State as fast as possible.
Thank you. The @lock macro helps solve a very specific problem, and I was not at all aware of it.
As I dug into the this issue, I discovered I did not really understand how the $ interpolation for Threads.@spawn works for structures (more discussion here). At any rate, now I’m trying to find a way to measure the impact of using lots of copy operations when all I want to do is read state for a very brief amount of time (but ensure that, for a few milliseconds, I see the full state exactly as it was when my calculation started, no updates). But during this, I may be spawning another process that does need the most up-to-date state…so I cannot lock updates to state during my calculation…
Looks like I need each spawned process to have it’s own copy. But given that only a few changes will happen during the spawned process, still seems like a waste to copy everything.
1 Like