Optimum use of Julia for two tasks

General Usage Performance
1

I have been working on an application that first collects data and then stores it to file. Then the file is read, processed and summary information stored in a database. The system is meant to operate continuously.

function main()
    # the following two functions must operate in parallel
    acquire()   # call the function to acquire the data
    process()  # call the function to process the data
end

function acquire()
    while true
        # periodically acquire data and store to file
        # this function relies on [ccall](https://digilent.com/shop/mcc-172-iepe-measurement-daq-hat-for-raspberry-pi/) into a .so file to perform its acquisition
        # the documentation of the ccall indicates a separate c process is used
        # whether this is blocking or not I don't know
    end 
end

function process()
    while true
        # look for files stored by acquire() and process them
    end 
end

The current implementation is on a Raspberry PI, which has a 4 core CPU and 8 GB of RAM.

I can see a number of alternatives of how to implement this.

  1. Start two independent Julia sessions, one to acquire and another to process the data.
  2. One julia session with two processes julia -p 2 and run acquire() and process() in separate processes
  3. One julia session and use asynchronous programming to create two tasks that continue forever.
  4. Use an external package like JobScheduler.jl. This seems similar to 3 but we can also assign the amount of memory and priority of each function.
  5. Finally forget parallelism and every time an acquisition finishes, process the data and then check if another acquisition should be performed.

My question is which of these should I use. Considerations are resource use and ease of implementation. I suspect that with two sessions of Julia running as in scenario 1, that Julia is loaded into memory twice, making it inefficient in that way. The same may be true for using multiple processes as in scenario 2? That makes scenario 3 or 4 the most appealing.

Before starting down this path, I would appreciate your collective expertise. A link to a tutorial would be really appreciated.

2

Probably that one, with Channels (Asynchronous Programming · The Julia Language).

3

Here is a simple example using duckdb of concurrent readers and writers in Julia.

1 Like