How to interact with a Julia process inside a Julia process

I want to measure compilation and TTFP latencies. So my idea was to run different scripts in fresh Julia processes which I start from a main Julia process so that each time Julia has to reliably compile everything anew.

There are two problems:

  • How can I keep a Julia process open and run separate chunks of code in it synchronously? For example, first I would run @timed using Package, then do something with the result in the main Julia process, then execute the next block that makes further use of Package.

I tried something like

open(`julia`, "w", some_iobuffer) do process
    println(process, some_commands)
    println(process, some_more_commands)

The problem is that println(process, ... is asynchronous, so the iobuffer doesn’t contain the expected written out data afterwards. That leads me to the second question:

  • How can I send data from the child process to the main process? In practice that would just be a couple of NamedTuples without any special other types embedded inside of them. Currently, because the iobuffer method doesn’t work, I’m writing to a csv file, but that seems clunky.

Would it have to be a separate julia process? (more so than explicitly adding a worker process)

julia> using Distributed

julia> @time using DataFrames
  1.043179 seconds (1.73 M allocations: 118.945 MiB, 2.35% gc time, 0.50% compilation time)

julia> w = addprocs(1)
1-element Vector{Int64}:

julia> @time @everywhere 2 begin; @time using DataFrames; end
      From worker 2:	  0.933681 seconds (1.52 M allocations: 106.554 MiB, 6.33% gc time, 0.51% compilation time)
  1.405225 seconds (91 allocations: 4.406 KiB)

returning results would be as easy as

timing_result = remotecall_fetch(2) do # arg is the worker id
    return remote_global_variables

Ah that’s interesting, I didn’t think of using Distributed. Well my goals are just that the process has to compile the packages used completely from scratch, and that it uses a different environment than the calling code. I’ll try it out!