How to reuse object in multiple parallel calls instead of making copies in each call

Specific Domains Julia at Scale
1

I’m using pmap() to make a large number (~12000) of parallel calls to perform power flow computations based on the pseudocode below:

'''
PD, QD are K x N matrices where N ~= 12000 and K = number of loads in current case
network_data is a dictionary of dictionaries parsed from a .m case file
PD, QD, network_data are sent to all worker processes with ParallelDataTransfer.jl and are read only
'''
@everywhere function compute_pf(i)
    ndata = deepcopy(network_data)  # we want to write to this, so need to make a deep copy
    update all ndata's pd, qd fields with i-th column of PD, QD matrices  
    do ac_pf and dc_pf with PowerModels, JuMP, and Ipopt
    return ac_pf and dc_pf solution dicts and solution time as an Array{Any} of length 4
end 

function main()
    N = 12000
    ret = pmap(compute_pf, 1:N)  # do things with ret
end

My question is, is it possible to reuse the ndata dictionary we deepcopied from network_data, the next time compute_pf() runs on a different i? If that’s possible, each worker process would make only 1 deep copy at the beginning, instead of ~12000 or however many calls I run. I think this is the main bottleneck, unless I decide to rewrite the whole ac_pf and dc_pf functions myself specifically optimized for this purpose. From JuMP: reuse solver objects? it seems that the solver objects won’t have much impact, so that really only leaves the deepcopy part. Any help here would be appreciated.

@ccoffrin I don’t know if this was already mentioned by someone or if it’s on any future release agenda, but would it be possible to provide options so that for basic power flow computations, we don’t need to pass in the entire network_data dictionary? That could lower the memory requirements. Thank you!