Thanks, that’s a great idea! Moving objects around in Julia is much nicer than the clunkier
execnet, so its an overall win.
A down-side is that PyCall has to be built/precompiled each time we launch, which adds ~10 seconds to startup. Certainly would be great if two built versions could be stored separately, but for now its a fine trade-off for me. Here’s a first attempt that basically works:
id_py2worker, = addprocs(1, restrict=true)
# launch our Python 2 worker and build PyCall with Python 2
@everywhere id_py2worker begin
ENV["PYTHON"] = "python2"
# in background, rebuild PyCall back to the original version (the py2worker has already
# loaded Python 2, so that will stick)
ENV["PYTHON"] = orig_python
end, id_py2worker, PyCall.python)
The we can check its all working:
julia> @fetchfrom id_py2worker PyCall.pyversion
julia> @fetchfrom id_py2worker py"""
julia> @fetchfrom id_py2worker py"sys.version"
"2.7.16 (default, Apr 6 2019, 01:42:57) \n[GCC 8.3.0]"
One problem is that if Revise is already loaded on the main process, building back the original Python will cause the Python 2 workers to update and in fact segfault. I can’t figure out how to stop that from happening. (This issue could be one solution)