One of the limitations of using python for distributed computing is how well REPL state can be propagated across a cluster. If you define a bunch of functions in the REPL that call each other that you then want to execute remotely, there isn’t really a robust way to figure out how they depend on one another and send the subset needed to execute a given request. If you have a run_on_cluster(foo)
call frameworks may be smart enough to serialize the code for foo
but not necessarily for everything that it in turn calls. Does Julia handle this any better? If so what internal APIs are being used?