# PythonCall.jl with CondaPkg.jl in MPI scenario

**URL:** <https://discourse.julialang.org/t/pythoncall-jl-with-condapkg-jl-in-mpi-scenario/121191>\
**Category:** Performance\
**Tags:** question, package, mpi, pythoncall, condapkg\
**Created:** [October 11, 2024, 11:50am UTC](https://discourse.julialang.org/t/pythoncall-jl-with-condapkg-jl-in-mpi-scenario/121191 "2024-10-11T11:50:22Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![J-C-Q](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/j-c-q/32/210847_2.png) [@J-C-Q](https://discourse.julialang.org/u/J-C-Q)\
**Post date:** [October 11, 2024, 11:50am UTC](https://discourse.julialang.org/t/pythoncall-jl-with-condapkg-jl-in-mpi-scenario/121191/1 "2024-10-11T11:50:23Z")

</div>

Hi everyone,

I’m working on a julia package that has PythonCall.jl and CondaPkg.jl in its dependencies.

Now, I want to do some calculations using my package on a cluster with MPI.jl. The problem is (this is my understanding) that whenever I use my package, CondaPkg.jl has to resolve the environment, which cannot be done in parallel (the resolve function in CondaPkg.jl explicitly creates a lock file, so I get a lot of

```julia
Info: CondaPkg: Waiting for lock to be freed. You may delete this file if no other process is resolving.

```

).  
This means that for a large world size, my compute job times out because it takes too long to resolve on every rank.

I know however that nothing changed in the conda environment since I last called resolve, so in theory (in my head) it should be possible to skip this part in this case.

Does someone have an idea how I can solve this problem?  
Thank you in advance 🙂

Here is a link to the CondaPkg.jl resolve function: [https://github.com/JuliaPy/CondaPkg.jl/blob/c0ee1ecac08f9281276fd759e32663f0dee93213/src/resolve.jl#L505C10-L505C17](https://github.com/JuliaPy/CondaPkg.jl/blob/c0ee1ecac08f9281276fd759e32663f0dee93213/src/resolve.jl#L505C10-L505C17)

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [October 11, 2024, 12:26pm UTC](https://discourse.julialang.org/t/pythoncall-jl-with-condapkg-jl-in-mpi-scenario/121191/2 "2024-10-11T12:26:25Z")

</div>

See also [CondaPkg + PythonCall does not behave nicely on read-only filesystems · Issue #142 · JuliaPy/CondaPkg.jl · GitHub](https://github.com/JuliaPy/CondaPkg.jl/issues/142)

It looks like there is no option to disable the lockfile, but the author is open to a PR to add that feature.

Alternatively, you can simply manage your own Python installation and [point PythonCall to that](https://juliapy.github.io/PythonCall.jl/dev/pythoncall/#pythoncall-config). That’s what I would recommend for now in a cluster environment.

---

<div class="post-metadata">

**Author:** ![J-C-Q](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/j-c-q/32/210847_2.png) [@J-C-Q](https://discourse.julialang.org/u/J-C-Q)\
**Post date:** [October 11, 2024, 12:37pm UTC](https://discourse.julialang.org/t/pythoncall-jl-with-condapkg-jl-in-mpi-scenario/121191/3 "2024-10-11T12:37:31Z")

</div>

Thanks a lot! The issue that you pointed out essentially boils down to the same problem I’m experiencing. I guess I will try my luck on a PR.

Thanks again!