# Can we import \`dill\` in PyCall rather than pickle?

**URL:** <https://discourse.julialang.org/t/can-we-import-dill-in-pycall-rather-than-pickle/32299>\
**Category:** General Usage\
**Tags:** pycall\
**Created:** [December 15, 2019, 12:20pm UTC](https://discourse.julialang.org/t/can-we-import-dill-in-pycall-rather-than-pickle/32299 "2019-12-15T12:20:44Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![innerlee](https://avatars.discourse-cdn.com/v4/letter/i/b19c9b/32.png) [@innerlee](https://discourse.julialang.org/u/innerlee)\
**Post date:** [December 15, 2019, 12:20pm UTC](https://discourse.julialang.org/t/can-we-import-dill-in-pycall-rather-than-pickle/32299/1 "2019-12-15T12:20:44Z")

</div>

The current version of PyCall, and when python3 is used, produces error in pmap:

```julia
julia> using Distributed

julia> addprocs(2);

julia> @everywhere using PyCall
julia> @everywhere math=pyimport("math")

julia> a = [1,2];
julia> pmap(x->math.sin(x), a)
ERROR: PyError ($(Expr(:escape, :(ccall(#= /home/lizz/.julia/packages/PyCall/ttONZ/src/pyfncall.jl:44 =# @pysym(:PyObject_Call), PyPtr, (PyPtr,
 PyPtr, PyPtr), o, pyargsptr, kw))))) <class 'TypeError'>
TypeError("can't pickle module objects")

```

This, of course, is because the use of `pickle` package in [this line](https://github.com/JuliaPy/PyCall.jl/blob/master/src/serialize.jl#L7). Can we replace it with `dill`? I have verified that dill resolves the issue.

The only concern is the license issue. Can we import 3-clause BSD licensed python packages like:

```julia
pickle() = ispynull(_pickle) ? copy!(_pickle, pyimport(PyCall.pyversion.major ≥ 3 ? "dill" : "cPickle")) : _pickle

```

?

---

<div class="post-metadata">

**Author:** ![innerlee](https://avatars.discourse-cdn.com/v4/letter/i/b19c9b/32.png) [@innerlee](https://discourse.julialang.org/u/innerlee)\
**Post date:** [December 15, 2019, 2:31pm UTC](https://discourse.julialang.org/t/can-we-import-dill-in-pycall-rather-than-pickle/32299/2 "2019-12-15T14:31:31Z")

</div>

related pr: [https://github.com/JuliaPy/PyCall.jl/pull/731](https://github.com/JuliaPy/PyCall.jl/pull/731)

---

<div class="post-metadata">

**Author:** ![tkf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkf/32/17635_2.png) [@tkf](https://discourse.julialang.org/u/tkf)\
**Post date:** [December 15, 2019, 3:09pm UTC](https://discourse.julialang.org/t/can-we-import-dill-in-pycall-rather-than-pickle/32299/3 "2019-12-15T15:09:49Z")

</div>

Why not `pyimport` in the anonymous function; e.g., `pmap(x -> pyimport("math").sin(x), a)`? There are other ways to workaround this. But I don’t think customizing serialization is the right way to do it.

---

<div class="post-metadata">

**Author:** ![innerlee](https://avatars.discourse-cdn.com/v4/letter/i/b19c9b/32.png) [@innerlee](https://discourse.julialang.org/u/innerlee)\
**Post date:** [December 15, 2019, 3:23pm UTC](https://discourse.julialang.org/t/can-we-import-dill-in-pycall-rather-than-pickle/32299/4 "2019-12-15T15:23:12Z")

</div>

import pkg for each iteration doesn’t feels right to me

---

<div class="post-metadata">

**Author:** ![tkf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkf/32/17635_2.png) [@tkf](https://discourse.julialang.org/u/tkf)\
**Post date:** [December 15, 2019, 3:38pm UTC](https://discourse.julialang.org/t/can-we-import-dill-in-pycall-rather-than-pickle/32299/5 "2019-12-15T15:38:34Z")

</div>

Python modules are cached so it’s a very cheap operation. I’d just use it untill it turned out to be a bottleneck after benchmarking and profiling. Having said that, there are other solutions:

- `@everywhere pysin(x) = math.sin(x)` then `pmap(pysin, a)`
- You can `copy!` the module `math` or the function `math.sin` to a constant as mentioned in PyCall README. This can be used as per-process memoization or in ` __init__ `.

---

<div class="post-metadata">

**Author:** ![innerlee](https://avatars.discourse-cdn.com/v4/letter/i/b19c9b/32.png) [@innerlee](https://discourse.julialang.org/u/innerlee)\
**Post date:** [December 15, 2019, 4:39pm UTC](https://discourse.julialang.org/t/can-we-import-dill-in-pycall-rather-than-pickle/32299/6 "2019-12-15T16:39:38Z")

</div>

```julia
julia> using Distributed

julia> addprocs(2);

julia> @everywhere using PyCall

julia> @everywhere const math = PyNULL()

julia> @everywhere copy!(math, pyimport("math"))

julia> pmap(x->math.sin(x), [1,2])
ERROR: PyError ($(Expr(:escape, :(ccall(#= /home/lizz/.julia/packages/PyCall/ttONZ/src/pyfncall.jl:44 =# @pysym(:PyObject_Call), PyPtr, (PyPtr, PyPtr, PyPtr), o, pyargsptr, kw))))) <class 'TypeError'>
TypeError("can't pickle module objects")

```

---

<div class="post-metadata">

**Author:** ![innerlee](https://avatars.discourse-cdn.com/v4/letter/i/b19c9b/32.png) [@innerlee](https://discourse.julialang.org/u/innerlee)\
**Post date:** [December 15, 2019, 4:43pm UTC](https://discourse.julialang.org/t/can-we-import-dill-in-pycall-rather-than-pickle/32299/7 "2019-12-15T16:43:44Z")

</div>

It seems that the reliable solution is wrap each module into a function.

```julia
julia> using Distributed

julia> addprocs(2);

julia> @everywhere using PyCall

julia> @everywhere Image() = pyimport("PIL.Image")

julia> pmap(["y0.jpg", "y1.jpg"]) do b
       img = Image().open(b)
       println(img.size)
       img.close()
       end;
      From worker 2: (112, 112)
      From worker 3: (112, 112)

```

---

<div class="post-metadata">

**Author:** ![tkf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkf/32/17635_2.png) [@tkf](https://discourse.julialang.org/u/tkf)\
**Post date:** [December 15, 2019, 4:52pm UTC](https://discourse.julialang.org/t/can-we-import-dill-in-pycall-rather-than-pickle/32299/8 "2019-12-15T16:52:24Z")

</div>

If you are OK with `Image().open(b)` you should be OK with `pyimport("PIL.Image").open(b)` (performance-wise).

Your code with `copy!` didn’t work because you are not using memoization/caching pattern. You need to call `copy!` inside the function you passed to `pmap`, guarded by `ispynull`.

---

<div class="post-metadata">

**Author:** ![innerlee](https://avatars.discourse-cdn.com/v4/letter/i/b19c9b/32.png) [@innerlee](https://discourse.julialang.org/u/innerlee)\
**Post date:** [December 15, 2019, 4:59pm UTC](https://discourse.julialang.org/t/can-we-import-dill-in-pycall-rather-than-pickle/32299/9 "2019-12-15T16:59:01Z")

</div>

Performance-wise I’m okay with it because in my case, processing images are very slow operations. If this is the right way to do, then I will accept it because it requires minimum changes.

Thanks for the tips!