# Using @pywith for PyCall

**URL:** <https://discourse.julialang.org/t/using-pywith-for-pycall/68878>\
**Category:** General Usage\
**Tags:** pycall\
**Created:** [September 28, 2021, 2:14pm UTC](https://discourse.julialang.org/t/using-pywith-for-pycall/68878 "2021-09-28T14:14:00Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![rubaiyat](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rubaiyat/32/27423_2.png) [@rubaiyat](https://discourse.julialang.org/u/rubaiyat)\
**Post date:** [September 28, 2021, 2:14pm UTC](https://discourse.julialang.org/t/using-pywith-for-pycall/68878/1 "2021-09-28T14:14:00Z")

</div>

Hello everyone,

I’m trying to run a Python package named PyBLP from Julia using Pycall, and ran into an issue using @pywith.

All the PyBLP commands run just fine, however when it comes to using “parallel” through a “with” statement, I’m getting an error.

Here’s the first part of the code, which should run once you install PyBLP and uses PyBLP’s example datasets.

```julia
using PyCall
using Conda

pyblp = pyimport("pyblp")
pd = pyimport("pandas")
np = pyimport("numpy")

pyblp.options.digits = 2
pyblp.options.verbose = false
pyblp. __version__

product_data = pd.read_csv(pyblp.data.NEVO_PRODUCTS_LOCATION)
product_data.head()

logit_formulation = pyblp.Formulation("prices", absorb="C(product_ids)")
problem = pyblp.Problem(logit_formulation, product_data)
logit_results = problem.solve()
logit_results.compute_elasticities()

```

The very last command “compute\_elasticities()” can be implemented in parallel as per the PyBLP documentation. The exact Python command should be:

```julia
with pyblp.parallel(2):
elasticities = results.compute_elasticities()

```

I’m trying to find the corresponding code using Pycall but haven’t been successful. I first tried:

```julia

@pywith pyblp("parallel")("2") as f begin
f.(results = logit_results.compute_elasticities())
end

```

which gives:

```julia
ERROR: PyError ($(Expr(:escape, :(ccall(#= /home/rubaiyat/.julia/packages/PyCall/BD546/src/pyfncall.jl:43 =# @pysym(:PyObject_Call), PyPtr, (PyPtr, PyPtr, PyPtr), o, pyargsptr, kw))))) <class 'TypeError'>
TypeError("'module' object is not callable")

```

I also tried:

```julia

py""" 
	with pyblp.parallel(2) : elasticities = logit_results.compute_elasticities()
"""

```

which gives

```julia
PyError ($(Expr(:escape, :(ccall(#= /home/rubaiyat/.julia/packages/PyCall/BD546/src/pyeval.jl:34 =# @pysym(:Py_CompileString), PyPtr, (Cstring, Cstring, Cint), s, fname, input_type))))) <class 'IndentationError'>
IndentationError('unexpected indent', ('/home/rubaiyat/.julia/packages/PyCall/BD546/src/pyeval.jl', 2, 4, ' with pyblp.parallel(2) : elasticities = logit_results.compute_elasticities()\n'))

```

An additional question before wrapping up: if I were to start 2 parallel processes as above from PyCall, would I need to start Julia using 2 threads, or do I start Julia using 1 thread and then the python call would spawn multiple threads when I run the above command?

I’d appreciate any help on this, thanks!

---

<div class="post-metadata">

**Author:** ![rubaiyat](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rubaiyat/32/27423_2.png) [@rubaiyat](https://discourse.julialang.org/u/rubaiyat)\
**Post date:** [September 28, 2021, 10:22pm UTC](https://discourse.julialang.org/t/using-pywith-for-pycall/68878/2 "2021-09-28T22:22:35Z")

</div>

Update: couldn’t get the @pywith method to work but got a workaround using py" " " … " " ". I’ll post it here in case anyone finds it useful.

To run the elasticities in parallel:

```julia
py""" 
import pyblp as pyblp

def run_par(logit_results):
	with pyblp.parallel(2) : 
		elasticities = logit_results.compute_elasticities()
	
	return elasticities
"""

ans = py"run_par"(logit_results)

```

To solve the logit problem in parallel, this time with info on threads:

```julia

py""" 
import pyblp as pyblp

def run_par(problem):
	pyblp.options.verbose = True
	with pyblp.parallel(2) : 
		logit_results = problem.solve()
	
	return logit_results
"""

ans = py"run_par"(problem)

```

which gives:

```julia
Starting a pool of 2 processes ...
Started the process pool after 00:00:00.
Solving the problem ...
Updating the weighting matrix ...
Computed results after 00:00:00.

Problem Results Summary:
==========================================
GMM Objective Clipped Weighting Matrix
Step Value Shares Condition Number
---- --------- ------- ----------------
 1 +1.9E+02 0 +6.9E+07    
==========================================

Estimating standard errors ...
Computed results after 00:00:00.

Problem Results Summary:
==========================================
GMM Objective Clipped Weighting Matrix
Step Value Shares Condition Number
---- --------- ------- ----------------
 2 +1.9E+02 0 +5.7E+07    
==========================================

Cumulative Statistics:
========================
Computation Objective 
   Time Evaluations
----------- -----------
 00:00:00 2     
========================

Beta Estimates (Robust SEs in Parentheses):
==========
  prices  
----------
 -3.0E+01 
(+1.0E+00)
==========
Terminating the pool of 2 processes ...
Terminated the process pool after 00:00:00.

```

If anyone figures out a way with @pywith I’d be happy to see it!

It also looks like Julia doesn’t need to be started on multiple threads for pyblp.parallel to spawn multiple threads, as can be seen from the output of the code above. Let me know if I’m wrong on this one though.

---

<div class="post-metadata">

**Author:** ![marius311](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/marius311/32/3953_2.png) [@marius311](https://discourse.julialang.org/u/marius311)\
**Post date:** [September 29, 2021, 12:06am UTC](https://discourse.julialang.org/t/using-pywith-for-pycall/68878/3 "2021-09-29T00:06:08Z")

</div>

> [@rubaiyat](#):
>
> `pyblp("parallel")`

I think the issue may just be that this should be `pyblp.parallel` ? Otherwise you are indeed trying to call a module, as the error message suggests.

---

<div class="post-metadata">

**Author:** ![rubaiyat](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rubaiyat/32/27423_2.png) [@rubaiyat](https://discourse.julialang.org/u/rubaiyat)\
**Post date:** [September 29, 2021, 8:06pm UTC](https://discourse.julialang.org/t/using-pywith-for-pycall/68878/4 "2021-09-29T20:06:03Z")

</div>

Did you mean this?

```julia
@pywith pyblp.parallel("2") as f begin
f.(results = logit_results.compute_elasticities())
end

```

This throws the following error:

```julia
 @pywith pyblp.parallel("2") as f begin
       f.(results = logit_results.compute_elasticities())
       end
ERROR: PyError ($(Expr(:escape, :(ccall(#= /home/rubaiyat/.julia/packages/PyCall/BD546/src/pyfncall.jl:43 =# @pysym(:PyObject_Call), PyPtr, (PyPtr, PyPtr, PyPtr), o, pyargsptr, kw))))) <class 'TypeError'>
TypeError('processes must be an int.')
  File "/home/rubaiyat/.julia/conda/3/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/home/rubaiyat/.julia/conda/3/lib/python3.8/site-packages/pyblp/utilities/basics.py", line 69, in parallel
    raise TypeError("processes must be an int.")

```

I also tried the following:

```julia
@pywith pyblp.parallel(2) as f begin
f.(results = logit_results.compute_elasticities())
end

```

which throws this error:

```julia
ERROR: MethodError: objects of type Nothing are not callable
Stacktrace:
 [1] top-level scope
   @ ~/.julia/packages/PyCall/BD546/src/PyCall.jl:656

```

Maybe there’s a way of getting around the “processes must be an int” error (writing `pyblp.parallel(Int(2))` didn’t work though).

---

<div class="post-metadata">

**Author:** ![marius311](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/marius311/32/3953_2.png) [@marius311](https://discourse.julialang.org/u/marius311)\
**Post date:** [September 29, 2021, 8:36pm UTC](https://discourse.julialang.org/t/using-pywith-for-pycall/68878/5 "2021-09-29T20:36:03Z")

</div>

> [@rubaiyat](#):
>
> ```julia
> pyblp.parallel(2)
> 
> ```

Yea I think this is the right thing. Remaining error might be due to `f.(results = logit_results.compute_elasticities())` which actually parses as calling `f` with a keyword argument. Not sure if that’s exactly what you were going for.

---

<div class="post-metadata">

**Author:** ![rubaiyat](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rubaiyat/32/27423_2.png) [@rubaiyat](https://discourse.julialang.org/u/rubaiyat)\
**Post date:** [October 3, 2021, 12:19am UTC](https://discourse.julialang.org/t/using-pywith-for-pycall/68878/6 "2021-10-03T00:19:38Z")

</div>

You’re right, the following code works:

```julia
@pywith pyblp.parallel(4) as f begin
		pyblp.options.verbose = true
		logit_results = problem.solve()
		return logit_results
end

```

And for whatever reason, in my research setting, the method using `py """ ... """ ` was causing problems whereas the @pywith method shown here works fine.

Thanks!