Using @pywith for PyCall

General Usage
1

Hello everyone,

I’m trying to run a Python package named PyBLP from Julia using Pycall, and ran into an issue using @pywith.

All the PyBLP commands run just fine, however when it comes to using “parallel” through a “with” statement, I’m getting an error.

Here’s the first part of the code, which should run once you install PyBLP and uses PyBLP’s example datasets.

using PyCall
using Conda

pyblp = pyimport("pyblp")
pd = pyimport("pandas")
np = pyimport("numpy")

pyblp.options.digits = 2
pyblp.options.verbose = false
pyblp.__version__

product_data = pd.read_csv(pyblp.data.NEVO_PRODUCTS_LOCATION)
product_data.head()

logit_formulation = pyblp.Formulation("prices", absorb="C(product_ids)")
problem = pyblp.Problem(logit_formulation, product_data)
logit_results = problem.solve()
logit_results.compute_elasticities()

The very last command “compute_elasticities()” can be implemented in parallel as per the PyBLP documentation. The exact Python command should be:

with pyblp.parallel(2):
elasticities = results.compute_elasticities()

I’m trying to find the corresponding code using Pycall but haven’t been successful. I first tried:


@pywith pyblp("parallel")("2") as f begin
f.(results = logit_results.compute_elasticities())
end

which gives:

ERROR: PyError ($(Expr(:escape, :(ccall(#= /home/rubaiyat/.julia/packages/PyCall/BD546/src/pyfncall.jl:43 =# @pysym(:PyObject_Call), PyPtr, (PyPtr, PyPtr, PyPtr), o, pyargsptr, kw))))) <class 'TypeError'>
TypeError("'module' object is not callable")

I also tried:


py""" 
	with pyblp.parallel(2) : elasticities = logit_results.compute_elasticities()
"""

which gives

PyError ($(Expr(:escape, :(ccall(#= /home/rubaiyat/.julia/packages/PyCall/BD546/src/pyeval.jl:34 =# @pysym(:Py_CompileString), PyPtr, (Cstring, Cstring, Cint), s, fname, input_type))))) <class 'IndentationError'>
IndentationError('unexpected indent', ('/home/rubaiyat/.julia/packages/PyCall/BD546/src/pyeval.jl', 2, 4, '    with pyblp.parallel(2) : elasticities = logit_results.compute_elasticities()\n'))

An additional question before wrapping up: if I were to start 2 parallel processes as above from PyCall, would I need to start Julia using 2 threads, or do I start Julia using 1 thread and then the python call would spawn multiple threads when I run the above command?

I’d appreciate any help on this, thanks!

2

Update: couldn’t get the @pywith method to work but got a workaround using py" " " … " " ". I’ll post it here in case anyone finds it useful.

To run the elasticities in parallel:

py""" 
import pyblp as pyblp

def run_par(logit_results):
	with pyblp.parallel(2) : 
		elasticities = logit_results.compute_elasticities()
	
	return elasticities
"""

ans = py"run_par"(logit_results)

To solve the logit problem in parallel, this time with info on threads:


py""" 
import pyblp as pyblp

def run_par(problem):
	pyblp.options.verbose = True
	with pyblp.parallel(2) : 
		logit_results = problem.solve()
	
	return logit_results
"""

ans = py"run_par"(problem)

which gives:

Starting a pool of 2 processes ...
Started the process pool after 00:00:00.
Solving the problem ...
Updating the weighting matrix ...
Computed results after 00:00:00.

Problem Results Summary:
==========================================
GMM   Objective  Clipped  Weighting Matrix
Step    Value    Shares   Condition Number
----  ---------  -------  ----------------
 1    +1.9E+02      0         +6.9E+07    
==========================================

Estimating standard errors ...
Computed results after 00:00:00.

Problem Results Summary:
==========================================
GMM   Objective  Clipped  Weighting Matrix
Step    Value    Shares   Condition Number
----  ---------  -------  ----------------
 2    +1.9E+02      0         +5.7E+07    
==========================================

Cumulative Statistics:
========================
Computation   Objective 
   Time      Evaluations
-----------  -----------
 00:00:00         2     
========================

Beta Estimates (Robust SEs in Parentheses):
==========
  prices  
----------
 -3.0E+01 
(+1.0E+00)
==========
Terminating the pool of 2 processes ...
Terminated the process pool after 00:00:00.

If anyone figures out a way with @pywith I’d be happy to see it!

It also looks like Julia doesn’t need to be started on multiple threads for pyblp.parallel to spawn multiple threads, as can be seen from the output of the code above. Let me know if I’m wrong on this one though.

3

I think the issue may just be that this should be pyblp.parallel ? Otherwise you are indeed trying to call a module, as the error message suggests.

4

Did you mean this?

@pywith pyblp.parallel("2") as f begin
f.(results = logit_results.compute_elasticities())
end

This throws the following error:

 @pywith pyblp.parallel("2") as f begin
       f.(results = logit_results.compute_elasticities())
       end
ERROR: PyError ($(Expr(:escape, :(ccall(#= /home/rubaiyat/.julia/packages/PyCall/BD546/src/pyfncall.jl:43 =# @pysym(:PyObject_Call), PyPtr, (PyPtr, PyPtr, PyPtr), o, pyargsptr, kw))))) <class 'TypeError'>
TypeError('processes must be an int.')
  File "/home/rubaiyat/.julia/conda/3/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/home/rubaiyat/.julia/conda/3/lib/python3.8/site-packages/pyblp/utilities/basics.py", line 69, in parallel
    raise TypeError("processes must be an int.")

I also tried the following:

@pywith pyblp.parallel(2) as f begin
f.(results = logit_results.compute_elasticities())
end

which throws this error:

ERROR: MethodError: objects of type Nothing are not callable
Stacktrace:
 [1] top-level scope
   @ ~/.julia/packages/PyCall/BD546/src/PyCall.jl:656

Maybe there’s a way of getting around the “processes must be an int” error (writing pyblp.parallel(Int(2)) didn’t work though).

5

Yea I think this is the right thing. Remaining error might be due to f.(results = logit_results.compute_elasticities()) which actually parses as calling f with a keyword argument. Not sure if that’s exactly what you were going for.

1 Like
6

You’re right, the following code works:

@pywith pyblp.parallel(4) as f begin
		pyblp.options.verbose = true
		logit_results = problem.solve()
		return logit_results
end

And for whatever reason, in my research setting, the method using py """ ... """ was causing problems whereas the @pywith method shown here works fine.

Thanks!