Embed python packages into julia

General Usage
1

Hi, I have a problem with loading python package dragonfly into julia and call it, don’t know why, since with the same way it works for another case here
https://discourse.julialang.org/t/call-python-packages-from-julia/75324

Here is my example

function black_box_function(; x)
    return -x^2 - x + 1
end

using PyCall

temp = pyimport("dragonfly")
max_val, max_pt, history = temp.minimise_function(black_box_function, py"[[-10,10]]", 20)
print(max_val, max_pt)

For comparison, here put the program written in python, it works well

from dragonfly import minimise_function

def func(x):
    return -x**2 - x + 1

min_val, min_pt, history = minimise_function(func, [[-10, 10]], 20)
print(min_val, min_pt)
2

What is the error message?
Which operating system do you use?
How did you install Python?
How did you install dragonfly?

1 Like
3

I use windows system, and use pip to install dragonfly, the latest version of python is installed in my computer and I use VSCode to use it, I think I already install both python and dragonfly successfully, because I can run it within python environment, the error message is as follows, althouth there are some warnings, that shouldn’t be the problem, I think

C:\Users\admin\AppData\Local\Programs\Python\Python310\lib\site-packages\dragonfly\utils\oper_utils.py:30: UserWarning: cannot_fortran' (C:\Users\admin\AppData\Local\Programs\Python\Python310\lib\site-packages\dragonfly\utils\direct_fortran\__init__.
Could not import Fortran direct library. Dragonfly can still be used, but might be slightly slower. To get rid of this warning. gfortran) and the python-dev package and reinstall Dragonfly.
  warn('%s\n%s'%(e, fortran_err_msg))
┌ Warning: `vendor()` is deprecated, use `BLAS.get_config()` and inspect the output instead
│   caller = npyinitialize() at numpy.jl:67
└ @ PyCall C:\Users\admin\.julia\packages\PyCall\L0fLP\src\numpy.jl:67
ERROR: LoadError: PyCall.PyError("\$(Expr(:escape, :(ccall(#= C:\\Users\\admin\\.julia\\packages\\PyCall\\L0fLP\\src\\pyfncal
PyPtr, PyPtr), o, pyargsptr, kw))))", PyObject(Ptr{PyCall.PyObject_struct} @0x00007ffb31525890), PyObject(Ptr{PyCall.PyObject.PyObject_struct} @0x000000006a402080))
Stacktrace:      
  [1] pyerr_check
    @ C:\Users\admin\.julia\packages\PyCall\L0fLP\src\exception.jl:62 [inlined]
  [2] pyerr_check
    @ C:\Users\admin\.julia\packages\PyCall\L0fLP\src\exception.jl:66 [inlined]
  [4] macro expansion
    @ C:\Users\admin\.julia\packages\PyCall\L0fLP\src\exception.jl:97 [inlined]
  [5] #107
    @ C:\Users\admin\.julia\packages\PyCall\L0fLP\src\pyfncall.jl:43 [inlined]
  [6] disable_sigint
    @ .\c.jl:458 [inlined]
  [7] __pycall!
    @ C:\Users\admin\.julia\packages\PyCall\L0fLP\src\pyfncall.jl:42 [inlined]
  [8] _pycall!(ret::PyObject, o::PyObject, args::Tuple{typeof(black_box_function), Matrix{Int64}, Int64}, nargs::Int64, kw::Ptr{Nothing})
    @ PyCall C:\Users\admin\.julia\packages\PyCall\L0fLP\src\pyfncall.jl:29
  [9] _pycall!(ret::PyObject, o::PyObject, args::Tuple{typeof(black_box_function), Matrix{Int64}, Int64}, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ PyCall C:\Users\admin\.julia\packages\PyCall\L0fLP\src\pyfncall.jl:11
 [10] (::PyObject)(::Function, ::Vararg{Any}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ PyCall C:\Users\admin\.julia\packages\PyCall\L0fLP\src\pyfncall.jl:86
 [11] (::PyObject)(::Function, ::Vararg{Any})
    @ PyCall C:\Users\admin\.julia\packages\PyCall\L0fLP\src\pyfncall.jl:86
 [12] top-level scope
    @ C:\Users\admin\.julia\packages\PyCall\L0fLP\src\pyeval.jl:232
in expression starting at C:\Users\admin\Desktop\Bayesian Optimization\Python\test07.jl:8
4

Well, Python on Windows is problematic, and using pip to install packages is problematic…
Conda used to be better than pip, but there are also newer developments like GitHub - cjdoris/PythonCall.jl: Python and Julia in harmony. which keeps all Python packages separated per Julia environment.

And I see that you have a space in your folder name: “C:\Users\admin\Desktop\Bayesian Optimization\Python\test07.jl:8”.

Can you do the same test when using a folder without any space in the name?

2 Likes
5

+1 for PythonCall.jl. It just makes life so much easier.

Install:

(dragonfly) pkg> add CondaPkg, PythonCall
julia> using CondaPkg
(dragonfly) pkg> conda add numpy
(dragonfly) pkg> conda add --pip dragonfly-opt
(dragonfly) pkg> conda resolve

Run:

julia> using PythonCall

julia> @pyexec("""
       global my_min, minimise_function, func
       from dragonfly import minimise_function

       def func(x):
           return -x**2 - x + 1
           
       def my_min(domain, max_capital):
           return minimise_function(func, domain, max_capital)
       """)

julia> my_min(domain, max_capital) = @pyeval("my_min")(domain, max_capital)
my_min (generic function with 1 method)

julia> min_val, min_pt, history = my_min(pylist([[-10, 10]]), 20);
Optimisation with bo(ei-ucb-ttei-add_ucb) using capital 20.0 (return_value)
Capital spent on initialisation: 5.0000(0.2500).
Legend: <iteration_number> (<num_successful_queries>, <fraction_of_capital_spent>):: curr_max=<current_maximum_value>, acqs=<num_times_each_acquisition_was_used>
#013 (011, 0.550):: curr_max=109.00000, acqs=[ei:1, ucb:2, ttei:0, add_ucb:3], 
#022 (021, 1.050):: curr_max=109.00000, acqs=[ei:2, ucb:4, ttei:2, add_ucb:8],

julia> @info :Result min_val min_pt
┌ Info: Result
│   min_val = Python float: -108.99999999999993
└   min_pt = Python ndarray: array([10.])

You can also work with the python function directly:

julia> @py import dragonfly: minimise_function

julia> minimise_function
Python function: <function minimise_function at 0x7fe59d1745e0>

but that goes beyond my knowledge of how this dragonfly package works, and mixing julia and python objects can get a bit more dicey than just wrapping everything into a julia function

Edit: wow

julia> using PythonCall

julia> @py import dragonfly: minimise_function

julia> func(x) = -x[1]^2 - x[1] + 1
func (generic function with 1 method)

julia> min_val, min_pt, history = minimise_function(func, pylist([[-10, 10]]), 20);

julia> @info :Result min_val min_pt
┌ Info: Result
│   min_val = Python float: -108.99999999999993
└   min_pt = Python ndarray: array([10.])
2 Likes