Package works in regular Julia, but generates error when called using PyJulia -- relative path

General Usage
1

Hey Folks, I was trying to figure out how to solve this this tricky issue. So I developed a package on my local machine and it works just fine. I have a CSV file saved in the package that I use to store some of the ground truth data. Once I got the package working in its own local environment, I was able to install the package to my main Julia 1.4 installation using ] add http://github... So that installed the package to my .julia folder. I tested this install, and it also works just fine–it returns a julia DataFrame as expected.

Finally, I tried to call the Julia package from python using Pyjulia. Now I am able to import julia and import a julia file that calls my package. However, now when I try and run the Julia package, I am getting an error about the import of the CSV file–which is located in the same src directory for the package. So it might be that Pyjulia is starting from a different starting point and that is throwing off the installation or something?

I was trying to figure out how to debug an issue like this? I was not able to set a breakpoint for my package in the ~/.julia folder. Seems like that code gets compiled on install or something. So how would I figure out how to resolve an issue like this. Any help would be appreciated. @tkf if you have any suggestions, please pass them along.

2

So you have a Julia package and its Git repository contains a CSV file? How do you get the path of the CSV file? Do you make sure that it doesn’t depend on the current working directory by using joinpath(@__DIR__, "relative", "path", "to", "file.csv")?

3

@tkf I did not try that, but seems like a better way to go. I was just loading it from the relative path of the module. The code worked when the package was loaded locally, or even when I installed from github to my local ~/.julia folder. It only broke when using PyCall. I have a julia file that calls a function from that package, and then I have a python function that calls the Main.<julia_function>, and that is where it broke. I can try what you suggested though, and see if that works. Thanks for the tip, I would never have guessed that.

4

Maybe you can call pwd() in Julia and PyJulia. It’ll clarify the difference. Anyway, I think we need more information to solve this, if my random guess is incorrect. For example, sharing backtrace can be very helpful sometimes.

2 Likes
5

@tkf I use joinpath() as you suggested, and that worked great. So I will use that from now on. At least now I am getting the code to run and it returns a dataframe in raw form :). I am now trying to use the Pandas package to get the dataframe to return a Dataframe to python, but I am getting a conversion error.

I don’t mean to pull you into debugging my code, but I can post the stacktrace in case you can quickly see what the problem is. To my simple eyes it seems like there is an error when converting to Pandas where the convert method for ::Type{Union{}} seems to have trouble converting to float. Does the way that this error is written suggest that the Union{} is ambiguous or Any since there are no types specified within the Union? NOTE, there are some text fields in the Julia Dataframe–since I needed to remind myself what I did. There are missing values in many of the columns of the Julia DataFrame, and that did not seem to cause any issues when computing on the Julia side–only when doing the Pandas.jl conversion of the Julia Dataframe.

Traceback (most recent call last):
  File "test_pythonfile.py", line 50, in <module>
    a = Main.run_julia_model(initial_values, 10, 20)
RuntimeError: Julia exception: MethodError: convert(::Type{Union{}}, ::Float64) is ambiguous. Candidates:
  convert(::Type{Union{}}, x) in Base at essentials.jl:169
  convert(::Type{T}, x::Number) where T<:Number in Base at number.jl:7
  convert(::Type{T}, arg) where T<:VecElement in Base at baseext.jl:8
  convert(::Type{T}, x::Number) where T<:AbstractChar in Base at char.jl:179
Possible fix, define
  convert(::Type{Union{}}, ::Number)
Stacktrace:
 [1] setindex!(::Array{Union{},1}, ::Float64, ::Int64) at ./array.jl:825
 [2] _construct_pandas_from_iterabletable(::DataFrame) at /home/krishnab/.julia/packages/Pandas/rAPmB/src/tabletraits.jl:37
 [3] DataFrame at /home/krishnab/.julia/packages/Pandas/rAPmB/src/Pandas.jl:457 [inlined]
 [4] run_julia_model(::Dict{Any,Any}, ::Int64, ::Int64) at /media/krishnab/lakshmi/sandbox/julia/pyjulia/test_julia.jl:6
 [5] invokelatest(::Any, ::Any, ::Vararg{Any,N} where N; kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at ./essentials.jl:712
 [6] invokelatest(::Any, ::Any, ::Vararg{Any,N} where N) at ./essentials.jl:711
 [7] _pyjlwrap_call(::Function, ::Ptr{PyCall.PyObject_struct}, ::Ptr{PyCall.PyObject_struct}) at /home/krishnab/.julia/packages/PyCall/zqDXB/src/callback.jl:28
 [8] pyjlwrap_call(::Ptr{PyCall.PyObject_struct}, ::Ptr{PyCall.PyObject_struct}, ::Ptr{PyCall.PyObject_struct}) at /home/krishnab/.julia/packages/PyCall/zqDXB/src/callback.jl:49

Here are the original Julia file and Python files.

#test_julia.jl
using DemoInertia
using DataFrames
import Pandas

function run_julia_model(iv::Dict, runs, duration)
    Pandas.DataFrame(DemoInertia.get_summaries(iv, runs, duration))
end

The python code. Note that I abbreviated the initial_values dictionary, as it is actually larger.

# test_python.py
import julia
jl = julia.Julia(compiled_modules=False)
jl.include('test_julia.jl')  # my file with functions
from julia import Main

initial_values = {"a": 1,
            "b" : "text",
            "c" : 3.2}

a = Main.run_julia_model(initial_values, 10, 20)
print(a)