Julia weird behavior with array passing between Julia and Python

General Usage
1

Hi,

I work on a project where users can use numerical solvers in a cross-language manner.

In this project, arrays are passed as pointers to the actual data and re-wrapped in different languages according to their shapes (that is, in Python a NumPy array is built, for Julia native array is built, so on). All this changes happen using Python C API and Julia C API, without using any packages for conversion.

Something has happened after upgrading from Julia 1.11.8 to 1.12.4, and arrays are not passed correctly anymore because the dimensions on the Julia size somehow are not set correctly.
I mean, I have not changed anything in my code: this error occurs only due to the change in the used Julia version.

I change only the Julia version using juliaup add ... and juliaup default ... commands.
Then I remove the built code completely (make clean) and build it again (make debug).

With Julia 1.11.8:

ipdb> p y.shape
(1,)

which is correct, as I am passing a 1D array with only one element.

With Julia 1.12.0:

ipdb> p y.shape
(135881294944352,)

Running the second time, I get a different "shape": `(130044954482816,)`, which kinda gives me a hint that somehow the array structure has changed.

With julia 1.12.4:

ipdb> p y.shape
(127460310906272,)

and next time it is again a random number: `(130742402242656,)`.

Could somebody explain me why this happens? My code is the same, I just recompile it.
Python is obviously also the same, C compiler as well. The only thing that changes here is Julia's version. I cannot see any changes in the changelog for Julia 1.12 that there were some changes in the array structure.

Thanks!
2

How exactly are you passing the array and its size? Are you looking at the Array .size field? How are you accessing that? Are you sure it’s an Array?

1 Like
3

This is how I pass a Julia array back to Python:

module CallbackWrapper
export make_wrapper_over_c_callback

import SciMLBase

using OpenInterfaces: OIFArrayF64, OIF_ARRAY_C_CONTIGUOUS, OIF_ARRAY_F_CONTIGUOUS

function _oif_array_f64_pointer_from_array_f64(arr::AbstractArray{T,N}) where {T<:Float64,N}
    ndim = ndims(arr)
    dimensions = Base.unsafe_convert(Ptr{Clong}, collect(size(arr)))
    data = Base.unsafe_convert(Ptr{Float64}, arr)
    oif_arr = Ref(
        OIFArrayF64(
            ndim,
            dimensions,
            data,
            OIF_ARRAY_C_CONTIGUOUS | OIF_ARRAY_F_CONTIGUOUS,
        ),
    )
    return oif_arr
end

it seems to me that the error is that I convert size(arr) to Ptr{Clong}, although the definition of OIFArrayF64 uses Int64:

struct OIFArrayF64
    nd::Int32
    dimensions::Ptr{Int64}
    data::Ptr{Float64}
    flags::Int32
end

for dimensions. Clong probably resolves to 32-bit integers instead of 64-bit integers. I wonder, how it worked before at all :slight_smile: I’ll check it when I am back to the office tomorrow and report it here.

4

That’s one thing, but it may work, depending on your cpu’s byte order. The other problem is that the output of collect is a Vector which is not referenced (a reference from a pointer doesn’t count), so it may easily happen that it’s freed by the garbage collector at any time. And possibly reused for something else. That may be the odd value you’re seeing.

2 Likes
5

Both of these are missing a call to Base.cconvert, and a GC.@preserve of the result of the call to Base.cconvert. I have a PR to try and clarify this in the docs

1 Like