Finalizer only works with mutable structs?

[yuyichao]

@foobar_lv2

I.e. just put a @noinline function barrier outside of contexts where we take pointers and trust that the calling context keeps our stuff alive?

I would not rely on that in the long term.

[Mikhail_Kagalenko]

So having figured out, I think, passing array of pointers to structs, I got stuck passing array of 1-element arrays. The C function I need to use has the interface like this test function

typedef struct
{
  int a;
} _M;

typedef _M m_t[1];

void
print_M_arr(int N, m_t *ptr)
{
  for( int n = 0; n < N; n++) {
   fprintf(stdout, "M[%d]->a: %d\n",n, ptr[n]->a);
  }
}

And the on the Julia side,

mutable struct M
    a :: Cint
end
N=3
m = [[M(n)] for n = 1:N]
m_p = [Base.unsafe_convert(Ref{M},Base.cconvert(Ref{M},m[n])) for n = 1:N]

GC.@preserve m begin
    ccall((:print_M_arr, "./libmp_test"), Cvoid, (Cint, Ref{Ptr{M}},), N, m_p)
end

That prints garbage. I didn’t find any recent tips on how to do this call.

[yuyichao]

@Mikhail_Kagalenko

m_p = [Base.unsafe_convert(Ref{M},Base.cconvert(Ref{M},m[n])) for n = 1:N]

Again, be aware that this can return completely garbage since the return value of cconvert(Ref{M}, ...) is not kept safe anywhere. But thtat’s probably not what’s causing the issue.

@Mikhail_Kagalenko

ccall((:print_M_arr, "./libmp_test"), Cvoid, (Cint, Ref{Ptr{M}},), N, m_p)

And no, this is the wrong type. I get that this is due to one of the biggest misconception people have in C,

Array type and pointer type are NOT the same thing. Again, they are NOT the same.

What you have in your C code is simply a pointer to an array type and the pointer is pointing to the array, not the pointer to the array. You do not have a pointer to a pointer type, which is what you’ve created in julia. What you need is simply a pointer to M.

[c42f]

@Mikhail_Kagalenko

The C function I need to use has the interface like this test function

[…]

void print_M_arr(int N, m_t *ptr)

This C signature is certainly incompatible with [[M(n)] for n = 1:N] as you’ve found out, but how close is this to what you’re really trying to do? These things are subtle and we could give more precise help if you post the real interface you’re trying to call rather than a toy example.

[Mikhail_Kagalenko]

@yuyichao

Array type and pointer type are NOT the same thing. Again, they are NOT the same.

Yes, I know that. In fact, I wrote that passing array of pointers to structs works in the very message you replied to.

@yuyichao

What you need is simply a pointer to M .

Well, I did try

m = [M(n) for n = 1:N]
GC.@preserve m begin
    m_p=Base.unsafe_convert(Ref{M},Base.cconvert(Ref{M},m))
    ccall((:print_M_arr, "./libmps_calls"), Cvoid, (Cint, Ref{M},), N, m_p)
end

The type of m_p here is Ptr{M}, that prints garbage as well.

[Mikhail_Kagalenko]

@c42f

but how close is this to what you’re really trying to do?

It is almost exacly what I am tring to call. Really it is the
mpq_t type from GNU GMP library. In Julia that would translate to struct containing two immutable structs. I am pretty sure that once the example works, the rest should fall into place.

[Sukera]

With the danger of you already knowing, Julias BigInts are GMP backed (see here).

[c42f]

@Mikhail_Kagalenko

Really it is the mpq_t type from GNU GMP library. In Julia that would translate to struct containing two immutable structs.

Ah hah, thanks. You’re right that this case should be modeled as a struct containing two immutable structs. Let’s consider mpq_init from Initializing Rationals (GNU MP 6.3.0). From the GMP header we have

typedef struct
{
  __mpz_struct _mp_num;
  __mpz_struct _mp_den;
} __mpq_struct;

typedef __mpq_struct *mpq_ptr;

typedef __mpq_struct mpq_t[1];

void __gmpq_init (mpq_ptr);  // Documented as taking `mpq_t` but that's equivalent

I expect the mpq_t typedef is really just a convenience which allows declaring local variables in C as mpq_t. The actual header uses mpq_ptr here rather than mpq_t for mpq_init (real name __gmpq_init) but those are equivalent in this usage.

To call from Julia, do something like:

struct MPZ
    alloc::Cint
    size::Cint
    d::Ptr{Base.GMP.Limb}
end

mutable struct MPQ
    num::MPZ
    den::MPZ

    function MPQ()
        mpq = new()
        ccall((:__gmpq_init, :libgmp), Cvoid, (Ref{MPQ},), mpq)
        # TODO: add finalizer!
        mpq
    end
end

mpq = MPQ()

# Set value from string
ccall((:__gmpq_set_str, :libgmp), Cint, (Ref{MPQ}, Cstring, Cint), mpq, "41/152", 10)

# Invert
ccall((:__gmpq_inv, :libgmp), Cvoid, (Ref{MPQ}, Ref{MPQ}), mpq, mpq)

# Get string representation (FIXME: This current leaks memory!)
unsafe_string(ccall((:__gmpq_get_str, :libgmp), Cstring, (Cstring, Cint, Ref{MPQ}), C_NULL, 10, mpq))

[yuyichao]

@Mikhail_Kagalenko

In fact, I wrote that passing array of pointers to structs works in the very message you replied to.

What do you mean? I’m a little confused why you are trying to pass with more level of pointers then…

@Mikhail_Kagalenko

GC.@preserve m begin 
    m_p=Base.unsafe_convert(Ref{M},Base.cconvert(Ref{M},m))

Again, this is wrong. You need to preserve the return value of cconvert. Also, you almost never need to write out the conversion explicitly with ccall. Doing it explicitly and doing it wrong just make it worse. But anyway, that’s now why you get garbage.

@Mikhail_Kagalenko

The type of m_p here is Ptr{M}

This doesn’t matter. What matters is how you get the pointer. This one is a case (I think there’s an issue for it) that the default conversion allows the conversion in cases that it really shouldn’t. Your m is an array of reference, (array of pointer) in memory. Since M is mutable, it’s impossible to pass an array of M as C array, the memory layout is completely incompatible. You have to use immutable type here.

[c42f]

Having said all that about how to call into GMP, I should point out that Julia has native support for big rationals:

julia> q = big(41//152)
41//152

julia> 1/q
152//41

julia> q^15
1555098314991537910888601//534138422146939893094821310496768

[Mikhail_Kagalenko]

Passing a single mpq_t is not a problem, that works both with the simplified example and the mpq_t type. However, as I stated above, I am trying to use a function that accepts and returns arrays of mpq_t

[Mikhail_Kagalenko]

Julia’s Rational{BigInt} is not interoperable with mpq_t, though.

[Mikhail_Kagalenko]

@yuyichao

What do you mean?

In my first reply in this discussion I gave an example.

@yuyichao

Since M is mutable, it’s impossible to pass an array of M as C array, the memory layout is completely incompatible. You have to use immutable type here.

I understand why it is impossible to pass an array where structures are located sequentially in memory, unless Julia structs are immutable. But does the type of C call print_m(int N, m_t *arr) imply that the structures are located in one contiguous block of memory? The arr[n] value in C code is a pointer to struct, not the struct. On the other hand, sizeof(m_t) will be equal to the size of the struct, not the pointer, so maybe that is the issue with my code.

[yuyichao]

@Mikhail_Kagalenko

In my first reply in this discussion I gave an example.

Well, if you meant that you know the C code you give for pointer to array is different from the previous post of pointer to pointer, then yes, you are correct. However, it appears that you still got that wrong. You are adding one more level of indirection in the julia code for the pointer to array than the pointer to pointer. In reality, you should remove one level of indirection.

@Mikhail_Kagalenko

imply that the structures are located in one contiguous block of memory?

Yes, that’s exactly what I said. (See below)

@Mikhail_Kagalenko

The arr[n] value in C code is a pointer to struct, not the struct.

Nope, it’s not. It’s an array of the struct, not a pointer to the struct. Again, array and pointer are different in C. C just have some very relaxed rule for array to pointer type decay which makes it possible for you to treat array as pointers when passing them arround. They are completely different when you are dealing with object layout.

[Mikhail_Kagalenko]

@yuyichao

The arr[n] value in C code is a pointer to struct, not the struct.

Nope, it’s not. It’s an array of the struct, not a pointer to the struct.

The name of array in C code decays to the pointer to its first element, so in C code arr[n] is a pointer to the struct.
http://c-faq.com/aryptr/aryptrequiv.html

[yuyichao]

Nope. arr[n] is not a name. (edit: actually I’m not sure about the not a name part, but it is not a pointer for sure…)

yyc:~/tmp
yuyichao% cat a.c
//

struct A {
    char a[100];
};

typedef struct A AA[1];

int f(AA *aa)
{
    return sizeof(aa[0]);
}
yyc:~/tmp
yuyichao% clang a.c -S -emit-llvm -o - -O2
; ModuleID = 'a.c'
source_filename = "a.c"
target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
target triple = "x86_64-pc-linux-gnu"

%struct.A = type { [100 x i8] }

; Function Attrs: norecurse nounwind readnone sspstrong uwtable
define dso_local i32 @f([1 x %struct.A]* nocapture readnone) local_unnamed_addr #0 {
  ret i32 100
}

attributes #0 = { norecurse nounwind readnone sspstrong uwtable "correctly-rounded-divide-sqrt-fp-math"="false" "disable-tail-calls"="false" "less-precise-fpmad"="false" "min-legal-vector-width"="0" "no-frame-pointer-elim"="false" "no-infs-fp-math"="false" "no-jump-tables"="false" "no-nans-fp-math"="false" "no-signed-zeros-fp-math"="false" "no-trapping-math"="false" "stack-protector-buffer-size"="8" "target-cpu"="x86-64" "target-features"="+cx8,+fxsr,+mmx,+sse,+sse2,+x87" "unsafe-fp-math"="false" "use-soft-float"="false" }

!llvm.module.flags = !{!0, !1, !2}
!llvm.ident = !{!3}

!0 = !{i32 1, !"wchar_size", i32 4}
!1 = !{i32 7, !"PIC Level", i32 2}
!2 = !{i32 7, !"PIE Level", i32 2}
!3 = !{!"clang version 9.0.0 (tags/RELEASE_900/final)"}

[Mikhail_Kagalenko]

By the way

@yuyichao

Again, this is wrong. You need to preserve the return value of cconvert .

Are you saying that I need to insert another GC.@preserve before ccall?

[yuyichao]

No, you don’t need another one, you simply placed the one at the wrong place. You don’t need to do anything about m. Again… you only need to preserve the return value of cconvert.

[Mikhail_Kagalenko]

C standard says

A reference to an object of type array-of-T which appears in an expression decays (with three exceptions) into a pointer to its first element; the type of the resultant pointer is pointer-to-T.

(the three exceptions do not apply here) If Clang doesn’t do that, it is wrong.

[yuyichao]

I can tell you with 100% certainty that clang is correct and gcc does the same thing.