Retrieving an array from a Ptr{Ptr{Cvoid}}

,

Hi!

I have a Julia wrapper function that calls a C function from a shared library. This C function updates an output structure databuf, which is an instance of

mutable struct MTKt_DataBuffer
    nline::Cint
    nsample::Cint
    datasize::Cint
    datatype::MTKt_DataType
    imported::MTKt_boolean
    data::MTKt_DataBufferType
    vdata::Ptr{Ptr{Cvoid}}
    dataptr::Ptr{Cvoid}
end

Upon returning from the ccall,

julia> databuf
MTKt_DataBuffer(512,
    2048,
    2,
    JMtk15.MTKe_uint16,
    JMtk15.MTK_FALSE,
    MTKt_DataBufferType((0x00, 0x00, 0x00, 0x00, 0x00, 0x42, 0x43, 0xba)),
    Ptr{Ptr{Nothing}} @0x00007fcfba434200,
    Ptr{Nothing} @0x000000014b250000)

julia> databuf.nline
512

julia> databuf.nsample
2048

julia> databuf.datatype
MTKe_uint16::MTKt_DataType = 0x00000006

How do I retrieve the 2D array vdata? I’ve tried unsafe_wrap, but I don’t seem to get what I was expecting (namely an array of 512 lines and 2048 columns UInt16 values):

julia> outm = unsafe_wrap(Array, databuf.vdata::Ptr{Ptr{Nothing}}, (databuf.nline, databuf.nsample); own = false)
512Ă—2048 Matrix{Ptr{Nothing}}:
 Ptr{Nothing} @0x000000014b250000  …  Ptr{Nothing} @0x0000000000000000
 Ptr{Nothing} @0x000000014b251000     Ptr{Nothing} @0x0000000000000000
 Ptr{Nothing} @0x000000014b252000     Ptr{Nothing} @0x00007fcfbb96a9c0
 Ptr{Nothing} @0x000000014b253000     Ptr{Nothing} @0x000000014aa94210
 Ptr{Nothing} @0x000000014b254000     Ptr{Nothing} @0x000000010b3beb60
 Ptr{Nothing} @0x000000014b255000  …  Ptr{Nothing} @0x00007fcfbb96a9a8
 Ptr{Nothing} @0x000000014b256000     Ptr{Nothing} @0x000000014be386e0
 Ptr{Nothing} @0x000000014b257000     Ptr{Nothing} @0x0300000000000006
 Ptr{Nothing} @0x000000014b258000     Ptr{Nothing} @0x000000014be387e0
 Ptr{Nothing} @0x000000014b259000     Ptr{Nothing} @0x0000000000000000
 ⋮                                 ⋱  
 Ptr{Nothing} @0x000000014b447000     Ptr{Nothing} @0x0000000000000000
 Ptr{Nothing} @0x000000014b448000     Ptr{Nothing} @0x0000000000000000
 Ptr{Nothing} @0x000000014b449000  …  Ptr{Nothing} @0x00007fcfbac33d20
 Ptr{Nothing} @0x000000014b44a000     Ptr{Nothing} @0x0000000000000000
 Ptr{Nothing} @0x000000014b44b000     Ptr{Nothing} @0x000000010b3aef60
 Ptr{Nothing} @0x000000014b44c000     Ptr{Nothing} @0x0000000000000000
 Ptr{Nothing} @0x000000014b44d000     Ptr{Nothing} @0x00007fcfbac34210
 Ptr{Nothing} @0x000000014b44e000  …  Ptr{Nothing} @0x0300000000000007
 Ptr{Nothing} @0x000000014b44f000     Ptr{Nothing} @0x0000000000000000

julia> outm[100, 100]
Ptr{Nothing} @0x0000000000000000

Lastly, how to deal with a Ptr{Ptr{Ptr{Cvoid}}}? Thanks for suggestions…

Do you have any documentation on what vdata means? My best guess is that

vp = unsafe_wrap(Array, databuf.vdata, databuf,nline)

gives you pointers to individual line arrays and that

[unsafe_wrap(Array, Ptr{T}(p), databuf.nsample) for p in vp]

gives you all the line data, assuming you set T to whatever datatype MTKe_uint16::MTKt_DataType = 0x00000006 happens to encode.

Why not pass the type to unsafe_wrap?

unsafe_wrap(Array{UInt16, 2}, databuf.vdata, (databuf.nline, databuf.nsample); own=false)

Also note that C uses row-major ordering whereas Julia column-major so it might be necessary to switch the dimension order.

As I answered in the DM message, vdata needs to be unsafe_loaded once and then we can use unsafe_wrap to extract the array.

There must still be something missing:

@barucden:

julia> outm = unsafe_wrap(Array{UInt16, 2}, databuf.vdata, (databuf.nline, databuf.nsample); own=false)
ERROR: StackOverflowError:

or @Gnimuc:

julia> outm1 = unsafe_load(databuf.vdata)
Ptr{Nothing} @0x000000014b250000

julia> outm2 = unsafe_wrap(Array{UInt16, 2}, outm1, (databuf.nline, databuf.nsample); own=false)
ERROR: StackOverflowError:

https://nasa.github.io/MISR-Toolkit/Mtk_struct.html

MTKt_DataBuffer

The MTKt_DataBuffer structure is used to return geo-located data planes (“arrays”) of MISR data from the MtkReadData routine. It is designed to handle any C datatype. It does this using a union pointer data field which points to the same data pointed to by the dataptr field. The dataptr field points to the main data array which is stored in row-major form. The number of bytes of the data plane can be determined using buf.nline, buf.nsample and sizeof the datatype field. It can be used for bulk transfers. There is also a row vector (Illiffe vector) pointing to each row of the main array to support C bracket index notation for a 2D-array. It can be ignored. The structure does maintain dynamic memory, so use MtkDataBufferFree when finished with this structure.
Type Definition:


typedef enum {
  MTKe_void=0,
  MTKe_char8,
  MTKe_uchar8,
  MTKe_int8,
  MTKe_uint8,
  MTKe_int16,
  MTKe_uint16,
  MTKe_int32,
  MTKe_uint32,
  MTKe_int64,
  MTKe_uint64,
  MTKe_float,
  MTKe_double
} MTKt_DataType;

typedef union {
  void **v;
  MTKt_char8 **c8;
  MTKt_uchar8 **uc8;
  MTKt_int8 **i8;
  MTKt_uint8 **u8;
  MTKt_int16 **i16;
  MTKt_uint16 **u16;
  MTKt_int32 **i32;
  MTKt_uint32 **u32;
  MTKt_int64 **i64;
  MTKt_uint64 **u64;
  MTKt_float **f;
  MTKt_double **d;
} MTKt_DataBufferType;

typedef struct {
  int nline;                    /**< Number of lines */
  int nsample;                  /**< Number of samples */
  int datasize;                 /**< Data element size (bytes) */
  MTKt_DataType datatype;       /**< Data type (enumeration) */
  MTKt_DataBufferType data;     /**< Data type access union */
  void **vdata;                 /**< Row major 2D array with Illiffe vector */
  void *dataptr;                /**< Pointer data buffer */
} MTKt_DataBuffer;
Example Usage:

#include “MisrToolkit.h”

MTKt_status status;
MTKt_Region region = MTK_REGION_INIT;
MTKt_DataBuffer buf = MTKT_DATABUFFER_INIT;
MTKt_MapInfo mapinfo = MTKT_MAPINFO_INIT;

status = MtkSetRegionByPathBlockRange(32, 65, 125, &region);
if (status != MTK_SUCCESS) error;

status = MtkReadData(“MISR_file.hdf”, “MISR_gridname”, “MISR_fieldname”, region, &buf, &mapinfo)
if (status != MTK_SUCCESS) error;

for (l = 0; l < buf.nline; l++)
   for (s = 0; s < buf.nsample; s++)
       printf(“Data Buffer[%d][%d]: %f\n”, l, s, buf.data.f[l][s]);

MtkDataBufferFree(&buf);

This suggests we can directly use unsafe_wrap dataptr instead of the mysterious vdata.

Still getting the error message:

julia> outm3 = unsafe_wrap(Array{UInt16, 2}, databuf.dataptr, (databuf.nline, databuf.nsample); own=false)
ERROR: StackOverflowError:
1 Like

Because it gives StackOverflowError and has no moral support in the docstring?

unsafe_wrap(Array, pointer::Ptr{T}, dims; own = false)

Wrap a Julia Array object around the data at the address given by pointer, without
making a copy. The pointer element type T determines the array element type.

outm3 = unsafe_wrap(Array, Ptr{UInt16}(databuf.dataptr), (databuf.nline, databuf.nsample); own=false)

Thanks a lot, @stevengj: That worked (albeit in hexadecimal):

julia> outm3 = unsafe_wrap(Array, Ptr{UInt16}(databuf.dataptr), (databuf.nline, databuf.nsample); own=false)
512Ă—2048 Matrix{UInt16}:
 0xffeb  0x1c54  0x068c  0x0fbc  0xffeb  …  0xffeb  0x1810  0x1324  0x14e4
 0xffeb  0x1d1c  0x0698  0x1038  0xffeb     0xffeb  0x1860  0x1300  0x14d8
 0xffeb  0x1d28  0x06ac  0x109c  0xffeb     0xffeb  0x1b38  0x12d0  0x128c
 0xffeb  0x1d0c  0x06b4  0x11fc  0xffeb     0xffeb  0x1c0c  0x12cc  0x1114
 0xffeb  0x1ca0  0x07f4  0x1130  0xffeb     0xffeb  0x1b64  0x12dc  0x10f8
 0xffeb  0x1be8  0x0968  0x1074  0xffeb  …  0xffeb  0x1bac  0x12bc  0x11a4
 0xffeb  0x1d08  0x07dc  0x10bc  0xffeb     0xffeb  0x1b8c  0x125c  0x12d8
 0xffeb  0x1e60  0x06d8  0x10f8  0xffeb     0xffeb  0x1b20  0x1294  0x0de4
 0xffeb  0x1e68  0x06c4  0x1184  0xffeb     0xffeb  0x1aac  0x12a8  0x0b20
 0xffeb  0x1d90  0x06c4  0x1218  0xffeb     0xffeb  0x1a20  0x12dc  0x1098
      ⋮                                  ⋱               ⋮          
 0x1bcc  0x06dc  0x0d98  0xffeb  0x1ad8     0x1970  0x13cc  0x124c  0xffeb
 0x1cc8  0x06bc  0x0fdc  0xffeb  0x1b90     0x1980  0x1410  0x1298  0xffeb
 0x1cc0  0x06b0  0x1198  0xffeb  0x1c04  …  0x1910  0x1428  0x1330  0xffeb
 0x1cdc  0x06a8  0x1184  0xffeb  0x1cb8     0x1900  0x13f8  0x1408  0xffeb
 0x1d20  0x06a4  0x1208  0xffeb  0x1bbc     0x196c  0x13e8  0x14d4  0xffeb
 0x1cc4  0x068c  0x1348  0xffeb  0x1ab8     0x1978  0x1440  0x1410  0xffeb
 0x1b74  0x06a8  0x1340  0xffeb  0x1a48     0x1a04  0x1414  0x1334  0xffeb
 0x1ad0  0x069c  0x124c  0xffeb  0x19fc  …  0x1c20  0x13a4  0x134c  0xffeb
 0x1b50  0x068c  0x10c4  0xffeb  0x1aa8     0x19a0  0x1360  0x14a0  0xffeb

That’s just how Julia prints unsigned integers.