# Transfer/ Unbox Julia's DataFrame objects and use as C++ object

**URL:** <https://discourse.julialang.org/t/transfer-unbox-julias-dataframe-objects-and-use-as-c-object/105440>\
**Category:** General Usage\
**Tags:** question, interoperability, dataframes, c\
**Created:** [October 26, 2023, 11:59am UTC](https://discourse.julialang.org/t/transfer-unbox-julias-dataframe-objects-and-use-as-c-object/105440 "2023-10-26T11:59:58Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![Ivan\_Navarro](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ivan_navarro/32/202737_2.png) [@Ivan\_Navarro](https://discourse.julialang.org/u/Ivan_Navarro)\
**Post date:** [October 26, 2023, 11:59am UTC](https://discourse.julialang.org/t/transfer-unbox-julias-dataframe-objects-and-use-as-c-object/105440/1 "2023-10-26T11:59:58Z")

</div>

**NOTE** : Cross posting from [Transfer/ Unbox Julia's DataFrame objects and use as C++ object - Stack Overflow](https://stackoverflow.com/questions/77366489/transfer-unbox-julias-dataframe-objects-and-use-as-c-object)

I would like to (un)box complex objects like DataFrames created in Julia, from C++. So I can use as input data for my C++ code.

For example, by loading MixedModels sample data `dyestuff`, and describing it as DataFrame, I can get:

```julia
using DataFrames, MixedModels

dyestuff = MixedModels.dataset(:dyestuff)
describe(DataFrame(dyestuff))

```

![image](https://global.discourse-cdn.com/julialang/original/3X/d/0/d0afa7c78a07f0cd791f8592ac9e9f0a34b01049.png)

Note that MixedModel output is of type `Arrow.Table`. By using `fieldnames(typeof(dyestuff))`, the available attributes are `(:names, :types, :columns, :lookup, :schema, :metadata)`.

**How can I effectively and efficiently transform/unbox complex object (like data frames) from Julia to C++?**

Because it is not possible (AFAIK) to share such complex data frame object, I felt it could be convenient to use the separated attributes as primitive objects and create my own DataFrame class in C++. So in C++ I am trying:

```plaintext
// to make sure I can execute julia code from C++
jl_eval_string("println(describe(DataFrame(dyestuff)))"); // 2×7 DataFrame
jl_eval_string("println(typeof(dyestuff))"); // Arrow.Table

// Get the length of column names attibute in julia's dyestuff dataframe.
jl_value_t *n_p = jl_eval_string("length(getfield(dyestuff, :names))");
int n = jl_unbox_int16(n_p); // output is 2 because we have 2 column names.

// Trying to load the actual column names (as array of strings) by following the Julia's manual:
jl_array_t *names_list = (jl_array_t *)jl_eval_string("String.(getfield(dyestuff, :names))");
string *names = (string *)jl_array_data(names_list);
cout << jl_array_len(names_list) << endl;
for (size_t i = 0; i < jl_array_len(names_list); i++)
{
    cout << " " << names[i] << endl;
}

```

But when printing out the names I am getting strange characters which seems to be binary of the object, but not the column names.

```julia
 batch��
��Rmath��
��yield�F���1ath��
��NaNMath�F���min th�F��� min ��
��QuadGK�F���1adGKG���1adGK��
��GLMs(G���max eXG��� max ��
��JSON3epG���1ON3�G���1ON3��
��NLopt�G��� pt�G���1optL��
��AdaptH��� tH��� 0H��� HH���│9`H��� dfxH���90df_�H���90df_�H���Any �H��� Any �H���90ets�H���90ndI���Any 4 I��� Any 8I���90dlPI��� inghI���90pe�I���90it2�I���┼ets�I��� code�I��� 1tf��
�� 1 ��
��│J��� 

```

Note that out put of Julia’s code is:

```julia
julia> String.(getfield(dyestuff, :names))
2-element Vector{String}:
 "batch"
 "yield"

```

Getting the names attribute is just the first step. But other challenges will come when working with other Data Frame attibutes like types, columns, metadata, lookup, etc. So any suggestion in that front will be helpful. Thanks in advance!

---

<div class="post-metadata">

**Author:** ![mkitti](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mkitti/32/12459_2.png) [@mkitti](https://discourse.julialang.org/u/mkitti)\
**Post date:** [October 26, 2023, 12:21pm UTC](https://discourse.julialang.org/t/transfer-unbox-julias-dataframe-objects-and-use-as-c-object/105440/2 "2023-10-26T12:21:50Z")

</div>

If this is an `Arrow.Table` why don’t you just use Arrow?

[https://arrow.apache.org/docs/cpp/](https://arrow.apache.org/docs/cpp/)

Language interop is half rhe purpose of Arrow.

[https://arrow.apache.org/docs/cpp/](https://arrow.apache.org/docs/cpp/)

---

<div class="post-metadata">

**Author:** ![Ivan\_Navarro](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ivan_navarro/32/202737_2.png) [@Ivan\_Navarro](https://discourse.julialang.org/u/Ivan_Navarro)\
**Post date:** [October 26, 2023, 1:17pm UTC](https://discourse.julialang.org/t/transfer-unbox-julias-dataframe-objects-and-use-as-c-object/105440/3 "2023-10-26T13:17:02Z")

</div>

Thanks @mkitti for the idea. Actually that was my first approach, but had issues to compile and use Arrow from C++ for any reason. And I took this other approach as exercise to understand how to share complex objects or structures between the two languages.

---

<div class="post-metadata">

**Author:** ![jling](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jling/32/212909_2.png) [@jling](https://discourse.julialang.org/u/jling)\
**Post date:** [October 26, 2023, 1:40pm UTC](https://discourse.julialang.org/t/transfer-unbox-julias-dataframe-objects-and-use-as-c-object/105440/4 "2023-10-26T13:40:08Z")

</div>

> [@Ivan\_Navarro](#):
>
> How can I effectively and efficiently transform/unbox complex object (like data frames) from Julia to C++?

you shouldn’t, `DataFrames.jl` doesn’t have a defined memory layout, it’s not meant to be used as IPC. You should use Arrow IPC stream

---

<div class="post-metadata">

**Author:** ![Ivan\_Navarro](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ivan_navarro/32/202737_2.png) [@Ivan\_Navarro](https://discourse.julialang.org/u/Ivan_Navarro)\
**Post date:** [October 26, 2023, 2:23pm UTC](https://discourse.julialang.org/t/transfer-unbox-julias-dataframe-objects-and-use-as-c-object/105440/5 "2023-10-26T14:23:37Z")

</div>

Yeah, I see that I should give another try to Arrow. However, Imagine that we are not talking about data frames, but other simpler but still structured data, like array of strings (as described in the problem) or a dictionary in Julia which I want to access from C++ directly. Let me be more precise in the questions:

1. I’ve read that `jl_string_ptr` is used to access to strings (instead of using an hypothetical jl\_unbox\_string), but How to access if such strings are inside an array, like in the problem I explained?

2. How can I access/ “unbox” dictionary (which values are arrays) or JSON objects defined in Julia, from C++?

This is very useful in scenarios in which I generate data from Julia but still I want to use my C++ algorithms as they are. I’d like access such structures from my C++ code, and the legacy code does not use Arrow, but I can manage such objects (dictionaries and arrays of strings).

Thanks!

---

<div class="post-metadata">

**Author:** ![jling](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jling/32/212909_2.png) [@jling](https://discourse.julialang.org/u/jling)\
**Post date:** [October 26, 2023, 2:50pm UTC](https://discourse.julialang.org/t/transfer-unbox-julias-dataframe-objects-and-use-as-c-object/105440/6 "2023-10-26T14:50:17Z")

</div>

> [@Ivan\_Navarro](#):
>
> strings

Julia String is UTF-8 and not null-terminated etc. So again you probably shouldn’t rely on it in the long term.

The only thing you can rely on on `isbit` types, and the way to do it is outlined here: [Embedding Julia · The Julia Language](https://docs.julialang.org/en/v1/manual/embedding/#Working-with-Arrays)

---

<div class="post-metadata">

**Author:** ![Ivan\_Navarro](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ivan_navarro/32/202737_2.png) [@Ivan\_Navarro](https://discourse.julialang.org/u/Ivan_Navarro)\
**Post date:** [October 27, 2023, 9:05am UTC](https://discourse.julialang.org/t/transfer-unbox-julias-dataframe-objects-and-use-as-c-object/105440/7 "2023-10-27T09:05:44Z")

</div>

Thanks @jling for pointing this information. Very helpful as reference!  
However, from the same, I understand that Julia’s `strings` type is bits-type. Also, you are right that strings/chars in C++ are null-terminated, and that’s why Julia has `Cstring` types which is returning a memory position to get shared with C/C++, I believe.

I verified that Julia `string` are bits-type:

```julia
julia> isbits(string)
true

```

Also, In [Bits Type](https://docs.julialang.org/en/v1/manual/calling-c-and-fortran-code/#man-bits-types), they point that `Array{T,N}` is also a bits type and say:

" _When an array is passed to C as a `Ptr{T}` argument, it is not reinterpret-cast: Julia requires that the element type of the array matches `T` , and the address of the first element is passed._  
…  
_If an array of eltype `Ptr{T}` is passed as a `Ptr{Ptr{T}}` argument, `Base.cconvert` will attempt to first make a null-terminated copy of the array with each element replaced by its `Base.cconvert` version. This allows, for example, passing an `argv` pointer array of type `Vector{String}` to an argument of type `Ptr{Ptr{Cchar}}`_  
"

```julia
julia> s = String.(getfield(dyestuff, :names))
2-element Vector{String}:
 "batch"
 "yield"

julia> s[1]
"batch"

julia> Base.unsafe_convert(Cstring, s)
Cstring(0x00007f162afd7c30)

```

Hence, I still feel there could be a way to share a Julia’s Array of strings to C.

---

<div class="post-metadata">

**Author:** ![jling](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jling/32/212909_2.png) [@jling](https://discourse.julialang.org/u/jling)\
**Post date:** [October 27, 2023, 9:42am UTC](https://discourse.julialang.org/t/transfer-unbox-julias-dataframe-objects-and-use-as-c-object/105440/8 "2023-10-27T09:42:40Z")

</div>

CString indeed works, and maybe interface may auto convert. I was saying you can’t pluck a pointer to Vector of any good old Julia string and expect C++ to just be able to use the same memory. That only works for a small set of primitive ish types

---

<div class="post-metadata">

**Author:** ![Ivan\_Navarro](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ivan_navarro/32/202737_2.png) [@Ivan\_Navarro](https://discourse.julialang.org/u/Ivan_Navarro)\
**Post date:** [October 27, 2023, 10:04am UTC](https://discourse.julialang.org/t/transfer-unbox-julias-dataframe-objects-and-use-as-c-object/105440/9 "2023-10-27T10:04:23Z")

</div>

> [@jling](#):
>
> I was saying you can’t pluck a pointer to Vector of any good old Julia string and expect C++ to just be able to use the same memory. That only works for a small set of primitive ish types

Sure. Thanks for pointing and confirming this.

> [@jling](#):
>
> CString indeed works, and maybe interface may auto convert.

Do you know how would it be? Any reference page that you may know? Would be helpful.  
Thanks.

---

<div class="post-metadata">

**Author:** ![jling](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jling/32/212909_2.png) [@jling](https://discourse.julialang.org/u/jling)\
**Post date:** [October 27, 2023, 10:49am UTC](https://discourse.julialang.org/t/transfer-unbox-julias-dataframe-objects-and-use-as-c-object/105440/10 "2023-10-27T10:49:45Z")

</div>

[C Interface · The Julia Language](https://docs.julialang.org/en/v1/base/c/#Base.@ccall) probably

---

<div class="post-metadata">

**Author:** ![GunnarFarneback](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gunnarfarneback/32/1827_2.png) [@GunnarFarneback](https://discourse.julialang.org/u/GunnarFarneback)\
**Post date:** [October 27, 2023, 11:50am UTC](https://discourse.julialang.org/t/transfer-unbox-julias-dataframe-objects-and-use-as-c-object/105440/11 "2023-10-27T11:50:13Z")

</div>

> [@Ivan\_Navarro](#):
>
> Imagine that we are not talking about data frames, but other simpler but still structured data, like array of strings (as described in the problem) or a dictionary in Julia which I want to access from C++ directly.

If you want to keep things simple I recommend

- Decompose your data into sufficiently primitive types on the Julia side.
- If you want to call Julia functions from C++, go through `@cfunction` pointers.

In general I try to keep the interaction with the `julia.h` functions as small as possible. The code in [GitHub - GunnarFarneback/DynamicallyLoadedEmbedding.jl: Embed Julia with dynamical loading of libjulia at runtime.](https://github.com/GunnarFarneback/DynamicallyLoadedEmbedding.jl) is fairly minimalistic in that respect.

---

<div class="post-metadata">

**Author:** ![mkitti](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mkitti/32/12459_2.png) [@mkitti](https://discourse.julialang.org/u/mkitti)\
**Post date:** [October 27, 2023, 4:20pm UTC](https://discourse.julialang.org/t/transfer-unbox-julias-dataframe-objects-and-use-as-c-object/105440/12 "2023-10-27T16:20:18Z")

</div>

> [@Ivan\_Navarro](#):
>
> Also, you are right that strings/chars in C++ are null-terminated

Only C strings are null terminated. C++ std::string explicitly tracks the string length.

---

<div class="post-metadata">

**Author:** ![jling](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jling/32/212909_2.png) [@jling](https://discourse.julialang.org/u/jling)\
**Post date:** [October 27, 2023, 4:36pm UTC](https://discourse.julialang.org/t/transfer-unbox-julias-dataframe-objects-and-use-as-c-object/105440/13 "2023-10-27T16:36:37Z")

</div>

> <https://stackoverflow.com/questions/11752705/does-stdstring-have-a-null-terminator/11752722#11752722>

It looks like at least in newish C++, one can very easily go from/to C string and seems like the underlying data is also null terminated?