# Limitation of mixed Tuple types in Arrow.write()?

**URL:** <https://discourse.julialang.org/t/limitation-of-mixed-tuple-types-in-arrow-write/75035>\
**Category:** General Usage\
**Tags:** dataframes, arrow\
**Created:** [January 22, 2022, 2:47pm UTC](https://discourse.julialang.org/t/limitation-of-mixed-tuple-types-in-arrow-write/75035 "2022-01-22T14:47:50Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![SortofDamocles](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sortofdamocles/32/15709_2.png) [@SortofDamocles](https://discourse.julialang.org/u/SortofDamocles)\
**Post date:** [January 22, 2022, 2:47pm UTC](https://discourse.julialang.org/t/limitation-of-mixed-tuple-types-in-arrow-write/75035/1 "2022-01-22T14:47:51Z")

</div>

When I run the following:

```julia
using Arrow, DataFrames

x = ((1,), (1,), (), ());
y = ((1, 2), (), (), ());
df = DataFrame(col = [x, y]);
Arrow.write("test.arrow", df)

```

I get the following error:

```julia
LoadError: ArgumentError: type does not have a definite number of fields
Stacktrace:
  [1] fieldcount(t::Any)
    @ Base .\reflection.jl:764
  [2] arrowvector(::ArrowTypes.StructKind, x::ArrowTypes.ToArrow{Tuple{Int64, Vararg{Int64}}, Arrow.ToStruct{Tuple{Int64, Vararg{Int64}}, 1, ArrowTypes.ToArrow{Tuple{Tuple{Int64, Vararg{Int64}}, Tuple{Vararg{Int64}}, Tuple{}, Tuple{}}, Vector{Tuple{Tuple{Int64, Vararg{Int64}}, Tuple{Vararg{Int64}}, Tuple{}, Tuple{}}}}}}, i::Int64, nl::Int64, fi::Int64, de::Dict{Int64, Any}, ded::Vector{Arrow.DictEncoding}, meta::Base.ImmutableDict{String, String}; kw::Base.Pairs{Symbol, Union{Nothing, Integer}, NTuple{6, Symbol}, NamedTuple{(:dictencode, :maxdepth, :compression, :largelists, :denseunions, :dictencodenested), Tuple{Bool, Int64, Nothing, Bool, Bool, Bool}}})
    @ Arrow C:\Users\rNr\.julia\packages\Arrow\x6smw\src\arraytypes\struct.jl:93

```

(plus many many more lines of traceback).

But each of the following _do_ work:  
`df = DataFrame(col = [x])`,  
`df = DataFrame(col = [y])`,  
and

```julia
using Arrow, DataFrames

x = [[1,], [1,], [], []];
y = [[1, 2], [], [], []];
df = DataFrame(col = [x, y]);
Arrow.write("test.arrow", df)

```

Is this the intended behavior? Why would arrays work but tuples not?

---

<div class="post-metadata">

**Author:** ![SortofDamocles](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sortofdamocles/32/15709_2.png) [@SortofDamocles](https://discourse.julialang.org/u/SortofDamocles)\
**Post date:** [January 22, 2022, 8:14pm UTC](https://discourse.julialang.org/t/limitation-of-mixed-tuple-types-in-arrow-write/75035/2 "2022-01-22T20:14:01Z")

</div>

I have solved the problem by explicitly typing the vector before I create the DataFrame. Here is edited working code:

```julia
x = ((1,), (1,), (), ());
y = ((1, 2), (), (), ());
T = Union{
    Tuple{Tuple{Int64}, Tuple{Int64}, Tuple{}, Tuple{}},
    Tuple{Tuple{Int64, Int64}, Tuple{}, Tuple{}, Tuple{}}
};
C = T[x, y];
df = DataFrame(col = C);
Arrow.write("test.arrow", df)

```

For my actual use case, I _also_ needed to increase the `maxdepth` keyword in Arrow.write().

---

<div class="post-metadata">

**Author:** ![nilshg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nilshg/32/2283_2.png) [@nilshg](https://discourse.julialang.org/u/nilshg)\
**Post date:** [January 22, 2022, 8:23pm UTC](https://discourse.julialang.org/t/limitation-of-mixed-tuple-types-in-arrow-write/75035/3 "2022-01-22T20:23:45Z")

</div>

Because Tuples have their length encoded in the type, and by putting these mixed length tuples together you do indeed create a type without a definite number of fields:

```julia
julia> [x y]
1×2 Matrix{Tuple{Tuple{Int64, Vararg{Int64}}, Tuple{Vararg{Int64}}, Tuple{}, Tuple{}}}:
 ((1,), (1,), (), ()) ((1, 2), (), (), ())

julia> [x]
1-element Vector{Tuple{Tuple{Int64}, Tuple{Int64}, Tuple{}, Tuple{}}}:
 ((1,), (1,), (), ())

julia> [y]
1-element Vector{Tuple{Tuple{Int64, Int64}, Tuple{}, Tuple{}, Tuple{}}}:
 ((1, 2), (), (), ())

```

(note the `Vararg` in the first type) so I would say this is intended in the sense that this really is a type that isn’t supported.

As the answer on StackOverflow said this is a very weird thing to use in a tabluar data format, so I wouldn’t expect great support for something like this. In general to write all sorts of weird objects to disk and read them back in serialization might be a better way to go.