Limitation of mixed Tuple types in Arrow.write()?

General Usage
,
1

When I run the following:

using Arrow, DataFrames

x = ((1,), (1,), (), ());
y = ((1, 2), (), (), ());
df = DataFrame(col = [x, y]);
Arrow.write("test.arrow", df)

I get the following error:

LoadError: ArgumentError: type does not have a definite number of fields
Stacktrace:
  [1] fieldcount(t::Any)
    @ Base .\reflection.jl:764
  [2] arrowvector(::ArrowTypes.StructKind, x::ArrowTypes.ToArrow{Tuple{Int64, Vararg{Int64}}, Arrow.ToStruct{Tuple{Int64, Vararg{Int64}}, 1, ArrowTypes.ToArrow{Tuple{Tuple{Int64, Vararg{Int64}}, Tuple{Vararg{Int64}}, Tuple{}, Tuple{}}, Vector{Tuple{Tuple{Int64, Vararg{Int64}}, Tuple{Vararg{Int64}}, Tuple{}, Tuple{}}}}}}, i::Int64, nl::Int64, fi::Int64, de::Dict{Int64, Any}, ded::Vector{Arrow.DictEncoding}, meta::Base.ImmutableDict{String, String}; kw::Base.Pairs{Symbol, Union{Nothing, Integer}, NTuple{6, Symbol}, NamedTuple{(:dictencode, :maxdepth, :compression, :largelists, :denseunions, :dictencodenested), Tuple{Bool, Int64, Nothing, Bool, Bool, Bool}}})
    @ Arrow C:\Users\rNr\.julia\packages\Arrow\x6smw\src\arraytypes\struct.jl:93

(plus many many more lines of traceback).

But each of the following do work:
df = DataFrame(col = [x]),
df = DataFrame(col = [y]),
and

using Arrow, DataFrames

x = [[1,], [1,], [], []];
y = [[1, 2], [], [], []];
df = DataFrame(col = [x, y]);
Arrow.write("test.arrow", df)

Is this the intended behavior? Why would arrays work but tuples not?

2

I have solved the problem by explicitly typing the vector before I create the DataFrame. Here is edited working code:

x = ((1,), (1,), (), ());
y = ((1, 2), (), (), ());
T = Union{
    Tuple{Tuple{Int64}, Tuple{Int64}, Tuple{}, Tuple{}},
    Tuple{Tuple{Int64, Int64}, Tuple{}, Tuple{}, Tuple{}}
};
C = T[x, y];
df = DataFrame(col = C);
Arrow.write("test.arrow", df)

For my actual use case, I also needed to increase the maxdepth keyword in Arrow.write().

3

Because Tuples have their length encoded in the type, and by putting these mixed length tuples together you do indeed create a type without a definite number of fields:

julia> [x y]
1×2 Matrix{Tuple{Tuple{Int64, Vararg{Int64}}, Tuple{Vararg{Int64}}, Tuple{}, Tuple{}}}:
 ((1,), (1,), (), ())  ((1, 2), (), (), ())

julia> [x]
1-element Vector{Tuple{Tuple{Int64}, Tuple{Int64}, Tuple{}, Tuple{}}}:
 ((1,), (1,), (), ())

julia> [y]
1-element Vector{Tuple{Tuple{Int64, Int64}, Tuple{}, Tuple{}, Tuple{}}}:
 ((1, 2), (), (), ())

(note the Vararg in the first type) so I would say this is intended in the sense that this really is a type that isn’t supported.

As the answer on StackOverflow said this is a very weird thing to use in a tabluar data format, so I wouldn’t expect great support for something like this. In general to write all sorts of weird objects to disk and read them back in serialization might be a better way to go.