Juliac with TypedTables

General Usage
,
1

I want to compile functions processing tables with help of juliac. I use TypedTables. Unfortunately the code cannot be compiled as soon the tables have more than 32 columns. This is probably caused by the type-instability that occurs, when the used tuples exceed 32 elements. (The compiler collapses the types and infers them as Any.). (see https://github.com/JuliaLang/JuliaC.jl/issues/140
Here is an example:

using TypedTables

function (@main)(args::Vector{String})::Cint
  ts = (
    a = Table(c1 = [1], c2 = [2], c3 = [3], c4 = [4], c5 = [5], c6 = [6], c7 = [7], c8 = [8], c9 = [9], c10 = [10], c11 = [11], c12 = [12], c13 = [13], c14 = [14], c15 = [15], c16 = [16], c17 = [17], c18 = [18], c19 = [19], c20 = [20], c21 = [21], c22 = [22], c23 = [23], c24 = [24], c25 = [25], c26 = [26], c27 = [27], c28 = [28], c29 = [29], c30 = [30], c31 = [31]),
    b = Table(c1 = [1], d2 = [2], d3 = [3], d4 = [4], d5 = [5], d6 = [6], d7 = [7], d8 = [8], d9 = [9], d10 = [10], d11 = [11], d12 = [12], d13 = [13], d14 = [14], d15 = [15], d16 = [16], d17 = [17], d18 = [18], d19 = [19], d20 = [20], d21 = [21], d22 = [22], d23 = [23], d24 = [24], d25 = [25], d26 = [26], d27 = [27], d28 = [28], d29 = [29], d30 = [30], d31 = [31])
  )
  for s in [:a,:b]
    t = getfield(ts, s)
    b = map(r -> r.c1 == 1, t)
  end
  return 0
end

It results in compile error messages. The first one starts with:

Verifier error #1: unresolved call from statement Core._apply_iterate(Base.iterate, Core.tuple, %new()::Vector{Int64})::Tuple{Vararg{Int64}}
Stacktrace:
  [1] _totuple(T::Type{NTuple{32, Int64}}, itr::Base.Generator{@NamedTuple{c1::Vector{Int64}, d2::Vector{Int64}, d3::Vector{Int64}, d4::Vector{Int64}, d5::Vector{Int64}, d6::Vector{Int64}, d7::Vector{Int64}, d8::Vector{Int64}, d9::Vector{Int64}, d10::Vector{Int64}, d11::Vector{Int64}, d12::Vector{Int64}, d13::Vector{Int64}, d14::Vector{Int64}, d15::Vector{Int64}, d16::Vector{Int64}, d17::Vector{Int64}, d18::Vector{Int64}, d19::Vector{Int64}, d20::Vector{Int64}, d21::Vector{Int64}, d22::Vector{Int64}, d23::Vector{Int64}, d24::Vector{Int64}, d25::Vector{Int64}, d26::Vector{Int64}, d27::Vector{Int64}, d28::Vector{Int64}, d29::Vector{Int64}, d30::Vector{Int64}, d31::Vector{Int64}, d32::Vector{Int64}}, TypedTables.var"#39#40"{Int64}})
  ...

How can I get this code compiled for juliac?
Do you have experience with that?
Is there ongoing work to make tuples containing more than 32 elements type-stable, also for getting it compiled with juliac?

2

is the code type unstable also when using Julia normally? In Ark.jl there are some tuples under the hood whose size depends on the size of the world (and can easily enough exceed the 32 element size), but I don’t experience any type instability in normal Julia usage even when I surpass thag threshold, I wonder if TypedTables.jl can’t be improved in some way in this regard. I can’t test at the moment what happens if a world surpasses that with Juliac, but will try.

3

I agree that it’s more complicated than “compiler gives up at 32”, though I don’t know how it really works. I’ve seen type inference exceed 32-element tuples, at least according to available reflection e.g. let N=42; code_warntype(NTuple{N, Float64}, (NTuple{N, Int64},)) end.

This specific example dispatched to this _totuple method that collects the generator’s elements into a vector before simply splatting it into a tuple expression. I have no idea how splatting works, but evidently it did better than giving up at Any; the verifier reports ::Tuple{Vararg{Int64}}. It statically maintained the vector’s element type, but the length is only known at runtime; as necessary as dynamic arrays are, FixedSizeArrays.jl had a point when it was announced with “what Array probably should have been.”

I couldn’t find an issue tracking this, but I have my doubts. That method has the argument T::Type{All32{E,N}}, All32 being an alias for all tuples with 32 or more elements of the same concrete type E. There is also a Any32 for heterogeneous tuples. These serve to limit how far other tuple-instantiating methods recursively inline themselves regardless of statically known lengths, thus limiting “code blowup” as commented above the aliases. This used to be a limit of 16, and it was increased specifically to allow compilation of GPU kernels that index straightforward high-order tensors for a user’s needs (Pull Request #40468 · JuliaLang/julia). It was an acceptable relaxation of a necessary limit, not an aspiration to arbitrarily increase static tuple sizes.

As far as I know from the little I’ve looked into how statically typed languages do tables, data types are determined at runtime. It makes sense that typical programs wouldn’t be compiled for tables with fixed column numbers or types. I don’t know what that’ll look like in a JuliaC future, but it’s probably not the way TypedTables work now.