Using large NTuples makes Julia hang

General Usage
, , ,
9

Ok, digging even further, even creation of the array is (sometimes?!) fine (all in a fresh session):

[sukera@tempman ~]$ julia -q
julia> struct Foo
           id:: UInt64
           tag::NTuple{32, UInt8}
           start_comment::NTuple{4096, UInt8}
           end_comment::NTuple{4096, UInt8}
       end

julia> string2ntuple(s::String, len) = ntuple(i -> i <= ncodeunits(s) ? codeunit(s, i) : 0x0, len)
string2ntuple (generic function with 1 method)

julia> @time dataset = Vector{Foo}(undef, 1);
  0.000011 seconds (1 allocation: 8.188 KiB)

julia> @time dataset[1] = Foo(
               0,
               string2ntuple("test_tag", 32),
               string2ntuple("start_comment", 4096),
               string2ntuple("end_comment", 4096),
           ); # silence output
  0.022675 seconds (4.25 k allocations: 344.307 KiB, 98.39% compilation time)

julia> @time dataset[1] = Foo(
               0,
               string2ntuple("test_tag", 32),
               string2ntuple("start_comment", 4096),
               string2ntuple("end_comment", 4096),
           ); # silence output
  0.000272 seconds (9 allocations: 89.016 KiB)

julia> @time data2 = Foo[];
  0.000010 seconds (1 allocation: 64 bytes)

julia> @time push!(data2, Foo(
               0,
               string2ntuple("test_tag", 32),
               string2ntuple("start_comment", 4096),
               string2ntuple("end_comment", 4096),
           )); # silence output
  0.014009 seconds (9.06 k allocations: 700.702 KiB, 97.74% compilation time)

You may notice that I’ve silenced all outputs - seems like this is what’s causing the slowdown, so there’s an issue there! Creating & typing the array inline (like dataset = Foo[data]) is also still borked.

For reference:

julia> @time show(stdout, dataset)
Foo[Foo(0x0000000000000000, [...] # I've ommited all the empty data
)] 32.848714 seconds (588.41 k allocations: 60.553 MiB, 0.04% gc time, 99.28% compilation time)

so that’s just for showing the result to the REPL.

julia> versioninfo()
Julia Version 1.9.0-DEV.171
Commit 842c27fad3 (2022-03-11 06:44 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: 4 × Intel(R) Core(TM) i7-6600U CPU @ 2.60GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-13.0.1 (ORCJIT, skylake)
  Threads: 4 on 4 virtual cores
5 Likes