Using large NTuples makes Julia hang

Maurizio_Tomasi
Sat, 02 Apr 2022 06:38:56 +0000
General Usage

Julia 1.7.2 on Linux x86_64 seems to hang when instantiating structures containing large NTuple types, as the following code shows:

struct Foo
    id:: UInt64
    tag::NTuple{32, UInt8}
    start_comment::NTuple{4096, UInt8}
    end_comment::NTuple{4096, UInt8}
end

string2ntuple(s::String, len) = ntuple(i -> i <= ncodeunits(s) ? codeunit(s, i) : '\0', len)

println("Going to create an array of Foo…")
dataset = [
    Foo(
        0,
        string2ntuple("test_tag", 32),
        string2ntuple("start_comment", 4096),
        string2ntuple("end_comment", 4096),
    ),
]
println("…done")

This code prints Going to create an array of Foo… and then hangs; using htop, I see that Julia is allocating a awful lot of memory (several GBs), and I have to press Ctrl+C to get back to the REPL.

The reason why I am creating these large NTuple types is because I need to create HDF5 datasets based on compound types, and a few of these require fixed-length strings with the same sizes as the ones in Foo above. To do this, I need to create a compound data type that has the same memory layout as the following C++ struct:

struct Foo {
    uint64_t id;
    uint8_t tag[32];
    uint8_t start_comment[4096];
    uint8_t end_comment[4096];
}

My Julia code closely follows the approach shown in the tests included in the HDF5.jl library, where however the size of the NTuple is significantly smaller (20 elements).

So, I have two questions:

1. Is the fact that Julia requires a lot of memory to compile that code expected, or should I file a bug?

2. Is there some other alternative approach that I could employ to define Foo in a way that matches the C++ definition?

24 posts - 8 participants