Correct way to dereference large memory?

General Usage Performance
1

I want to read binary files into some large tuples by Ref, but my julia stuck at code below.

julia> a = Ref{NTuple{36000, UInt8}}()
julia> a[] #julia v1.5.2 will stuck here
2

Could you explain more about what you want to do exactly? The code you’ve posted just creates an Ref to some undefined data. Accesing the contents of that is expected to fail/not give you the data form a file.

Reading from a file into an NTuple can be done like this read(<file>, NTuple{36000, UInt8}). There’s probably a better way to do what you want to do though, like read(<file>, 36000) which returns the data as a vector of UInt8 (which is probably more convenient to work with than an NTuple).

1 Like
3

Thx for advise, but it seems that NTuple{36000, UInt8} doesn’t store Undef data. What I want to do is to read a binary file right into a large struct or tuple. Code below won’t work either.

julia> struct LargeStruct
         value::NTuple{36000, UInt8}
       end
julia> sizeof(LargeStruct) #36000
julia> large_struct = Ref{LargeStruct}()
julia> read!(<file>, large_struct)
julia> large_struct[].value[1]
4

The problem here is not actually the referencing or the memory. Your code is fine. It’s the fact that Julia’s compiler handles large immutable struct like tuples very badly. So badly in fact that large structs are unusable in Julia.
Hopefully this will be adressed in the future - it’s certainly something I’ve seen mentioned many times, and is an annoying limitation to Julia.
For now, the solution is to limit the size of immutable structs to, say, 100 fields, and use heap-allocated vectors instead, i.e. Vector{UInt8}.

2 Likes
5

Thx a lot for your answer. I think I need to parse my binary files “field by field” now.

6

You might have a look at Memory-mapped I/O · The Julia Language

7

uhhh, I may have missed something, but seems to me that the real problem here is that Tuple are immutable objects, so it does not make sense to create a uninitialized tuple object to then try to read data into it.

What others said is true, you should avoid using large Tuples, they are not performant. I think the threshold where performance takes a hit is 32+ elements, but I may be wrong.

3 Likes