# Correct way to dereference large memory?

**URL:** https://discourse.julialang.org/t/correct-way-to-dereference-large-memory/47395
**Category:** Performance
**Tags:** question
**Created:** [September 28, 2020, 9:46am UTC](https://discourse.julialang.org/t/correct-way-to-dereference-large-memory/47395 "2020-09-28T09:46:00Z")
**Posts on this page:** 7
**Page:** 1

<div class="post-metadata">

### Author: ![Ibb](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ibb/32/12083_2.png) [@Ibb](https://discourse.julialang.org/u/Ibb)
#### Post date: [September 28, 2020, 9:46am UTC](https://discourse.julialang.org/t/correct-way-to-dereference-large-memory/47395/1 "2020-09-28T09:46:01Z")

</div>

I want to read binary files into some large tuples by Ref, but my julia stuck at code below.

```julia
julia> a = Ref{NTuple{36000, UInt8}}()
julia> a[] #julia v1.5.2 will stuck here

```

---

<div class="post-metadata">

### Author: ![Sukera](https://avatars.discourse-cdn.com/v4/letter/s/ce7236/32.png) [@Sukera](https://discourse.julialang.org/u/Sukera)
#### Post date: [September 28, 2020, 10:13am UTC](https://discourse.julialang.org/t/correct-way-to-dereference-large-memory/47395/2 "2020-09-28T10:13:52Z")

</div>

Could you explain more about what you want to do exactly? The code you’ve posted just creates an `Ref` to some undefined data. Accesing the contents of that is expected to fail/not give you the data form a file.

Reading from a file into an `NTuple` can be done like this `read(<file>, NTuple{36000, UInt8})`. There’s probably a better way to do what you want to do though, like `read(<file>, 36000)` which returns the data as a vector of UInt8 (which is probably more convenient to work with than an NTuple).

---

<div class="post-metadata">

### Author: ![Ibb](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ibb/32/12083_2.png) [@Ibb](https://discourse.julialang.org/u/Ibb)
#### Post date: [September 28, 2020, 10:33am UTC](https://discourse.julialang.org/t/correct-way-to-dereference-large-memory/47395/3 "2020-09-28T10:33:04Z")

</div>

Thx for advise, but it seems that NTuple{36000, UInt8} doesn’t store Undef data. What I want to do is to read a binary file right into a large struct or tuple. Code below won’t work either.

```julia
julia> struct LargeStruct
         value::NTuple{36000, UInt8}
       end
julia> sizeof(LargeStruct) #36000
julia> large_struct = Ref{LargeStruct}()
julia> read!(<file>, large_struct)
julia> large_struct[].value[1]

```

---

<div class="post-metadata">

### Author: ![jakobnissen](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jakobnissen/32/13477_2.png) [@jakobnissen](https://discourse.julialang.org/u/jakobnissen)
#### Post date: [September 28, 2020, 10:44am UTC](https://discourse.julialang.org/t/correct-way-to-dereference-large-memory/47395/4 "2020-09-28T10:44:07Z")

</div>

The problem here is not actually the referencing or the memory. Your code is fine. It’s the fact that Julia’s compiler handles large immutable struct like tuples very badly. So badly in fact that large structs are unusable in Julia.  
Hopefully this will be adressed in the future - it’s certainly something I’ve seen mentioned many times, and is an annoying limitation to Julia.  
For now, the solution is to limit the size of immutable structs to, say, 100 fields, and use heap-allocated vectors instead, i.e. `Vector{UInt8}`.

---

<div class="post-metadata">

### Author: ![Ibb](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ibb/32/12083_2.png) [@Ibb](https://discourse.julialang.org/u/Ibb)
#### Post date: [September 28, 2020, 10:57am UTC](https://discourse.julialang.org/t/correct-way-to-dereference-large-memory/47395/5 "2020-09-28T10:57:20Z")

</div>

Thx a lot for your answer. I think I need to parse my binary files “field by field” now.

---

<div class="post-metadata">

### Author: ![stephancb](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stephancb/32/14243_2.png) [@stephancb](https://discourse.julialang.org/u/stephancb)
#### Post date: [September 28, 2020, 11:30am UTC](https://discourse.julialang.org/t/correct-way-to-dereference-large-memory/47395/6 "2020-09-28T11:30:25Z")

</div>

You might have a look at [Memory-mapped I/O · The Julia Language](https://docs.julialang.org/en/v1/stdlib/Mmap/)

---

<div class="post-metadata">

### Author: ![Henrique\_Becker](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/henrique_becker/32/15443_2.png) [@Henrique\_Becker](https://discourse.julialang.org/u/Henrique_Becker)
#### Post date: [September 28, 2020, 1:34pm UTC](https://discourse.julialang.org/t/correct-way-to-dereference-large-memory/47395/7 "2020-09-28T13:34:02Z")

</div>

uhhh, I may have missed something, but seems to me that the real problem here is that Tuple are immutable objects, so it does not make sense to create a uninitialized tuple object to then try to read data into it.

What others said is true, you should avoid using large Tuples, they are not performant. I think the threshold where performance takes a hit is 32+ elements, but I may be wrong.
