# RAM needed to initialise large matrices

**URL:** <https://discourse.julialang.org/t/ram-needed-to-initialise-large-matrices/78302>\
**Category:** General Usage\
**Tags:** memory, memory-allocation\
**Created:** [March 22, 2022, 8:04pm UTC](https://discourse.julialang.org/t/ram-needed-to-initialise-large-matrices/78302 "2022-03-22T20:04:12Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![fipelle](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fipelle/32/4772_2.png) [@fipelle](https://discourse.julialang.org/u/fipelle)\
**Post date:** [March 22, 2022, 8:04pm UTC](https://discourse.julialang.org/t/ram-needed-to-initialise-large-matrices/78302/1 "2022-03-22T20:04:12Z")

</div>

Hi, I have a basic doubt on memory management. I am currently working on a high-dimensional problem in which I would need to use a handful of matrices with size 500,000 x 500,000. I was wondering how much ram is needed to initialise similar matrices of floats.

I have tried using `varinfo()` after having initialised a matrix `A=zeros(100000, 100000)` to get a rough idea, but I am not sure I understand the output. In fact, it says that `A` requires about 74.506 GiB (which sounds odd to me, considering that I am testing it on a MacBook Air with substantially less RAM).

How should I read the output of `varinfo()`? Also, is it enough to multiply it by 5 to get a rough idea of the RAM required for some matrix `B=zeros(500000, 500000)`?

---

<div class="post-metadata">

**Author:** ![goerch](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/goerch/32/29122_2.png) [@goerch](https://discourse.julialang.org/u/goerch)\
**Post date:** [March 22, 2022, 8:09pm UTC](https://discourse.julialang.org/t/ram-needed-to-initialise-large-matrices/78302/2 "2022-03-22T20:09:30Z")

</div>

> [@fipelle](#):
>
> `A=zeros(100000, 100000)`

You don’t get me to try this on windows. Back on the envelope calculation gives me a memory requirement of 80 GB. So for 500.000 x 500.000 you should multiply by 25.

Edit: checked it: clearly a troll.

Edit: @fipelle : you can’t risk system integrity for unsuspecting observers.

---

<div class="post-metadata">

**Author:** ![jling](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jling/32/212909_2.png) [@jling](https://discourse.julialang.org/u/jling)\
**Post date:** [March 22, 2022, 8:13pm UTC](https://discourse.julialang.org/t/ram-needed-to-initialise-large-matrices/78302/3 "2022-03-22T20:13:07Z")

</div>

```julia
julia> A=zeros(100, 100);

julia> varinfo()
  name size summary                               
  –––––––––––––––– ––––––––––– ––––––––––––––––––––––––––––––––––––––
  A 78.164 KiB 100×100 Matrix{Float64} 

julia> 100*100*64/8/1024
78.125

```

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [March 22, 2022, 8:18pm UTC](https://discourse.julialang.org/t/ram-needed-to-initialise-large-matrices/78302/4 "2022-03-22T20:18:11Z")

</div>

> [@fipelle](#):
>
> I am currently working on a high-dimensional problem in which I would need to use a handful of matrices with size 500,000 x 500,000. I was wondering how much ram is needed to initialise similar matrices of floats.

In double precision, an n \times n matrix requires 8n^2 bytes.

In short, you won’t be able to handle n = 500,000 (without a huge supercomputer), since that corresponds to **2TB** of memory. People working with such large matrices almost invariably exploit some special structure, e.g. sparsity (if your matrix is mostly zero). We could give you more specific advice, but we’d need to know more about your problem.

---

<div class="post-metadata">

**Author:** ![fipelle](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fipelle/32/4772_2.png) [@fipelle](https://discourse.julialang.org/u/fipelle)\
**Post date:** [March 22, 2022, 9:47pm UTC](https://discourse.julialang.org/t/ram-needed-to-initialise-large-matrices/78302/5 "2022-03-22T21:47:39Z")

</div>

Thank you!

> [@stevengj](#):
>
> sparsity (if your matrix is mostly zero).

That’s exactly what I am trying to do for handling the last huge matrix in my problem - I did manage to reduce the other matrices into smaller blocks. The last one is a selection matrix (ones and zeros, mostly zeros) of size 500,000 x 500,000.

Any suggestion? I am trying to use [Sparse Arrays · The Julia Language](https://docs.julialang.org/en/v1/stdlib/SparseArrays/) but a specialised structure for selection matrices would certainly be better.

---

<div class="post-metadata">

**Author:** ![goerch](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/goerch/32/29122_2.png) [@goerch](https://discourse.julialang.org/u/goerch)\
**Post date:** [March 22, 2022, 9:51pm UTC](https://discourse.julialang.org/t/ram-needed-to-initialise-large-matrices/78302/6 "2022-03-22T21:51:31Z")

</div>

> [@fipelle](#):
>
> That’s exactly what I am trying to do

That is not what you are trying to do with `zeros`.

---

<div class="post-metadata">

**Author:** ![fipelle](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fipelle/32/4772_2.png) [@fipelle](https://discourse.julialang.org/u/fipelle)\
**Post date:** [March 22, 2022, 9:52pm UTC](https://discourse.julialang.org/t/ram-needed-to-initialise-large-matrices/78302/7 "2022-03-22T21:52:34Z")

</div>

No, I was trying to get a rough idea of the memory needed for a matrix with size 500,000 x 500,000 - as indicated in the OP.

---

<div class="post-metadata">

**Author:** ![goerch](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/goerch/32/29122_2.png) [@goerch](https://discourse.julialang.org/u/goerch)\
**Post date:** [March 22, 2022, 9:53pm UTC](https://discourse.julialang.org/t/ram-needed-to-initialise-large-matrices/78302/8 "2022-03-22T21:53:20Z")

</div>

But others might try that on their system?

---

<div class="post-metadata">

**Author:** ![goerch](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/goerch/32/29122_2.png) [@goerch](https://discourse.julialang.org/u/goerch)\
**Post date:** [March 22, 2022, 9:54pm UTC](https://discourse.julialang.org/t/ram-needed-to-initialise-large-matrices/78302/10 "2022-03-22T21:54:49Z")

</div>

Forcing system instability due to OOM. In my eyes that is not OK.

---

<div class="post-metadata">

**Author:** ![fipelle](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fipelle/32/4772_2.png) [@fipelle](https://discourse.julialang.org/u/fipelle)\
**Post date:** [March 22, 2022, 10:01pm UTC](https://discourse.julialang.org/t/ram-needed-to-initialise-large-matrices/78302/11 "2022-03-22T22:01:23Z")

</div>

It should be evident from the history of my account that I am using this discourse properly and certainly not to troll other people - as I have just noticed you mentioned on top. While quite odd, my MacBook does not crash when initialising that matrix - and I am still wondering why btw.

If you have relevant suggestions, I could really use some help. However, I do not think that polluting this post with similar conspiracies is of any use.

---

<div class="post-metadata">

**Author:** ![goerch](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/goerch/32/29122_2.png) [@goerch](https://discourse.julialang.org/u/goerch)\
**Post date:** [March 22, 2022, 10:04pm UTC](https://discourse.julialang.org/t/ram-needed-to-initialise-large-matrices/78302/12 "2022-03-22T22:04:13Z")

</div>

> [@fipelle](#):
>
> my MacBook does not crash when initialising that matrix.

This doesn’t apply to Windows as far as I can tell (I checked). I accept this as kind of an apology.

Edit: but this could be quite a serious difference of memory management between the supported platforms.

Another edit: I didn’t really wait for my system to crash, only observed system monitor showing more and more memory being allocated.

---

<div class="post-metadata">

**Author:** ![Jeff\_Emanuel](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jeff_emanuel/32/15440_2.png) [@Jeff\_Emanuel](https://discourse.julialang.org/u/Jeff_Emanuel)\
**Post date:** [March 22, 2022, 10:29pm UTC](https://discourse.julialang.org/t/ram-needed-to-initialise-large-matrices/78302/13 "2022-03-22T22:29:01Z")

</div>

> [@fipelle](#):
>
> and I am still wondering why btw

virtual memory plus swap file sufficiently larger than physical ram.

---

<div class="post-metadata">

**Author:** ![fipelle](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fipelle/32/4772_2.png) [@fipelle](https://discourse.julialang.org/u/fipelle)\
**Post date:** [March 22, 2022, 10:30pm UTC](https://discourse.julialang.org/t/ram-needed-to-initialise-large-matrices/78302/14 "2022-03-22T22:30:06Z")

</div>

Thank you Jeff! Would you please expand on that?

---

<div class="post-metadata">

**Author:** ![Oscar\_Smith](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oscar_smith/32/25343_2.png) [@Oscar\_Smith](https://discourse.julialang.org/u/Oscar_Smith)\
**Post date:** [March 22, 2022, 10:37pm UTC](https://discourse.julialang.org/t/ram-needed-to-initialise-large-matrices/78302/15 "2022-03-22T22:37:31Z")

</div>

What’s happening here is that you are requesting the memory, but the OS only actually gives you physical memory when you write to it. Calling `zeros` doesn’t write to memory, so the OS doesn’t actually have to give you any memory.

---

<div class="post-metadata">

**Author:** ![Jeff\_Emanuel](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jeff_emanuel/32/15440_2.png) [@Jeff\_Emanuel](https://discourse.julialang.org/u/Jeff_Emanuel)\
**Post date:** [March 22, 2022, 10:39pm UTC](https://discourse.julialang.org/t/ram-needed-to-initialise-large-matrices/78302/16 "2022-03-22T22:39:26Z")

</div>

In modern computer systems, a process’s memory address space is mapped to physical ram through page tables. Blocks of memory (aka pages) can be offloaded to disk in the swap or paging file, and reloaded later as needed.

---

<div class="post-metadata">

**Author:** ![goerch](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/goerch/32/29122_2.png) [@goerch](https://discourse.julialang.org/u/goerch)\
**Post date:** [March 22, 2022, 10:39pm UTC](https://discourse.julialang.org/t/ram-needed-to-initialise-large-matrices/78302/17 "2022-03-22T22:39:40Z")

</div>

> [@Jeff\_Emanuel](#):
>
> virtual memory plus swap file sufficiently larger than physical ram.

So @fipelle should see indications of this going on in his system monitor, too? (i.e. not that a big difference in memory management between Windows and Mac OS then)

---

<div class="post-metadata">

**Author:** ![fipelle](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fipelle/32/4772_2.png) [@fipelle](https://discourse.julialang.org/u/fipelle)\
**Post date:** [March 22, 2022, 10:42pm UTC](https://discourse.julialang.org/t/ram-needed-to-initialise-large-matrices/78302/18 "2022-03-22T22:42:37Z")

</div>

I see. I am still going to use a sparse representation, but just for the sake of clarity: as long as there’s enough space on my hard disk it should work, right?

---

<div class="post-metadata">

**Author:** ![Jeff\_Emanuel](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jeff_emanuel/32/15440_2.png) [@Jeff\_Emanuel](https://discourse.julialang.org/u/Jeff_Emanuel)\
**Post date:** [March 22, 2022, 10:45pm UTC](https://discourse.julialang.org/t/ram-needed-to-initialise-large-matrices/78302/19 "2022-03-22T22:45:07Z")

</div>

It depends on the swap file configuration. The OS usually puts a limit on the swap file size. Paging will lead to very poor performance.

---

<div class="post-metadata">

**Author:** ![goerch](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/goerch/32/29122_2.png) [@goerch](https://discourse.julialang.org/u/goerch)\
**Post date:** [March 22, 2022, 10:45pm UTC](https://discourse.julialang.org/t/ram-needed-to-initialise-large-matrices/78302/20 "2022-03-22T22:45:45Z")

</div>

Do we have a difference in [demand paging](https://en.wikipedia.org/wiki/Demand_paging) here?

---

<div class="post-metadata">

**Author:** ![Jeff\_Emanuel](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jeff_emanuel/32/15440_2.png) [@Jeff\_Emanuel](https://discourse.julialang.org/u/Jeff_Emanuel)\
**Post date:** [March 22, 2022, 10:47pm UTC](https://discourse.julialang.org/t/ram-needed-to-initialise-large-matrices/78302/21 "2022-03-22T22:47:58Z")

</div>

From the sources:

> **[About the Virtual Memory System](https://developer.apple.com/library/archive/documentation/Performance/Conceptual/ManagingMemory/Articles/AboutMemory.html)**
>
> Guidelines for improving the performance of memory-related code.

[https://answers.microsoft.com/en-us/windows/forum/all/physical-and-virtual-memory-in-windows-10/e36fb5bc-9ac8-49af-951c-e7d39b979938](https://answers.microsoft.com/en-us/windows/forum/all/physical-and-virtual-memory-in-windows-10/e36fb5bc-9ac8-49af-951c-e7d39b979938)

[Next page](https://discourse.julialang.org/t/ram-needed-to-initialise-large-matrices/78302.md?page=2)