# RAM needed to initialise large matrices

**URL:** <https://discourse.julialang.org/t/ram-needed-to-initialise-large-matrices/78302>\
**Category:** General Usage\
**Tags:** memory, memory-allocation\
**Created:** [March 22, 2022, 8:04pm UTC](https://discourse.julialang.org/t/ram-needed-to-initialise-large-matrices/78302 "2022-03-22T20:04:12Z")\
**Posts on this page:** 12\
**Page:** 2

<div class="post-metadata">

**Author:** ![PetrKryslUCSD](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/petrkryslucsd/32/215825_2.png) [@PetrKryslUCSD](https://discourse.julialang.org/u/PetrKryslUCSD)\
**Post date:** [March 22, 2022, 11:00pm UTC](https://discourse.julialang.org/t/ram-needed-to-initialise-large-matrices/78302/22 "2022-03-22T23:00:16Z")

</div>

If you need to store 0 and 1, you only need one bit per matrix entry. Isn’t that right?

---

<div class="post-metadata">

**Author:** ![fipelle](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fipelle/32/4772_2.png) [@fipelle](https://discourse.julialang.org/u/fipelle)\
**Post date:** [March 22, 2022, 11:30pm UTC](https://discourse.julialang.org/t/ram-needed-to-initialise-large-matrices/78302/23 "2022-03-22T23:30:00Z")

</div>

That’s true! Also, I should be able to use `mul!(...)` to perform operations on bit matrices while ensuring that the output is also a BitMatrix.

However, would that be more efficient than using SparseArray?

---

<div class="post-metadata">

**Author:** ![Henrique\_Becker](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/henrique_becker/32/15443_2.png) [@Henrique\_Becker](https://discourse.julialang.org/u/Henrique_Becker)\
**Post date:** [March 22, 2022, 11:34pm UTC](https://discourse.julialang.org/t/ram-needed-to-initialise-large-matrices/78302/24 "2022-03-22T23:34:51Z")

</div>

Sparse representations need to store the indexes of the nonzero positions, so they probably will gain no advantage from making it only zeros or ones.

I would recommend profiling each alternative, it depends on how sparse is the matrix. If your matrix has a nonzero ratio higher than 1/32 then the `BitMatrix` may be a good idea.

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [March 23, 2022, 1:31am UTC](https://discourse.julialang.org/t/ram-needed-to-initialise-large-matrices/78302/25 "2022-03-23T01:31:23Z")

</div>

> [@fipelle](#):
>
> The last one is a selection matrix (ones and zeros, mostly zeros) of size 500,000 x 500,000.

If you use `SparseArrays` properly, then the storage will be proportional to the number of nonzero entries. How many nonzeros entries do you have?

(Note that you would _never_ call `zeros`, which allocates a dense matrix and stores all of the zeros explicitly.)

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [March 23, 2022, 1:35am UTC](https://discourse.julialang.org/t/ram-needed-to-initialise-large-matrices/78302/26 "2022-03-23T01:35:13Z")

</div>

> [@Henrique\_Becker](#):
>
> I would recommend profiling each alternative, it depends on how sparse is the matrix. If your matrix has a nonzero ratio higher than 1/32 then the `BitMatrix` may be a good idea.

If you have a 500000 \times 500000 matrix with that little sparsity, then even a `BitMatrix` requires over 30GB of memory, and the original post said that a “handful” of such matrices are required, which will quickly become impractical.

The basic point here is that if you have such huge matrices, you really need to think more carefully about what you are doing; you almost never want to store or compute with the whole matrix explicitly. You almost always want to exploit sparsity or some other structure.

---

<div class="post-metadata">

**Author:** ![Henrique\_Becker](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/henrique_becker/32/15443_2.png) [@Henrique\_Becker](https://discourse.julialang.org/u/Henrique_Becker)\
**Post date:** [March 23, 2022, 1:43am UTC](https://discourse.julialang.org/t/ram-needed-to-initialise-large-matrices/78302/27 "2022-03-23T01:43:53Z")

</div>

Oh, sorry, I though the original matrix was 80Gib, so a 32~64x reduction could bring it to something workable, but now I see that it was the 100000x100000 matrix that was that size.

---

<div class="post-metadata">

**Author:** ![c42f](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/c42f/32/52842_2.png) [@c42f](https://discourse.julialang.org/u/c42f)\
**Post date:** [March 23, 2022, 1:48am UTC](https://discourse.julialang.org/t/ram-needed-to-initialise-large-matrices/78302/28 "2022-03-23T01:48:40Z")

</div>

> [@Oscar\_Smith](#):
>
> Calling `zeros` doesn’t write to memory, so the OS doesn’t actually have to give you any memory.

I don’t think this is true? Calling `zeros(UInt8, 1000_000_000)` seems to first invoke the `undef` constructor to get `v = Vector{UInt8}(undef, (1000_000_000,))` which doesn’t have to allocate physical memory when it’s done via `mmap`. But this is followed by `fill!(v, zero(UInt8))` which would touch every byte in the array, causing many OS page faults which in turn allocate physical memory pages.

For another interesting thread on page faults, see here: [Julia slower in Windows - #16 by c42f](https://discourse.julialang.org/t/julia-slower-in-windows/21027/16)

(Edit: As a side note - it seems technically possible to do lazy zero fill in (a) the case that a large array happens to be allocated with [`mmap` and `MAP_ANONYMOUS`](https://man7.org/linux/man-pages/man2/mmap.2.html) somewhere deep in the allocator and (b) `zero(eltype(a))` happens to be a bits type with a bit pattern of zeros. I think this would only help in specialized circumstances, though.)

---

<div class="post-metadata">

**Author:** ![fipelle](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fipelle/32/4772_2.png) [@fipelle](https://discourse.julialang.org/u/fipelle)\
**Post date:** [March 23, 2022, 1:56am UTC](https://discourse.julialang.org/t/ram-needed-to-initialise-large-matrices/78302/29 "2022-03-23T01:56:05Z")

</div>

> [@stevengj](#):
>
> How many nonzeros entries do you have?

About 10,000 ones and the remaining elements are zeros. It seems a clear case for SparseArray.

I will leave the post open for a few days, so should you have more suggestions please add them below!

---

<div class="post-metadata">

**Author:** ![Sukera](https://avatars.discourse-cdn.com/v4/letter/s/ce7236/32.png) [@Sukera](https://discourse.julialang.org/u/Sukera)\
**Post date:** [March 23, 2022, 8:29am UTC](https://discourse.julialang.org/t/ram-needed-to-initialise-large-matrices/78302/30 "2022-03-23T08:29:08Z")

</div>

> [@c42f](#):
>
> (Edit: As a side note - it seems technically possible to do lazy zero fill in (a) the case that a large array happens to be allocated with [`mmap` and `MAP_ANONYMOUS`](https://man7.org/linux/man-pages/man2/mmap.2.html) somewhere deep in the allocator and (b) `zero(eltype(a))` happens to be a bits type with a bit pattern of zeros. I think this would only help in specialized circumstances, though.)

For a detailed discussion on the challenges that would crop up almost immediately when trying this, check out [Faster zeros with calloc](https://discourse.julialang.org/t/faster-zeros-with-calloc/69860). The TL;DR is that the cost of initialization doesn’t vanish but gets shifted to the first write instead and that growing such an array immediately looses the “zero initialized” property.

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [March 23, 2022, 11:50am UTC](https://discourse.julialang.org/t/ram-needed-to-initialise-large-matrices/78302/31 "2022-03-23T11:50:33Z")

</div>

> [@fipelle](#):
>
> About 10,000 ones and the remaining elements are zeros. It seems a clear case for SparseArray.

Definitely. (A sparse array should take \< 1MB of memory.)

One final tip is that you should always construct a sparse array all at once if possible, e.g. using the `sparse(I, J, V)` constructor. Adding or removing individual elements one at a time is costly.

---

<div class="post-metadata">

**Author:** ![tbeason](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tbeason/32/15898_2.png) [@tbeason](https://discourse.julialang.org/u/tbeason)\
**Post date:** [March 23, 2022, 1:39pm UTC](https://discourse.julialang.org/t/ram-needed-to-initialise-large-matrices/78302/32 "2022-03-23T13:39:06Z")

</div>

> [@stevengj](#):
>
> In double precision, an n \times n nmatrix requires 8n^2 bytes.

This is a very helpful statement for me! Perhaps others knew this already. What about integers or strings? Where can I find that information?

---

<div class="post-metadata">

**Author:** ![Henrique\_Becker](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/henrique_becker/32/15443_2.png) [@Henrique\_Becker](https://discourse.julialang.org/u/Henrique_Becker)\
**Post date:** [March 23, 2022, 1:50pm UTC](https://discourse.julialang.org/t/ram-needed-to-initialise-large-matrices/78302/33 "2022-03-23T13:50:00Z")

</div>

I mean, a matrix is `n` times `n`, if double precision is used then each element is 8 bytes, so `8n^2`.

`Int32` is 4 bytes, `Int64` is 8 bytes. Almost every “normal and modern” computer today is 64bits, so Julia will be using `Int64` (8 bytes).

`String` is a little more complicated, but if I am not wrong in general they use [UTF-8](https://en.wikipedia.org/wiki/UTF-8), what means that for [ASCII](https://en.wikipedia.org/wiki/ASCII) (i.e., most characters normally used in english) it is one byte per character (plus a static cost of a zero byte at the end of the string and maybe an `Int` with the total size too?). But if characters from other languages, emojis, etc… are used, then the that specific character is represented by 2-4 bytes in the sequence.

I believe this information is somewhere in the docs? I have to admit that this kinda of thing was inculcated into me by my graduation in computer science, so I do not have a source at the moment.

[Previous page](https://discourse.julialang.org/t/ram-needed-to-initialise-large-matrices/78302.md?page=1)