# Memory usage with SharedArray: \`htop\` shows allocation on all processes?

**URL:** <https://discourse.julialang.org/t/memory-usage-with-sharedarray-htop-shows-allocation-on-all-processes/413>\
**Category:** General Usage\
**Tags:** question\
**Created:** [November 18, 2016, 2:04pm UTC](https://discourse.julialang.org/t/memory-usage-with-sharedarray-htop-shows-allocation-on-all-processes/413 "2016-11-18T14:04:52Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![floswald](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/floswald/32/195_2.png) [@floswald](https://discourse.julialang.org/u/floswald)\
**Post date:** [November 18, 2016, 2:04pm UTC](https://discourse.julialang.org/t/memory-usage-with-sharedarray-htop-shows-allocation-on-all-processes/413/1 "2016-11-18T14:04:52Z")

</div>

Hi all,  
i’m trying to understand how much memory I am consuming on a single machine when allocating a shared array. i show `htop` output after each step.

- launch julia

```julia
  PID USER PRI NI VIRT RES S CPU% MEM% TIME+ Command
25353 florian.o 40 0 10.7G 165M ? 0.0 1.0 0:00.51 julia

```

- create an array of shared array on a single processor

```julia
x= [SharedArray(Float64,(100,5,8,5,2,30),pids=workers()) for i in 1:60];

# htop
  PID USER PRI NI VIRT RES S CPU% MEM% TIME+ Command
25532 florian.o 26 0 11.8G 192M ? 0.0 1.2 0:02.16 julia

```

- in a new session, create the shared array on 4 worker processes

```julia
# new session
addprocs(4)
x= [SharedArray(Float64,(100,5,8,5,2,30),pids=workers()) for i in 1:60];

# htop
  PID USER PRI NI VIRT RES S CPU% MEM% TIME+ Command                                                                                                                                                                                                                               
25562 florian.o 26 0 11.3G 206M ? 0.0 1.3 0:04.24 julia                                                                                                                                                                                                                                 
25567 florian.o 40 0 11.3G 192M ? 0.0 1.2 0:01.83 /Applications/Julia-0.5.app/Contents/Resources/julia/bin/julia -Ccore2 -J/Applications/Julia-0.5.app/Contents/Resources/julia/lib/julia/sys.dylib --compile=yes --depwarn=yes --bind-to 127.0.0.1 --worker jkYvc9op9KaIHMRn
25566 florian.o 40 0 11.3G 192M ? 0.0 1.2 0:01.86 /Applications/Julia-0.5.app/Contents/Resources/julia/bin/julia -Ccore2 -J/Applications/Julia-0.5.app/Contents/Resources/julia/lib/julia/sys.dylib --compile=yes --depwarn=yes --bind-to 127.0.0.1 --worker jkYvc9op9KaIHMRn
25565 florian.o 40 0 11.3G 191M ? 0.0 1.2 0:01.86 /Applications/Julia-0.5.app/Contents/Resources/julia/bin/julia -Ccore2 -J/Applications/Julia-0.5.app/Contents/Resources/julia/lib/julia/sys.dylib --compile=yes --depwarn=yes --bind-to 127.0.0.1 --worker jkYvc9op9KaIHMRn
25564 florian.o 40 0 11.2G 192M ? 0.0 1.2 0:01.78 /Applications/Julia-0.5.app/Contents/Resources/julia/bin/julia -Ccore2 -J/Applications/Julia-0.5.app/Contents/Resources/julia/lib/julia/sys.dylib --compile=yes --depwarn=yes --bind-to 127.0.0.1 --worker jkYvc9op9KaIHMRn

```

How do I read this? I can see that on a single process, creating that array consumes about 192M. If I share the array on 4 processes, it seems from `htop` that _each_ process consumes that same full amount of memory. I was under the impression that the idea of `SharedArray` was to store the object in a central location and let all processes use it. I would have expected to see for the worker processes in the `RES` column numbers like `165+(192-165)/4 = 171` or some thing like that. or is the `htop` output misleading here?

---

<div class="post-metadata">

**Author:** ![yuyichao](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yuyichao/32/20_2.png) [@yuyichao](https://discourse.julialang.org/u/yuyichao)\
**Post date:** [November 18, 2016, 2:30pm UTC](https://discourse.julialang.org/t/memory-usage-with-sharedarray-htop-shows-allocation-on-all-processes/413/2 "2016-11-18T14:30:05Z")

</div>

Shared mapping will show up in every processes that uses them. No single process owns the memory.

---

<div class="post-metadata">

**Author:** ![floswald](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/floswald/32/195_2.png) [@floswald](https://discourse.julialang.org/u/floswald)\
**Post date:** [November 18, 2016, 2:33pm UTC](https://discourse.julialang.org/t/memory-usage-with-sharedarray-htop-shows-allocation-on-all-processes/413/3 "2016-11-18T14:33:34Z")

</div>

Ok, just to be clear: I don’t consume 4_192 MB of memory as displayed here but just 1_192MB and it this gets displayed for each sharing process.

---

<div class="post-metadata">

**Author:** ![ihnorton](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ihnorton/32/26_2.png) [@ihnorton](https://discourse.julialang.org/u/ihnorton)\
**Post date:** [November 18, 2016, 2:46pm UTC](https://discourse.julialang.org/t/memory-usage-with-sharedarray-htop-shows-allocation-on-all-processes/413/4 "2016-11-18T14:46:30Z")

</div>

How are you running `htop`? And what OS? The linux documentation lists an `SHR` column that is supposed to show shared pages, but it may not be supported on all OS.

---

<div class="post-metadata">

**Author:** ![yuyichao](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yuyichao/32/20_2.png) [@yuyichao](https://discourse.julialang.org/u/yuyichao)\
**Post date:** [November 18, 2016, 2:51pm UTC](https://discourse.julialang.org/t/memory-usage-with-sharedarray-htop-shows-allocation-on-all-processes/413/5 "2016-11-18T14:51:18Z")

</div>

The non shared memory won’t be, well, shared, only the shared map (including the loaded code, the read only data, and the shared array). So you are consuming more than 1 \* 192M (since part of 192M is not shared) but the memory occupied by the array will be shared.

Also note that since you have not accessed the array yet, it will not show up in the `RES`, which only accounts for the memory loaded into the physical memory and you should expect `RES` to go up when you starts to access the memory.

htop 2.0.2 also (by default?) show the shared mapping but yours doesn’t seem to do that.

---

<div class="post-metadata">

**Author:** ![floswald](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/floswald/32/195_2.png) [@floswald](https://discourse.julialang.org/u/floswald)\
**Post date:** [November 18, 2016, 2:54pm UTC](https://discourse.julialang.org/t/memory-usage-with-sharedarray-htop-shows-allocation-on-all-processes/413/6 "2016-11-18T14:54:07Z")

</div>

this on macOS. the “memory” column in Activity Monitor shows roughly the same numbers. my htop is

```julia
➜ bk.jl git:(itp-v) ✗ brew info htop
htop: stable 2.0.2 (bottled), HEAD

```

here is `available columns` in my `htop`. anything useful?

```julia
Setup Active Columns Available Columns
Meters PID PID - Process/thread ID
Display options USER STATE - Process state (S sleeping, R running, D disk, Z zombie, T traced, W paging)
Colors PRIORITY PPID - Parent process ID
Columns NICE PGRP - Process group ID
                 M_SIZE SESSION - Process's session ID
                 M_RESIDENT TTY_NR - Controlling terminal
                 STATE TPGID - Process ID of the fg process group of the controlling terminal
                 PERCENT_CPU MINFLT - Number of minor faults which have not required loading a memory page from disk
                 PERCENT_MEM MAJFLT - Number of major faults which have required loading a memory page from disk
                 TIME PRIORITY - Kernel's internal priority for the process
                 Command NICE - Nice value (the higher the value, the more it lets other processes take priority)
                                      STARTTIME - Time the process was started
                                      PROCESSOR - Id of the CPU the process last executed on
                                      M_SIZE - Total program size in virtual memory
                                      M_RESIDENT - Resident set size, size of the text and data sections, plus stack usage
                                      ST_UID - User ID of the process owner
                                      PERCENT_CPU - Percentage of the CPU time the process used in the last sampling
                                      PERCENT_MEM - Percentage of the memory the process is using, based on resident memory size
                                      USER - Username of the process owner (or user ID if name cannot be determined)
                                      TIME - Total time the process has spent in user and system time
                                      NLWP - Number of threads in the process
                                      TGID - Thread group ID (i.e. process ID)

```

Anyway, I think I get the gist of it. Thanks for explaining!