# How to track total memory usage of Julia process over time

**URL:** https://discourse.julialang.org/t/how-to-track-total-memory-usage-of-julia-process-over-time/91167
**Category:** Profiling
**Tags:** memory-allocation
**Created:** [December 3, 2022, 6:06am UTC](https://discourse.julialang.org/t/how-to-track-total-memory-usage-of-julia-process-over-time/91167 "2022-12-03T06:06:13Z")
**Posts on this page:** 1
**Showing post:** 15

<div class="post-metadata">

### Author: ![Palli](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/palli/32/3380_2.png) [@Palli](https://discourse.julialang.org/u/Palli)
#### Post date: [December 6, 2022, 4:40pm UTC](https://discourse.julialang.org/t/how-to-track-total-memory-usage-of-julia-process-over-time/91167/15 "2022-12-06T16:40:40Z")

</div>

> [@sloede](#):
>
> It seems that the numbers obtained from Julia directly are missing quite a bit of untracked memory

Are you sure about that? If you’re thinking about Max. RSS, it’s not actually current RSS, but whatever was Max. at some point in the past. I.e. it must only go up. RSS from `meminfo_procfs` seems to only go up until some pressure is applied. I didn’t see RSS go down unless I put memory pressure on some other process, and then only by down by 1-180.680/292.375 = 38%. I was trying to allocate as much memory as could _in a different_ Julia process (I think I went pretty close to the max I could). Going to high I got:

```julia
julia> A = fill(2, 5000000000)
ERROR: OutOfMemoryError()

slightly lower:
julia> A = fill(2, 4000000000)
Killed

```

I’m not sure what the kernel does, it could always take memory from you, at least swap out from the unused part at the end of your memory allocation. I’m not sure if can look at holes in you heap. I guess it must infer those, the fragmentation, from your memory usage.

Just after starting Julia:

```julia
julia> Sys.maxrss()/1000/1000 # likely because starting up, processing (the default) sysimage uses a lot of mem.:
181.940224

julia> Base.gc_live_bytes()/2^20
9.989258766174316

julia> GC.gc()

julia> Sys.maxrss()/1000/1000 # interesting "mem use" actually went up, but I guess actual max (which is not possible to query?) only temporarily:
216.727552

julia> Base.gc_live_bytes()/2^20
2.956634521484375

```

> RSS is the Resident Set Size and is used to show how much memory is allocated to that process and is in RAM. It does not include memory that is swapped out. […] It does include all stack and heap memory.
> 
> VSZ is the Virtual Memory Size. It includes all memory that the process can access, including memory that is swapped out, memory that is allocated, but not used, and memory that is from shared libraries.

Sys.maxrss() uses **getrusage** (on Linux) according to its help, it actually does `ccall(:jl_maxrss, Csize_t, ())` and I see:

> **[Resource Usage (The GNU C Library)](https://www.gnu.org/software/libc/manual/html_node/Resource-Usage.html)**
>
> Resource Usage (The GNU C Library)

> In most systems, processes has only two valid values:
> 
> RUSAGE\_SELF
> 
> ```
> Just the current process.
> 
> ```
> 
> RUSAGE\_CHILDREN
> 
> ```
> All child processes (direct and indirect) that have already terminated. 
> 
> ```

I suppose RUSAGE\_SELF is used, but RUSAGE\_CHILDREN might also be good info (on a cluster). Also this what I though impossible to do (all threads in a process share memory space, so while they could track their own allocations, I thought the kernel couldn’t know about that, maybe I misunderstand what this is about):

> ```
> RUSAGE_THREAD (since Linux 2.6.26)
> Return resource usage statistics for the calling thread.
> 
> ```

What is this, it has no help string:

```julia
gc_total_bytes(gc_num::GC_Num) =
    gc_num.allocd + gc_num.deferred_alloc + gc_num.total_allocd

```

Is that total ever allocated?

```julia
struct GC_Num
[..]
   total_allocd ::Int64 # GC internal
[..]
end

gc_num() = ccall(:jl_gc_num, GC_Num, ())

# This type is to represent differences in the counters, so fields may be negative
struct GC_Diff

```

---

_[View the full topic](https://discourse.julialang.org/t/how-to-track-total-memory-usage-of-julia-process-over-time/91167)._