# Massive memory leak in Julia due to tiny memory leak in C

**URL:** <https://discourse.julialang.org/t/massive-memory-leak-in-julia-due-to-tiny-memory-leak-in-c/119526>\
**Category:** General Usage\
**Created:** [September 17, 2024, 8:40pm UTC](https://discourse.julialang.org/t/massive-memory-leak-in-julia-due-to-tiny-memory-leak-in-c/119526 "2024-09-17T20:40:35Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Yixiao-Zhang](https://avatars.discourse-cdn.com/v4/letter/y/b3f665/32.png) [@Yixiao-Zhang](https://discourse.julialang.org/u/Yixiao-Zhang)\
**Post date:** [September 17, 2024, 8:40pm UTC](https://discourse.julialang.org/t/massive-memory-leak-in-julia-due-to-tiny-memory-leak-in-c/119526/1 "2024-09-17T20:40:35Z")

</div>

The latest version (v4.9.2) of `netCDF-C` (a C library for outputting data in the network Common Data Form.) is known to have memory leak. The leaked memory is not more than several MBs (see [this link](https://github.com/Unidata/netcdf-c/issues/2626)). However, when I use `NCDatasets.jl`, which is a wrapper in Julia for `netcdf-C`, I find that the memory leak reaches several GBs in the demo below:

```Julia
using Printf
using NCDatasets

function create_dummpy_netcdf()
    dset = NCDataset("output.nc", "c", format=:netcdf4)
    close(dset)
end

function output_dummy_netcdf()
    dset = NCDataset("output.nc", "a")
    zeros(Float64, 1024, 1024)
    close(dset)
end

function main()

    create_dummpy_netcdf()
    for i in 1:1000
        output_dummy_netcdf()

        @info Printf.@sprintf "Max. RSS: %9.3f MiB\n" Sys.maxrss()/2^20
    end

end

main()

```

The total memory usage reaches 8.2 GiB in the end, which is roughly 1000 times the allocation of `zeros(Float64, 1024, 1024)`. Removing `zeros(Float64, 1024, 1024)` in `output_dummy_netcdf` makes the total memory usage limited to 400 MiB. It seems to me that the memory leak in the C code is “amplified” by allocations in Julia.

It is related to [this issue on GitHub](https://github.com/Alexander-Barth/NCDatasets.jl/issues/266). Alexander-Barth finds that this bug can be produced simply with `ccall`-ing functions in `libnetcdf.so`. I am bringing the discussion here because I think it is related to how Julia manages memory.

I have also tried using `Valgrind` to profile the heap memory usage. However, when running with `Valgrind`, this bug cannot be reproduced. If you know a better tool for profiling memory in Julia, please let me know.

This question has been posted on JuliaLang as #55794, as suggested by @d-netto.

---

<div class="post-metadata">

**Author:** ![d-netto](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/d-netto/32/212022_2.png) [@d-netto](https://discourse.julialang.org/u/d-netto)\
**Post date:** [September 17, 2024, 9:11pm UTC](https://discourse.julialang.org/t/massive-memory-leak-in-julia-due-to-tiny-memory-leak-in-c/119526/2 "2024-09-17T21:11:07Z")

</div>

Could you open an issue on JuliaLang ([Issues · JuliaLang/julia · GitHub](https://github.com/JuliaLang/julia/issues))?

---

<div class="post-metadata">

**Author:** ![juliohm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juliohm/32/215266_2.png) [@juliohm](https://discourse.julialang.org/u/juliohm)\
**Post date:** [September 17, 2024, 9:20pm UTC](https://discourse.julialang.org/t/massive-memory-leak-in-julia-due-to-tiny-memory-leak-in-c/119526/3 "2024-09-17T21:20:56Z")

</div>

cc: @Alexander-Barth

---

<div class="post-metadata">

**Author:** ![vchuravy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/vchuravy/32/8_2.png) [@vchuravy](https://discourse.julialang.org/u/vchuravy)\
**Post date:** [September 19, 2024, 1:30pm UTC](https://discourse.julialang.org/t/massive-memory-leak-in-julia-due-to-tiny-memory-leak-in-c/119526/4 "2024-09-19T13:30:14Z")

</div>

> [@Yixiao-Zhang](#):
>
> The total memory usage reaches 8.2 GiB in the end,

> “Maximum RSS” means **the maximum of the RSS since the process’s birth** , i.e. the largest it has ever been. So this number tells you the largest amount of physical memory your process has ever been using at any one instant.

This doesn’t mean that Julia is leaking memory. The GC might simply have not run yet, since there is no memory pressure.

When I run this code locally:

With an `@time` in front of main.

```julia
[ Info: Max. RSS: 478.117 MiB
[ Info: Max. RSS: 478.117 MiB
  0.932792 seconds (753.56 k allocations: 7.865 GiB, 18.04% gc time, 22.93% compilation time)

```