# Running out of memory saving files with HDF5

**URL:** <https://discourse.julialang.org/t/running-out-of-memory-saving-files-with-hdf5/27802>\
**Category:** Performance\
**Tags:** hdf5, memory, memory-allocation\
**Created:** [August 21, 2019, 2:57pm UTC](https://discourse.julialang.org/t/running-out-of-memory-saving-files-with-hdf5/27802 "2019-08-21T14:57:55Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![PhillipBC](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/phillipbc/32/9166_2.png) [@PhillipBC](https://discourse.julialang.org/u/PhillipBC)\
**Post date:** [August 21, 2019, 2:57pm UTC](https://discourse.julialang.org/t/running-out-of-memory-saving-files-with-hdf5/27802/1 "2019-08-21T14:57:55Z")

</div>

Hey, I am trying to run something quite large in Julia, and I need to store a large number of Arrays (for constructing sparse matrices). Below is a simplified version of what I need to do, where it creates a random array, and saves it to a file. The dimensions are approximately what I need to use.

(Running Julia 1.1.1 on Ubuntu 18.04)

```julia
for j in 1:1200
    file = h5open("example$(j).h5","w")
    write(file, "X", rand(1,700000))
    close(file)
    GC.gc()
end

```

I don’t see why it is eating up memory when running. My machine has 16GB of RAM, will run out of memory before the above will complete. I have tried lots of ways to get around it, I was originally using JLD, which didn’t have this problem when writing the files, but encountered the same issue when reading the files.

Thanks in advance,  
Phil.

---

<div class="post-metadata">

**Author:** ![hendri54](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/hendri54/32/9621_2.png) [@hendri54](https://discourse.julialang.org/u/hendri54)\
**Post date:** [August 21, 2019, 3:36pm UTC](https://discourse.julialang.org/t/running-out-of-memory-saving-files-with-hdf5/27802/2 "2019-08-21T15:36:08Z")

</div>

Rather than allocating a new array in each iteration, can you pre-allocate?

---

<div class="post-metadata">

**Author:** ![PhillipBC](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/phillipbc/32/9166_2.png) [@PhillipBC](https://discourse.julialang.org/u/PhillipBC)\
**Post date:** [August 21, 2019, 3:49pm UTC](https://discourse.julialang.org/t/running-out-of-memory-saving-files-with-hdf5/27802/3 "2019-08-21T15:49:39Z")

</div>

I can change it to:

```julia
for j in 1:1200
    x = rand(1,700000)
    file = h5open("example$(j).h5","w")
    write(file, "X", x)
    close(file)
    GC.gc()
end

```

But I need to calculate 1200 different arrays.

In the actual problem there are 1200 sparse matrices that need to be used in another loop, and storing all of them takes up too much memory. So I want to create and save them all, so I can load one as its needed, so I only have 1 matrix loaded at a time, rather than having 1200 loaded.

---

<div class="post-metadata">

**Author:** ![jamblejoe](https://avatars.discourse-cdn.com/v4/letter/j/ee7513/32.png) [@jamblejoe](https://discourse.julialang.org/u/jamblejoe)\
**Post date:** [August 21, 2019, 3:58pm UTC](https://discourse.julialang.org/t/running-out-of-memory-saving-files-with-hdf5/27802/4 "2019-08-21T15:58:19Z")

</div>

for the following code I do not see constant memory allocation:

```julia
A=Vector{Float64}(undef, 700000)
for j in 1:1200
    file = h5open("example$(j).h5","w")
    rand!(A)
    write(file, "A", A)
    close(file)
end

```

Can you confirm this @PhillipBC? But nevertheless, I do not understand why in the initial post there is a constant increase in memory usage. Does anyone know?

---

<div class="post-metadata">

**Author:** ![programista](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/programista/32/5372_2.png) [@programista](https://discourse.julialang.org/u/programista)\
**Post date:** [August 21, 2019, 4:01pm UTC](https://discourse.julialang.org/t/running-out-of-memory-saving-files-with-hdf5/27802/5 "2019-08-21T16:01:06Z")

</div>

julia\_1.0.4\_\_LT , Win7, 8GB, both codes run without any probelms.  
Paul

---

<div class="post-metadata">

**Author:** ![jamblejoe](https://avatars.discourse-cdn.com/v4/letter/j/ee7513/32.png) [@jamblejoe](https://discourse.julialang.org/u/jamblejoe)\
**Post date:** [August 21, 2019, 4:12pm UTC](https://discourse.julialang.org/t/running-out-of-memory-saving-files-with-hdf5/27802/6 "2019-08-21T16:12:50Z")

</div>

I can confirm that

```julia
for j in 1:1200
    file = h5open("example$(j).h5","w")
    write(file, "X", rand(1,700000))
    close(file)
    GC.gc()
end

```

and

```julia
for j in 1:1200
    x = rand(1,700000)
    file = h5open("example$(j).h5","w")
    write(file, "X", x)
    close(file)
    GC.gc()
end

```

both have a memory increase of roughly 50 MB per second on my machine - UBUNTU 16.04 LT

```julia
julia> versioninfo()
Julia Version 1.1.1
Commit 55e36cc308 (2019-05-16 04:10 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Core(TM) i3-8100 CPU @ 3.60GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.1 (ORCJIT, skylake)

```

---

<div class="post-metadata">

**Author:** ![jpsamaroo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jpsamaroo/32/46804_2.png) [@jpsamaroo](https://discourse.julialang.org/u/jpsamaroo)\
**Post date:** [August 21, 2019, 5:33pm UTC](https://discourse.julialang.org/t/running-out-of-memory-saving-files-with-hdf5/27802/7 "2019-08-21T17:33:51Z")

</div>

This is probably caused by [https://github.com/JuliaLang/julia/issues/30888](https://github.com/JuliaLang/julia/issues/30888) or its variant, since you say you’re on Ubuntu. Can you try this on a nightly build of Julia and see if it’s still a problem?

---

<div class="post-metadata">

**Author:** ![PhillipBC](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/phillipbc/32/9166_2.png) [@PhillipBC](https://discourse.julialang.org/u/PhillipBC)\
**Post date:** [August 22, 2019, 8:56am UTC](https://discourse.julialang.org/t/running-out-of-memory-saving-files-with-hdf5/27802/8 "2019-08-22T08:56:40Z")

</div>

@jpsamaroo I have now tried this on julia-1.2.0 and it is not happening anymore, thanks!