# More pressure on garbage collector

**URL:** <https://discourse.julialang.org/t/more-pressure-on-garbage-collector/6493>\
**Category:** General Usage\
**Created:** [October 17, 2017, 7:41am UTC](https://discourse.julialang.org/t/more-pressure-on-garbage-collector/6493 "2017-10-17T07:41:15Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![Tomas\_Pevny](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tomas_pevny/32/25466_2.png) [@Tomas\_Pevny](https://discourse.julialang.org/u/Tomas_Pevny)\
**Post date:** [October 17, 2017, 7:41am UTC](https://discourse.julialang.org/t/more-pressure-on-garbage-collector/6493/1 "2017-10-17T07:41:15Z")

</div>

Hello,

is it possible to manually increase pressure on Garbage Collector? I have several workers whose job is to download large files from S3, load them, and preprocess such that the master worker does not need to deal with this. But Julia crashes because all the memory is consumed, but this takes quite some time. I suspect that this is because garbage collector is not executed in time due to distributed environment. Can I somehow verify this idea?

Thanks for answers,  
Tomas

---

<div class="post-metadata">

**Author:** ![tamasgal](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamasgal/32/27946_2.png) [@tamasgal](https://discourse.julialang.org/u/tamasgal)\
**Post date:** [October 17, 2017, 8:01am UTC](https://discourse.julialang.org/t/more-pressure-on-garbage-collector/6493/2 "2017-10-17T08:01:59Z")

</div>

You can use `gc()` to force the garbage collection, however, I don’t think that this is the root of your problem. Have you benchmarked your scripts to check the memory usage and peaks?

```julia
help?> gc
search: gc gcd gcdx gc_enable eigvecs eigfact eigfact! logspace getsockname

  gc()

  Perform garbage collection. This should not generally be used.

```

---

<div class="post-metadata">

**Author:** ![Tomas\_Pevny](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tomas_pevny/32/25466_2.png) [@Tomas\_Pevny](https://discourse.julialang.org/u/Tomas_Pevny)\
**Post date:** [October 17, 2017, 8:07am UTC](https://discourse.julialang.org/t/more-pressure-on-garbage-collector/6493/3 "2017-10-17T08:07:31Z")

</div>

I do not really know, how to construct such a benchmark.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [October 17, 2017, 8:11am UTC](https://discourse.julialang.org/t/more-pressure-on-garbage-collector/6493/4 "2017-10-17T08:11:54Z")

</div>

`@time` would be a good start. Since the job is time-consuming, I would not recommend something more systematic like

> **[GitHub - JuliaCI/BenchmarkTools.jl: A benchmarking framework for the Julia...](https://github.com/JuliaCI/BenchmarkTools.jl)**
>
> A benchmarking framework for the Julia language. Contribute to JuliaCI/BenchmarkTools.jl development by creating an account on GitHub.

---

<div class="post-metadata">

**Author:** ![Tomas\_Pevny](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tomas_pevny/32/25466_2.png) [@Tomas\_Pevny](https://discourse.julialang.org/u/Tomas_Pevny)\
**Post date:** [October 17, 2017, 8:16am UTC](https://discourse.julialang.org/t/more-pressure-on-garbage-collector/6493/5 "2017-10-17T08:16:45Z")

</div>

Thanks, I will resort to memory hunting.  
Is it possible to see the size of the object?

---

<div class="post-metadata">

**Author:** ![anon61610682](https://avatars.discourse-cdn.com/v4/letter/a/ad7895/32.png) [@anon61610682](https://discourse.julialang.org/u/anon61610682)\
**Post date:** [October 17, 2017, 9:01am UTC](https://discourse.julialang.org/t/more-pressure-on-garbage-collector/6493/6 "2017-10-17T09:01:18Z")

</div>

`sizeof` should be doing the job for you. However, if you are using a custom datatype, you might need to overload `Base.sizeof` for your types, since normal `sizeof` will report wrong values in the sense that they will give you the pointer sizes for your architecture if you are using fields such as `A::Matrix{Float64}`, etc. See below for the example:

```julia
julia> A = randn(5,5);

julia> sizeof(A) # 5x5x8 bytes for Float64
200

julia> struct MyType
         A::Matrix{Float64}
       end

julia> obj = MyType(A);

julia> sizeof(obj) # 8 bytes on 64-bit architectures (i.e., `A` is simply a pointer)
8

julia> import Base.sizeof

julia> function sizeof(obj::MyType)
         res = 0
         for field in fieldnames(obj)
           res += sizeof(getfield(obj, field))
         end
         return res
       end
sizeof (generic function with 8 methods)

julia> sizeof(obj) # now we have correct value
200

```

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [October 17, 2017, 9:06am UTC](https://discourse.julialang.org/t/more-pressure-on-garbage-collector/6493/7 "2017-10-17T09:06:00Z")

</div>

See `Base.summarysize`.

---

<div class="post-metadata">

**Author:** ![Tomas\_Pevny](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tomas_pevny/32/25466_2.png) [@Tomas\_Pevny](https://discourse.julialang.org/u/Tomas_Pevny)\
**Post date:** [October 17, 2017, 9:34am UTC](https://discourse.julialang.org/t/more-pressure-on-garbage-collector/6493/8 "2017-10-17T09:34:22Z")

</div>

Thanks a lot for help. This seems to be nicely clear to me.

---

<div class="post-metadata">

**Author:** ![LeoK987](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/leok987/32/4880_2.png) [@LeoK987](https://discourse.julialang.org/u/LeoK987)\
**Post date:** [August 29, 2018, 2:30am UTC](https://discourse.julialang.org/t/more-pressure-on-garbage-collector/6493/9 "2018-08-29T02:30:15Z")

</div>

@Tomas_Pevny, would you kindly update your findings about the memory leak? I find I have similar issue related to reading files many times.

---

<div class="post-metadata">

**Author:** ![Tomas\_Pevny](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tomas_pevny/32/25466_2.png) [@Tomas\_Pevny](https://discourse.julialang.org/u/Tomas_Pevny)\
**Post date:** [August 29, 2018, 7:34am UTC](https://discourse.julialang.org/t/more-pressure-on-garbage-collector/6493/10 "2018-08-29T07:34:40Z")

</div>

On the end, I have found two bugs in my code:

1. I was repeatedly using eval to introduce functions, which slightly bloated Julia’s dictionary of functions
2. I was not closing stream from Transcoding streams.

Finding these two bugs was quite tedious, as I was cutting my code in halves to identify source of the problem.

---

<div class="post-metadata">

**Author:** ![LeoK987](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/leok987/32/4880_2.png) [@LeoK987](https://discourse.julialang.org/u/LeoK987)\
**Post date:** [August 29, 2018, 8:42am UTC](https://discourse.julialang.org/t/more-pressure-on-garbage-collector/6493/11 "2018-08-29T08:42:27Z")

</div>

Thanks very much for sharing.