# Methods to reduce gc time?

**URL:** <https://discourse.julialang.org/t/methods-to-reduce-gc-time/8840>\
**Category:** Performance\
**Created:** [February 5, 2018, 5:25pm UTC](https://discourse.julialang.org/t/methods-to-reduce-gc-time/8840 "2018-02-05T17:25:01Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![RandomString123](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/randomstring123/32/3194_2.png) [@RandomString123](https://discourse.julialang.org/u/RandomString123)\
**Post date:** [February 5, 2018, 5:25pm UTC](https://discourse.julialang.org/t/methods-to-reduce-gc-time/8840/1 "2018-02-05T17:25:01Z")

</div>

I have finally gotten some code I translated from R working. It is by no means optimized, and it is my first julia code, so I didn’t expect great performance but it is significantly slower than I would have expected. In my first runs I have noticed that a lot of my time is gc time. Are there methods to reduce this? I am running in an environment with plenty of memory available so it getting garbage collected in the middle of an operation is really not necessary.

> 92.123407 seconds (382.46 M allocations: 12.969 GiB, 64.78% gc time)  
> 1999638×9 DataFrames.DataFrame.

---

<div class="post-metadata">

**Author:** ![nalimilan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nalimilan/32/147_2.png) [@nalimilan](https://discourse.julialang.org/u/nalimilan)\
**Post date:** [February 5, 2018, 5:53pm UTC](https://discourse.julialang.org/t/methods-to-reduce-gc-time/8840/2 "2018-02-05T17:53:30Z")

</div>

It’s hard to tell without knowing what the code does, but in general you can reduce GC time by reducing the number of allocations, which generally implies using type-stable code. In the context of data frames, this notably implies applying functions to a whole column rather than iterating over them, or passing them as vectors to a function which will loop over them.

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [February 5, 2018, 5:56pm UTC](https://discourse.julialang.org/t/methods-to-reduce-gc-time/8840/3 "2018-02-05T17:56:38Z")

</div>

Extra allocations can be indicative of type-stability issues. Did you check `@code_warntype`?

---

<div class="post-metadata">

**Author:** ![RandomString123](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/randomstring123/32/3194_2.png) [@RandomString123](https://discourse.julialang.org/u/RandomString123)\
**Post date:** [February 5, 2018, 6:13pm UTC](https://discourse.julialang.org/t/methods-to-reduce-gc-time/8840/4 "2018-02-05T18:13:55Z")

</div>

I’d have to really dig down with @code\_warntype it appears to only do a function at a time. It is a great pointer though, I did not realize that existed.

As for the code it is reading in some large files, I suspect because I read them in a field at time and stream to a data frame using the data streaming API. I might try to run the profier on it and see if that gives any insight. It could also be that I am using Missings and .6, I might upgrade my env to .7dev and see if that helps.

---

<div class="post-metadata">

**Author:** ![RandomString123](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/randomstring123/32/3194_2.png) [@RandomString123](https://discourse.julialang.org/u/RandomString123)\
**Post date:** [February 6, 2018, 9:52pm UTC](https://discourse.julialang.org/t/methods-to-reduce-gc-time/8840/5 "2018-02-06T21:52:10Z")

</div>

An update…

I wrote code that calls the DataStreams methods directly on my objects and just drops the results and then compared that to streaming the data into a data frame (benchmarks below). It seems as though 2/3s of my time is streaming to the data frame. As an FYI this data has ~ 55000 rows, 24 cols each up to 350 characters in length

As a next step I am going to modify my file reading code to stream whole columns instead of a field at a time and see what effect that has on performance.

```julia
Main> @benchmark stream_to_thevoid()
BenchmarkTools.Trial:
  memory estimate: 1.40 GiB
  allocs estimate: 14714627
  --------------
  minimum time: 2.531 s (9.83% GC)
  median time: 2.544 s (9.89% GC)
  mean time: 2.544 s (9.89% GC)
  maximum time: 2.556 s (9.96% GC)
  --------------
  samples: 2
  evals/sample: 1

Main> @benchmark stream_to_df()
BenchmarkTools.Trial:
  memory estimate: 280.94 MiB
  allocs estimate: 5706479
  --------------
  minimum time: 627.397 ms (22.18% GC)
  median time: 686.483 ms (27.38% GC)
  mean time: 715.479 ms (30.52% GC)
  maximum time: 832.426 ms (40.66% GC)
  --------------
  samples: 7
  evals/sample: 1

```

---

<div class="post-metadata">

**Author:** ![RandomString123](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/randomstring123/32/3194_2.png) [@RandomString123](https://discourse.julialang.org/u/RandomString123)\
**Post date:** [February 7, 2018, 3:42am UTC](https://discourse.julialang.org/t/methods-to-reduce-gc-time/8840/6 "2018-02-07T03:42:01Z")

</div>

And another update for using column based streaming…

The column based streaming method I developed significantly lowers GC time and data frame creation time. This trial based on the same small sample file above lowers the overall and GC time to just under what was for just reading in data and dumping it above.

```julia
Main> @benchmark stream_to_df()
BenchmarkTools.Trial:
  memory estimate: 390.63 MiB
  allocs estimate: 9537896
  --------------
  minimum time: 2.320 s (3.38% GC)
  median time: 2.432 s (7.78% GC)
  mean time: 2.398 s (6.61% GC)
  maximum time: 2.440 s (8.53% GC)
  --------------
  samples: 3
  evals/sample: 1

```

Also using a column based streaming method has improved loading a much larger amount of data down to around 60% of its old value.

```julia
 62.887358 seconds (325.86 M allocations: 11.366 GiB, 69.74% gc time)
2035011×9 DataFrames.DataFrame. Omitted printing of 4 columns

```

One significant slowdown that I hope I can figure out how to get over is having to use strip after turning a byte array into a string.

```julia
strip(String(buf))

```

---

<div class="post-metadata">

**Author:** ![nalimilan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nalimilan/32/147_2.png) [@nalimilan](https://discourse.julialang.org/u/nalimilan)\
**Post date:** [February 7, 2018, 10:20am UTC](https://discourse.julialang.org/t/methods-to-reduce-gc-time/8840/7 "2018-02-07T10:20:25Z")

</div>

I’m not sure what you mean exactly by “column-based streaming method”, but if you care about performance then don’t use `Missing` on Julia 0.6. Use `DataArrays` instead, and `readtable` instead of `CSV.read`.

---

<div class="post-metadata">

**Author:** ![RandomString123](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/randomstring123/32/3194_2.png) [@RandomString123](https://discourse.julialang.org/u/RandomString123)\
**Post date:** [February 7, 2018, 2:15pm UTC](https://discourse.julialang.org/t/methods-to-reduce-gc-time/8840/8 "2018-02-07T14:15:15Z")

</div>

Just as some more background…I have been using a project I wrote in R as a learn Julia opportunity. As part of it I need to read in and convert to a usable format ~30GB of fixed width datafiles. To do this I wrote a fixed width file parser that uses DataStreams to convert the files into DataFrames. The parser is here: [GitHub - RandomString123/FWF.jl: Fixed width file parsing in Julia](https://github.com/RandomString123/FWF.jl)

The “column-based streaming method” is enabling ::Type{Data.Column} in the DataStreams API.  
Using Data.Field is nice for as needed parsing but seems to really slow down with a large number of fields.

And for what it is worth…I have gotten the time to read the file and stream it into a dataframe to a reasonableish level. I will take your suggestion and try using DataArrays for .6 testing and leave missings for .7, that might make up the rest of the performance gap. As for processing I think there is a lot of efficiency to be gained as I process in an “R” way which is not optimal in Julia.