# Faster conversion to CategoricalArray with many groups

**URL:** <https://discourse.julialang.org/t/faster-conversion-to-categoricalarray-with-many-groups/55057>\
**Category:** Performance\
**Created:** [February 11, 2021, 12:51pm UTC](https://discourse.julialang.org/t/faster-conversion-to-categoricalarray-with-many-groups/55057 "2021-02-11T12:51:35Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![danielw2904](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/danielw2904/32/10890_2.png) [@danielw2904](https://discourse.julialang.org/u/danielw2904)\
**Post date:** [February 11, 2021, 12:51pm UTC](https://discourse.julialang.org/t/faster-conversion-to-categoricalarray-with-many-groups/55057/1 "2021-02-11T12:51:35Z")

</div>

I need to convert columns in a larger (~20M rows) DataFrame to `CategoricalArray`s to be used with MixedModels and FixedEffectModels (I think the latter does not require `Categorical` but the former does). If there are many groups in a column conversion takes extremely long (it is the longest running line in my script). Is there a way to make conversion faster/parallel? Maybe Transducers with BangBang?

---

<div class="post-metadata">

**Author:** ![nilshg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nilshg/32/2283_2.png) [@nilshg](https://discourse.julialang.org/u/nilshg)\
**Post date:** [February 11, 2021, 1:46pm UTC](https://discourse.julialang.org/t/faster-conversion-to-categoricalarray-with-many-groups/55057/2 "2021-02-11T13:46:50Z")

</div>

I don’t have an answer unfortunately but another question - are MixedModels and FixedEffectModels fine with 20m row data? How much RAM do you have to make things work?

---

<div class="post-metadata">

**Author:** ![danielw2904](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/danielw2904/32/10890_2.png) [@danielw2904](https://discourse.julialang.org/u/danielw2904)\
**Post date:** [February 11, 2021, 3:59pm UTC](https://discourse.julialang.org/t/faster-conversion-to-categoricalarray-with-many-groups/55057/3 "2021-02-11T15:59:32Z")

</div>

Yes definitely fine. Results take a couple of minutes but in the overall project that is a short amount of time. I’m using a Xeon W-2145 CPU (3.70GHz, usually boosts to around 4.2GHz) with 8 cores and 64GB of RAM. Should also work on 32GB RAM with carefull data management.

---

<div class="post-metadata">

**Author:** ![danielw2904](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/danielw2904/32/10890_2.png) [@danielw2904](https://discourse.julialang.org/u/danielw2904)\
**Post date:** [February 11, 2021, 4:15pm UTC](https://discourse.julialang.org/t/faster-conversion-to-categoricalarray-with-many-groups/55057/4 "2021-02-11T16:15:00Z")

</div>

So as I wrote this the process was killed because I was out of memory but due to me including interactions and interactions of powers so I guess it works for reasonably sized models. What works is 2 FE, 1 FE interaction and 6 main effects, 4 interactions where 2 are continuous and 2 are with one categorical variable (one binary one with 10 groups). If I then add powers and more interactions I run out of memory.