# Parallelizing file processing with the least amount of overhead

**URL:** <https://discourse.julialang.org/t/parallelizing-file-processing-with-the-least-amount-of-overhead/100169>\
**Category:** Performance\
**Tags:** multithreading, file\
**Created:** [June 11, 2023, 6:48am UTC](https://discourse.julialang.org/t/parallelizing-file-processing-with-the-least-amount-of-overhead/100169 "2023-06-11T06:48:54Z")\
**Posts on this page:** 1\
**Showing post:** 3

<div class="post-metadata">

**Author:** ![CodeGodz](https://avatars.discourse-cdn.com/v4/letter/c/aeb1de/32.png) [@CodeGodz](https://discourse.julialang.org/u/CodeGodz)\
**Post date:** [June 14, 2023, 9:58am UTC](https://discourse.julialang.org/t/parallelizing-file-processing-with-the-least-amount-of-overhead/100169/3 "2023-06-14T09:58:28Z")

</div>

Aah just realized tkf has not been around for a while, @yuyichao (sorry for the tag just thought of you), do you have an idea on this with your multi-threading experience?

Let’s consider that processing a line takes much longer than reading it, and that we cannot assume users will have NVMEs or SSD, so we can only use a single thread for reading.

I’m okay with the `Channel` but as I said above overloading the number of threads will suddenly degrade performance and I cannot really expect everyone to try to figure out the best number of threads to match their IO. [ConcurrentCollections](https://discourse.julialang.org/t/concurrentcollections-jl-lock-free-dictionary-queue-etc-for-julia-1-7/72501) does not have that problem but it will flood memory as soon as there isn’t enough “CPU power” to keep up with reading

---

_[View the full topic](https://discourse.julialang.org/t/parallelizing-file-processing-with-the-least-amount-of-overhead/100169)._