# Thread- and process-based parallelisms in Transducers.jl (+ some news)

**URL:** <https://discourse.julialang.org/t/thread-and-process-based-parallelisms-in-transducers-jl-some-news/32285>\
**Category:** Package Announcements\
**Tags:** parallel, multithreading, distributed\
**Created:** [December 15, 2019, 2:39am UTC](https://discourse.julialang.org/t/thread-and-process-based-parallelisms-in-transducers-jl-some-news/32285 "2019-12-15T02:39:01Z")\
**Posts on this page:** 1\
**Showing post:** 1

<div class="post-metadata">

**Author:** ![tkf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkf/32/17635_2.png) [@tkf](https://discourse.julialang.org/u/tkf)\
**Post date:** [December 15, 2019, 2:39am UTC](https://discourse.julialang.org/t/thread-and-process-based-parallelisms-in-transducers-jl-some-news/32285/1 "2019-12-15T02:39:02Z")

</div>

It’s been a while since I added parallelism supports in Transducers.jl but I’ve never announced this feature properly. I just added a few utility functions and a tutorial so I think it’s good timing to do this.

Quoting [Overview of parallel processing in Transducers.jl](https://tkf.github.io/Transducers.jl/dev/parallelism/):

> Transducers.jl supports thread-based ([`reduce`](https://tkf.github.io/Transducers.jl/dev/manual/#Base.reduce)) and process-based ([`dreduce`](https://tkf.github.io/Transducers.jl/dev/manual/#Transducers.dreduce)) parallelisms with the same composable API; i.e. transducers. Having a uniform API to cover different parallelisms as well as sequential processing [`foldl`](https://tkf.github.io/Transducers.jl/dev/manual/#Base.foldl) is useful. Using multiple cores or machines for your computation is as easy as replacing `foldl` with `reduce` or `dreduce`; _you don’t need to re-write your transducers or reducing functions_.
> 
> See also:
> 
> - [Parallel processing tutorial](https://tkf.github.io/Transducers.jl/dev/examples/tutorial_parallel/) in Transducers.jl manual.
> - API documentation of [`reduce`](https://tkf.github.io/Transducers.jl/dev/manual/#Base.reduce) and [`dreduce`](https://tkf.github.io/Transducers.jl/dev/manual/#Transducers.dreduce).
> 
> ## Thread-based parallelism
> 
> Transducers.jl supports thread-based parallelism for Julia ≥ 1.0. You can use it by replacing `foldl` with `reduce`. With Julia ≥ 1.3, Transducers.jl supports _early termination_ to avoid unnecessary computation while guaranteeing the result to be _deterministic_; i.e., it does not depend on how computation tasks are scheduled.
> 
> ## Process-based parallelism
> 
> Transducers.jl supports process-based parallelism using [Distributed.jl](https://docs.julialang.org/en/latest/stdlib/Distributed/). You can use it by replacing `foldl` with `dreduce`. It can be used for horizontally scaling the computation. It is also useful for using external libraries that are not “thread-safe.”
> 
> Note that [early termination is not supported in `dreduce` yet](https://github.com/tkf/Transducers.jl/issues/88).

### Misc news

- I managed to upstream a few transducers to Julia `Base`! It’ll be available in Julia 1.4. For example, it makes `sum(y for x in 1:1000 for y in 1:x if y % 2 == 0)` ~3x faster. See: [Transducer as an optimization: map, filter and flatten by tkf · Pull Request #33526 · JuliaLang/julia](https://github.com/JuliaLang/julia/pull/33526)
- Recent versions of Transducers.jl include [`withprogress`](https://tkf.github.io/Transducers.jl/dev/manual/#Transducers.withprogress) that can be used to monitor the progress of your computation. This is done by emitting [ProgressLogging.jl](https://github.com/JunoLab/ProgressLogging.jl)-compatible progress events. It will show progress bars if you use Juno, [ConsoleProgressMonitor.jl](https://github.com/tkf/ConsoleProgressMonitor.jl), or [TerminalLoggers.jl](https://github.com/c42f/TerminalLoggers.jl). It can be used with thread- and process-based parallel reduce.
- Transducers.jl now can be used to create various table types from DataFrames.jl, TypedTables.jl, StructArrays.jl, etc. See [`copy`](https://tkf.github.io/Transducers.jl/dev/manual/#Base.copy) and its parallel versions [`tcopy`](https://tkf.github.io/Transducers.jl/dev/manual/#Transducers.tcopy) and [`dcopy`](https://tkf.github.io/Transducers.jl/dev/manual/#Transducers.dcopy).

---

_[View the full topic](https://discourse.julialang.org/t/thread-and-process-based-parallelisms-in-transducers-jl-some-news/32285)._