# Thread- and process-based parallelisms in Transducers.jl (+ some news)
**URL:** https://discourse.julialang.org/t/thread-and-process-based-parallelisms-in-transducers-jl-some-news/32285
**Category:** Package Announcements
**Tags:** parallel, multithreading, distributed
**Created:** [December 15, 2019, 2:39am UTC](https://discourse.julialang.org/t/thread-and-process-based-parallelisms-in-transducers-jl-some-news/32285 "2019-12-15T02:39:01Z")
**Posts on this page:** 6
**Page:** 1
### Author:  [@tkf](https://discourse.julialang.org/u/tkf)
#### Post date: [December 15, 2019, 2:39am UTC](https://discourse.julialang.org/t/thread-and-process-based-parallelisms-in-transducers-jl-some-news/32285/1 "2019-12-15T02:39:02Z")
It’s been a while since I added parallelism supports in Transducers.jl but I’ve never announced this feature properly. I just added a few utility functions and a tutorial so I think it’s good timing to do this.
Quoting [Overview of parallel processing in Transducers.jl](https://tkf.github.io/Transducers.jl/dev/parallelism/):
> Transducers.jl supports thread-based ([`reduce`](https://tkf.github.io/Transducers.jl/dev/manual/#Base.reduce)) and process-based ([`dreduce`](https://tkf.github.io/Transducers.jl/dev/manual/#Transducers.dreduce)) parallelisms with the same composable API; i.e. transducers. Having a uniform API to cover different parallelisms as well as sequential processing [`foldl`](https://tkf.github.io/Transducers.jl/dev/manual/#Base.foldl) is useful. Using multiple cores or machines for your computation is as easy as replacing `foldl` with `reduce` or `dreduce`; _you don’t need to re-write your transducers or reducing functions_.
>
> See also:
>
> - [Parallel processing tutorial](https://tkf.github.io/Transducers.jl/dev/examples/tutorial_parallel/) in Transducers.jl manual.
> - API documentation of [`reduce`](https://tkf.github.io/Transducers.jl/dev/manual/#Base.reduce) and [`dreduce`](https://tkf.github.io/Transducers.jl/dev/manual/#Transducers.dreduce).
>
> ## Thread-based parallelism
>
> Transducers.jl supports thread-based parallelism for Julia ≥ 1.0. You can use it by replacing `foldl` with `reduce`. With Julia ≥ 1.3, Transducers.jl supports _early termination_ to avoid unnecessary computation while guaranteeing the result to be _deterministic_; i.e., it does not depend on how computation tasks are scheduled.
>
> ## Process-based parallelism
>
> Transducers.jl supports process-based parallelism using [Distributed.jl](https://docs.julialang.org/en/latest/stdlib/Distributed/). You can use it by replacing `foldl` with `dreduce`. It can be used for horizontally scaling the computation. It is also useful for using external libraries that are not “thread-safe.”
>
> Note that [early termination is not supported in `dreduce` yet](https://github.com/tkf/Transducers.jl/issues/88).
### Misc news
- I managed to upstream a few transducers to Julia `Base`! It’ll be available in Julia 1.4. For example, it makes `sum(y for x in 1:1000 for y in 1:x if y % 2 == 0)` ~3x faster. See: [Transducer as an optimization: map, filter and flatten by tkf · Pull Request #33526 · JuliaLang/julia](https://github.com/JuliaLang/julia/pull/33526)
- Recent versions of Transducers.jl include [`withprogress`](https://tkf.github.io/Transducers.jl/dev/manual/#Transducers.withprogress) that can be used to monitor the progress of your computation. This is done by emitting [ProgressLogging.jl](https://github.com/JunoLab/ProgressLogging.jl)-compatible progress events. It will show progress bars if you use Juno, [ConsoleProgressMonitor.jl](https://github.com/tkf/ConsoleProgressMonitor.jl), or [TerminalLoggers.jl](https://github.com/c42f/TerminalLoggers.jl). It can be used with thread- and process-based parallel reduce.
- Transducers.jl now can be used to create various table types from DataFrames.jl, TypedTables.jl, StructArrays.jl, etc. See [`copy`](https://tkf.github.io/Transducers.jl/dev/manual/#Base.copy) and its parallel versions [`tcopy`](https://tkf.github.io/Transducers.jl/dev/manual/#Transducers.tcopy) and [`dcopy`](https://tkf.github.io/Transducers.jl/dev/manual/#Transducers.dcopy).
---
### Author:  [@carstenbauer](https://discourse.julialang.org/u/carstenbauer)
#### Post date: [December 15, 2019, 6:01am UTC](https://discourse.julialang.org/t/thread-and-process-based-parallelisms-in-transducers-jl-some-news/32285/2 "2019-12-15T06:01:36Z")
Great work!
---
### Author:  [@tkf](https://discourse.julialang.org/u/tkf)
#### Post date: [December 16, 2019, 3:10pm UTC](https://discourse.julialang.org/t/thread-and-process-based-parallelisms-in-transducers-jl-some-news/32285/3 "2019-12-16T15:10:58Z")
I forgot to mention this, but [Example for the Depth first multithread implementation performance gain as a motivation](https://discourse.julialang.org/t/example-for-the-depth-first-multithread-implementation-performance-gain-as-a-motivation/32328) reminded me that the early termination feature depends on that Julia scheduler being _depth-first_. The computed result is deterministic and scheduler independent. However, the depth-first scheduling makes it possible to terminate as early as possible by writing the reduction in divide-and-conquer approach. It makes the implementation very straightforward, if not trivial. A big thanks to Julia dev team!
---
### Author:  [@tkf](https://discourse.julialang.org/u/tkf)
#### Post date: [January 9, 2020, 1:51am UTC](https://discourse.julialang.org/t/thread-and-process-based-parallelisms-in-transducers-jl-some-news/32285/4 "2020-01-09T01:51:11Z")
Cross-posting:
> [@Multi-threaded worker processes](https://discourse.julialang.org/t/multi-threaded-worker-processes/33034/13):
>
> I just released a new version of Transducers.jl with a new `threads_basesize` option to [`dreduce`](https://tkf.github.io/Transducers.jl/dev/manual/#Transducers.dreduce). “Two-level” parallelism should automatically kick in if you set `JULIA_NUM_THREADS` appropriately in all workers; i.e., each worker process uses multiple threads.
---
### Author:  [@ianfiske](https://discourse.julialang.org/u/ianfiske)
#### Post date: [January 15, 2020, 7:23pm UTC](https://discourse.julialang.org/t/thread-and-process-based-parallelisms-in-transducers-jl-some-news/32285/5 "2020-01-15T19:23:56Z")
Does Transducers.jl support nested threaded parallelism similar to raw `@spawn`? That is, in the [contrived] example
```julia
using Transducers
function f1(x)
xs = x .+ rand(10000)
return reduce(+, Map(sin), xs)
end
reduce(+, Map(f1), 1:10000)
```
Is threaded-parallelism used at both the top-level `reduce` and also within each `f1`?
---
### Author:  [@tkf](https://discourse.julialang.org/u/tkf)
#### Post date: [January 15, 2020, 9:06pm UTC](https://discourse.julialang.org/t/thread-and-process-based-parallelisms-in-transducers-jl-some-news/32285/6 "2020-01-15T21:06:04Z")
Transducers.jl is implemented with `@spawn` in Julia \>= 1.3 so it naturally supports nested parallelism as in your example. (But I think it will crash in Julia \< 1.3.)