It’s been a while since I added parallelism supports in Transducers.jl but I’ve never announced this feature properly. I just added a few utility functions and a tutorial so I think it’s good timing to do this.
Transducers.jl supports thread-based (
reduce) and process-based (
dreduce) parallelisms with the same composable API; i.e. transducers. Having a uniform API to cover different parallelisms as well as sequential processing
foldlis useful. Using multiple cores or machines for your computation is as easy as replacing
dreduce; you don’t need to re-write your transducers or reducing functions.
Transducers.jl supports thread-based parallelism for Julia ≥ 1.0. You can use it by replacing
reduce. With Julia ≥ 1.3, Transducers.jl supports early termination to avoid unnecessary computation while guaranteeing the result to be deterministic; i.e., it does not depend on how computation tasks are scheduled.
Transducers.jl supports process-based parallelism using Distributed.jl. You can use it by replacing
dreduce. It can be used for horizontally scaling the computation. It is also useful for using external libraries that are not “thread-safe.”
- I managed to upstream a few transducers to Julia
Base! It’ll be available in Julia 1.4. For example, it makes
sum(y for x in 1:1000 for y in 1:x if y % 2 == 0)~3x faster. See: Transducer as an optimization: map, filter and flatten by tkf · Pull Request #33526 · JuliaLang/julia
- Recent versions of Transducers.jl include
withprogressthat can be used to monitor the progress of your computation. This is done by emitting ProgressLogging.jl-compatible progress events. It will show progress bars if you use Juno, ConsoleProgressMonitor.jl, or TerminalLoggers.jl. It can be used with thread- and process-based parallel reduce.
- Transducers.jl now can be used to create various table types from DataFrames.jl, TypedTables.jl, StructArrays.jl, etc. See
copyand its parallel versions