It’s been a while since I added parallelism supports in Transducers.jl but I’ve never announced this feature properly. I just added a few utility functions and a tutorial so I think it’s good timing to do this. Quoting Overview of parallel processing in Transducers.jl : Transducers.jl supports th…

I forgot to mention this, but Example for the Depth first multithread implementation performance gain as a motivation reminded me that the early termination feature depends on that Julia scheduler being depth-first. The computed result is deterministic and scheduler independent. However, the depth…

Cross-posting: [image] Multi-threaded worker processes I just released a new version of Transducers.jl with a new threads_basesize option to dreduce . “Two-level” parallelism should automatically kick in if you set JULIA_NUM_THREADS appropriately in all workers; i.e., each worker process uses m…

Does Transducers.jl support nested threaded parallelism similar to raw @spawn? That is, in the [contrived] example using Transducers function f1(x) xs = x .+ rand(10000) return reduce(+, Map(sin), xs) end reduce(+, Map(f1), 1:10000) Is threaded-parallelism used at both the top-level red…

Transducers.jl is implemented with @spawn in Julia >= 1.3 so it naturally supports nested parallelism as in your example. (But I think it will crash in Julia < 1.3.)

Thread- and process-based parallelisms in Transducers.jl (+ some news)

Package Announcements

tkf January 9, 2020, 1:51am 4

Cross-posting:

Topic		Replies	Views
Parallel implementation of Transducers.jl General Usage parallel	1	559	February 22, 2021
Writing effective parallel code Performance parallel	8	1827	December 18, 2019
ANN: Transducers.jl 0.3. taking "zeros" seriously, type stability improvements, fusible groupby, OnlineStats, "GPU support", and more Package Announcements	29	3304	December 2, 2020
Some questions on Transducers.jl General Usage	6	1040	February 22, 2021
Transducers and reduct-like reduction Performance	3	511	August 24, 2021

Thread- and process-based parallelisms in Transducers.jl (+ some news)

Related topics