Nice finds!
Yup, I think this is possible on current CUDA.jl as well. However, multi-gpu training of the same model (whether on the same machine or across machines) requires functionality that isn’t implemented anywhere in the Flux ecosystem yet.
Seems like all of these benchmarks are prior to the Flux Zygote transition (I see lots of Tracker). Knet likely performs even better now (and should probably receive more love), but Tracker → Zygote was a noticeable performance regression for certain workflows. ref. https://fluxml.ai/2020/06/29/acclerating-flux-torch.html, Flux vs pytorch cpu performance - #15 by d1cker, Poor performance relative to PyTorch · Issue #886 · FluxML/Flux.jl · GitHub.
For more holistic comparisons, see also
Is it a good time for a PyTorch developer to move to Julia? If so, Flux? Knet? - #18 by dfdx and https://discourse.julialang.org/t/where-does-julia-provide-the-biggest-benefits-over-other-ml-frameworks-for-research (the latter was started by a PyTorch contributor).