# Data-parallel training with conv nets in Julia

**URL:** <https://discourse.julialang.org/t/data-parallel-training-with-conv-nets-in-julia/12520>\
**Category:** Machine Learning\
**Tags:** distributed\
**Created:** [July 20, 2018, 4:41am UTC](https://discourse.julialang.org/t/data-parallel-training-with-conv-nets-in-julia/12520 "2018-07-20T04:41:12Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![jstrube](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jstrube/32/525_2.png) [@jstrube](https://discourse.julialang.org/u/jstrube)\
**Post date:** [July 20, 2018, 4:41am UTC](https://discourse.julialang.org/t/data-parallel-training-with-conv-nets-in-julia/12520/1 "2018-07-20T04:41:12Z")

</div>

I have a problem that lends itself well to data-parallel training (see, e.g., [CHEP 2018 Conference, Sofia, Bulgaria (9-13 July 2018): Scaling studies for deep learning in LArTPC event classification · Indico](https://indico.cern.ch/event/587955/contributions/2937539/)). I’ve been using a library that goes on top of tensorflow ([GitHub - matex-org/matex: Machine Learning Toolkit for Extreme Scale (MaTEx)](https://github.com/matex-org/matex)), but I’m facing some issues with tensorflow on a new machine, and I’m wondering if Flux or one of the other deep learning libs in julia couldn’t do the job as well.  
Does anybody have experience with training on multiple nodes in julia? Can I just use DistributedArrays for my data sets somehow and everything magically works?  
Any pointers would be appreciated.

---

<div class="post-metadata">

**Author:** ![Tomas\_Pevny](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tomas_pevny/32/25466_2.png) [@Tomas\_Pevny](https://discourse.julialang.org/u/Tomas_Pevny)\
**Post date:** [July 20, 2018, 5:04am UTC](https://discourse.julialang.org/t/data-parallel-training-with-conv-nets-in-julia/12520/2 "2018-07-20T05:04:16Z")

</div>

I have did that with Flux, running a separate copy of each model on different thread and than averaging the results. It was relatively easy and it was on the end couple lines of code.

I did that, since I need multiplication with large sparse matrices. On the end, I was about three times faster on CPU than Tensorflow on GPU .

---

<div class="post-metadata">

**Author:** ![jstrube](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jstrube/32/525_2.png) [@jstrube](https://discourse.julialang.org/u/jstrube)\
**Post date:** [July 20, 2018, 5:12am UTC](https://discourse.julialang.org/t/data-parallel-training-with-conv-nets-in-julia/12520/3 "2018-07-20T05:12:02Z")

</div>

Cool! Do you have an example that you could share?

---

<div class="post-metadata">

**Author:** ![Tomas\_Pevny](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tomas_pevny/32/25466_2.png) [@Tomas\_Pevny](https://discourse.julialang.org/u/Tomas_Pevny)\
**Post date:** [July 20, 2018, 6:21am UTC](https://discourse.julialang.org/t/data-parallel-training-with-conv-nets-in-julia/12520/4 "2018-07-20T06:21:06Z")

</div>

I can share a snippet, but it was like this.

say `model` is your model, I did

```julia
function _back!(model,loss,ds)
  l = loss(model,ds)
  isinf(l.tracker.data) && error("inf in the model");
  isnan(l.tracker.data)&& error("nan in the model");
  Flux.Tracker.back!(l)
  l.tracker.data
end

function Flux.Tracker.back!(pars,models,parss,dss,loss)
  foreach(s -> copy!(s,pars),parss)
  l = zeros(nthreads())
  @threads for i in 1:length(dss) 
    l[i] += _back!(models[i],loss,dss[i])
  end
  mean!(pars,parss...)
  l
end

models = [deepcopy(model) for i in 1:nthreads()]
parss = map(params,models)
pars = params(model)
Flux.Tracker.back!(pars,models,parss,dss,loss)

```

`dss` was a vector with data for each thread and and `loss` was the loss function.

You might need to implement some of the missing functions, like taking average of parameters or copying them, but you should have the idea, how I did it.

Hope this has helped.

---

<div class="post-metadata">

**Author:** ![jstrube](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jstrube/32/525_2.png) [@jstrube](https://discourse.julialang.org/u/jstrube)\
**Post date:** [July 20, 2018, 6:28am UTC](https://discourse.julialang.org/t/data-parallel-training-with-conv-nets-in-julia/12520/5 "2018-07-20T06:28:24Z")

</div>

Thanks! I’m sure I’ll take a while to get up to speed (total Flux newbie), but I appreciate your help.