Flux parallel execution

kkmann · August 6, 2018, 9:27am

Hi,

so, this might be a very dumb question but how do I train model in flux.jl in parallel on my cpu?

groot · December 10, 2018, 12:59am

Same question. MNIST demo in model zoo runs on a single core only. I am even surprised that Flux doesn’t run on multiple cores by default. Isn’t it mainly matrix multiplication which is paralleled by Julia?

Edit:

It does supported by julia. set JULIA_NUM_THREADS env variable will do the trick
I use following before running julia

export JULIA_NUM_THREADS=6

Tomas_Pevny · December 10, 2018, 5:33am

I have implemented a manual model-parallel computations, where each model is evaluated on a single thread and on the end their gradient is averaged. It is faster in some cases. But I believe that the overall goal should be to have underlying routines multi-threaded.

The implementation is here.

github.com

pevnak/FluxExtensions.jl/blob/master/src/learn.jl

using Printf, Serialization, BSON
function default_cb(step, load_time, calculation_time, training_loss)
	println(@sprintf("%d: training loss = %.4g load time= %.0fs (%.2fs per step)  compute time %.0fs (%.2fs per step) ",step , training_loss, load_time, load_time / step, calculation_time, calculation_time / step))
end

function wronggrad(model)
	ps = params(model)
	a = any(map(p -> any(isnan.(Flux.Tracker.grad(p))), ps)) 
	a && @warn("nan in a gradient")
	b = any(map(p -> any(isinf.(Flux.Tracker.grad(p))), ps)) 
	b && @warn("inf in a gradient")
	a || b
end
correctgrad(model) = !wronggrad(model)


"""
		serializestate(filename, model, loss, data)

		serialize the model and data to filename. Used mainly for debugging nans

This file has been truncated. show original

stegua · March 29, 2019, 3:48pm

In Windows you have to set the enviroment variable using the Control Panel.