Dot equals .=

General Usage Performance
1

I am sort of getting fuzzy on how to apply syntactic loop fusion from More Dots - Syntactic Loop Fusion in Julia | juliabloggers.com

Is this:

nnp.theta[hl] .= nnp.theta[hl] .+ (hp.alphaovermb .* (hp.lambda .* nnp.theta[hl]))

the same as this:

nnp.theta[hl][:] = nnp.theta[hl] .+ (hp.alphaovermb .* (hp.lambda .* nnp.theta[hl]))

theta[hl] is the weight matrix for one hidden layer in a neural net. Note that nnp.theta is an array containing the weight matrices for each layer. Indexing on [hl] gets the matrix for one layer, in this case 128 x 256. I pre-allocate all of the weight matrices and want to update in place. When I first did this way back with Julia 0.6, I used the second form. Then, I switched to the first form. What I forget is whether, in addition to fusing the loop for the right-hand-side, this also assigns in place to the left-hand-side.

I tested each way and got same results and timing essentially the same so the answer must be yes, these do the same thing. I use this all over the place to update weights, biases, and training data. Just trying to make sure I’ve done reasonable optimization. I also tried a manual loop but must have done something wrong as the results were different and it was much slower.

For minibatch training, I replaced a sliced based approach (each minibatch updating a pre-allocated “slice” holder) with a view based approach. I seem to think this made things slower. It’s a fair bit of work to go back, but I put back some of the slice-based plumbing to do that comparison. It seemed much slower.

I save timings of all runs, so I can go back and see if my memory is right…

2

No, they are not the same:

julia> using BenchmarkTools

julia> a,b,c = map(x->rand(100,100), 1:3);

julia> @btime $a[:] = $b.+$c;
  11.949 μs (2 allocations: 78.20 KiB)

julia> @btime $a .= $b.+$c;
  4.503 μs (0 allocations: 0 bytes)

i.e. the first version creates a temporary array and then copies this to a whereas the second one is +/- equivalent to:

for i=1:length(a)
  a[i] = b[i] + c[i]
end

No idea though how this impacts your use-case.

3

.= would win

My testing didn’t show such a large difference. The key is that .= does assign to an indexed left hand side, so a new object is not be assigned to the name, rather the existing storage is being updated. There must also be a difference
in not creating any temporaries on the right hand side, though I was under the impression (perhaps incorrectly) that dotting every operation would also achieve that.

What is the purpose of the $’s in front of variables?

Thanks.

4

The first one is +/- equivalent to

tmp = similar(a)
for i=1:length(a)
  tmp[i] = b[i] + c[i]
end
copy!(a, tmp)

The $ are a trick for the BenchmarkTools macros to make sure the referenced variables are not non-const globals, as that would impact performance.

Yep, always profile before spending lots of time on an optimisation which doesn’t do much.