# Implementation of composite quantile loss for Flux

**URL:** <https://discourse.julialang.org/t/implementation-of-composite-quantile-loss-for-flux/73978>\
**Category:** Machine Learning\
**Tags:** gpu, performance, flux\
**Created:** [January 3, 2022, 5:49pm UTC](https://discourse.julialang.org/t/implementation-of-composite-quantile-loss-for-flux/73978 "2022-01-03T17:49:58Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![johnbb](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnbb/32/34233_2.png) [@johnbb](https://discourse.julialang.org/u/johnbb)\
**Post date:** [January 3, 2022, 5:49pm UTC](https://discourse.julialang.org/t/implementation-of-composite-quantile-loss-for-flux/73978/1 "2022-01-03T17:49:58Z")

</div>

The composite quantile loss for a set of quantiles can be defined as

\sum\_i \sum\_ j (p\_j - 1\{y\_i \< q\_{ji}\}) (y\_i - q\_{ji})

where p\_j \in(0,1) denote the _j_-th quantile level/probability, y\_i the _i_-th target/observation, and q\_{ji} the p\_j-quantile for the _i_-th sample. For a year or two I have been using the following implementations

```julia
# qt size: level, case
function qtloss(qt::AbstractMatrix, y::AbstractVector, prob::AbstractVector; agg = mean)
    err = y .- qt' # case x level
    agg( (prob' .- (err .< 0)) .* err )    
end

function qtlossMax(qt::AbstractMatrix, y::AbstractVector, prob::AbstractVector; agg = mean)
    err = y .- qt'
    agg( max.(prob' .* err, (prob' .- 1) .* err) )
end

```

where the former is slightly faster in my experience. Btw, are there better alternatives with respect to speed and memory usage?

Now I would like to extend this implementation to Flux “image” models with data dimensions WHCN. C would here be the level/probability dimension and the dimensions WHN the “_i_-summation”. Any suggestions would be appreciated, in particular for GPUs. Reducing to the 2D-case with permutedims/reshape could be one option, but …

---

<div class="post-metadata">

**Author:** ![ToucheSir](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/touchesir/32/14411_2.png) [@ToucheSir](https://discourse.julialang.org/u/ToucheSir)\
**Post date:** [January 3, 2022, 9:45pm UTC](https://discourse.julialang.org/t/implementation-of-composite-quantile-loss-for-flux/73978/2 "2022-01-03T21:45:13Z")

</div>

Instead of reshaping the inputs down, you could pad `prob` and `y` with size-1 dims. That would also save a transpose on `qt` and `prob` down the line.

> [@johnbb](#):
>
> Btw, are there better alternatives with respect to speed and memory usage?

[GitHub - mcabbott/Tullio.jl: ⅀](https://github.com/mcabbott/Tullio.jl) is worth a try here.

---

<div class="post-metadata">

**Author:** ![johnbb](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnbb/32/34233_2.png) [@johnbb](https://discourse.julialang.org/u/johnbb)\
**Post date:** [January 4, 2022, 10:33am UTC](https://discourse.julialang.org/t/implementation-of-composite-quantile-loss-for-flux/73978/3 "2022-01-04T10:33:50Z")

</div>

@ToucheSir Not sure I fully understand, but I made a test using `permutedims` only (without Tullio) as follows

```julia
using Flux, Statistics, BenchmarkTools

# size qt: WHCN where C is no. of quantiles
function qtloss(qt::AbstractArray{T,4}, y::AbstractArray{T,4}, prob::AbstractVector{T}; agg = mean) where T<:Real
    err = permutedims(y, (3,1,2,4)) .- permutedims(qt, (3,1,2,4))
    agg( (prob .- (err .< 0)) .* err )
end

function create_data(; d = 256, vars = 25, n = 32, nprob = 20, device = gpu)
    x = rand(Float32, d, d, vars, n) |> device
    y = rand(Float32, d, d, 1, n) |> device
    prob = Float32.(range(0, 1, length=nprob+2)[2:end-1]) |> device
    return x, y, prob
end

device = gpu
x, y, prob = create_data(device = device);
opt = ADAM()
model = Conv((3,3), size(x,3) => length(prob), relu, pad=SamePad()) |> device
loss(x, y) = qtloss(model(x), y, prob)
@btime Flux.train!(loss, Flux.params($model), [($x,$y)], $opt)
# RTX2070: 24.496 ms (1124 allocations: 101.52 KiB)
# cpu: 4.632 s (540 allocations: 2.58 GiB)

```

Training times seem reasonable, but I do not understand the huge difference in memory allocated. Maybe the `permutedims` could also have been applied before the loss function, but not sure it would make any difference.

---

<div class="post-metadata">

**Author:** ![ToucheSir](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/touchesir/32/14411_2.png) [@ToucheSir](https://discourse.julialang.org/u/ToucheSir)\
**Post date:** [January 8, 2022, 6:58pm UTC](https://discourse.julialang.org/t/implementation-of-composite-quantile-loss-for-flux/73978/4 "2022-01-08T18:58:56Z")

</div>

This is what I meant by pre-padding:

```julia
function qtloss2(qt::AbstractArray{T,4}, y::AbstractVector{T}, prob::AbstractVector{T}; agg = mean) where T<:Real
    err = reshape(y, 1, 1, 1, :) .- qt
    prob = reshape(prob, 1, 1, :)
    agg((prob .- (err .< 0)) .* err)
end

```

Allocations and runtime are still higher than I’d expect, but lower than the version with `permutedims`:

```julia-repl
julia> @btime gradient($qtloss, $x, $y, $prob);
  969.884 ms (83 allocations: 1.59 GiB)

julia> @btime gradient($qtloss2, $x, $y2, $prob);
  580.226 ms (78 allocations: 1.25 GiB)

```

---

<div class="post-metadata">

**Author:** ![johnbb](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnbb/32/34233_2.png) [@johnbb](https://discourse.julialang.org/u/johnbb)\
**Post date:** [January 9, 2022, 7:40pm UTC](https://discourse.julialang.org/t/implementation-of-composite-quantile-loss-for-flux/73978/5 "2022-01-09T19:40:40Z")

</div>

Hmm. What is `y2` here? In the WHCN case, the targets `y` has dimension W,H,1,N.

---

<div class="post-metadata">

**Author:** ![ToucheSir](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/touchesir/32/14411_2.png) [@ToucheSir](https://discourse.julialang.org/u/ToucheSir)\
**Post date:** [January 9, 2022, 7:45pm UTC](https://discourse.julialang.org/t/implementation-of-composite-quantile-loss-for-flux/73978/6 "2022-01-09T19:45:38Z")

</div>

I didn’t glean that from the spec, so I just made `y2` a vector of targets like the original 1-D case. Since `y` is already the correct shape for broadcasting, you can remove the `reshape` on the first line and everything should still work just fine.

---

<div class="post-metadata">

**Author:** ![johnbb](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnbb/32/34233_2.png) [@johnbb](https://discourse.julialang.org/u/johnbb)\
**Post date:** [January 9, 2022, 10:34pm UTC](https://discourse.julialang.org/t/implementation-of-composite-quantile-loss-for-flux/73978/7 "2022-01-09T22:34:02Z")

</div>

Thanks @ToucheSir . Is there any reason for using `reshape(prob, 1, 1, :)` instead of `reshape(prob, 1, 1, :, 1)`?

---

<div class="post-metadata">

**Author:** ![ToucheSir](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/touchesir/32/14411_2.png) [@ToucheSir](https://discourse.julialang.org/u/ToucheSir)\
**Post date:** [January 9, 2022, 10:55pm UTC](https://discourse.julialang.org/t/implementation-of-composite-quantile-loss-for-flux/73978/8 "2022-01-09T22:55:27Z")

</div>

No reason in particular, it happened to save a couple of characters for this example. I think it _might_ also be easier to write a helper function that pads this way, but I haven’t actually tried to do so.