PReLU with long compilation time

cirobr
Tue, 10 Jun 2025 12:48:25 +0000
Performance

Cheers,

I have implemented a PReLU nonlinearity that seems to be working quite well in my models, except for the fact the model’s compilation time (*) is way longer than the equivalent model with ReLU instead.

Given my lack of deeper knowledge to go further, may I please ask for your help and advise.

Thanks in advance.

(*) Precompilation prior to running an epoch.

preluweights(ch_in::Int) = DepthwiseConv((1, 1), ch_in => ch_in;
                                 bias=false,
                                 init=rand32
)

struct ConvPReLU
    conv
end
@layer ConvPReLU

function ConvPReLU(ch_in::Int)
    return ConvPReLU(preluweights(ch_in))
end

function (m::ConvPReLU)(x)
    return max.(x, 0) .+ m.conv(min.(x, 0))
end

2 posts - 1 participant

Read full topic