# How to apply an activation function to a subset of output units?

**URL:** <https://discourse.julialang.org/t/how-to-apply-an-activation-function-to-a-subset-of-output-units/33202>\
**Category:** Machine Learning\
**Tags:** flux\
**Created:** [January 10, 2020, 3:34pm UTC](https://discourse.julialang.org/t/how-to-apply-an-activation-function-to-a-subset-of-output-units/33202 "2020-01-10T15:34:37Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![johnbb](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnbb/32/34233_2.png) [@johnbb](https://discourse.julialang.org/u/johnbb)\
**Post date:** [January 10, 2020, 3:34pm UTC](https://discourse.julialang.org/t/how-to-apply-an-activation-function-to-a-subset-of-output-units/33202/1 "2020-01-10T15:34:37Z")

</div>

Using Flux I would like to apply an activation function to all but the first output unit. More precisely, I would like to constrain all but the first unit to be non-negative. How can I achieve that? Thanks.

---

<div class="post-metadata">

**Author:** ![holylorenzo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/holylorenzo/32/34169_2.png) [@holylorenzo](https://discourse.julialang.org/u/holylorenzo)\
**Post date:** [January 11, 2020, 4:30pm UTC](https://discourse.julialang.org/t/how-to-apply-an-activation-function-to-a-subset-of-output-units/33202/2 "2020-01-11T16:30:47Z")

</div>

Since in Flux, all layers and activation functions are just functions, this is straightforward to implement.

Let’s say we’re working with a `Dense` layer with a `relu` activation. Now usually you would construct your layer with `Dense(nin, nout, relu)` for `relu` to be applied to every output of the `Dense` layer.

We can write a custom activation layer that applies a regular activation function to all but the first output as follows:

```julia
struct PartialActivation
    activationfn
end
(pa::PartialActivation)(xs) = map((i, x) -> i == 1 ? x : pa.activationfn(x), eachindex(xs), xs)

```

This will apply the `activationfn` to all but the first element.

To use it in a model, you will have to switch from something that probably looks like

```julia
Chain(.., Dense(10, 10, relu), ...)

```

to

```julia
Chain(.., Dense(10, 10), PartialActivation(relu) ...)

```

Hope this helps and feel free to ask questions!

---

<div class="post-metadata">

**Author:** ![johnbb](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnbb/32/34233_2.png) [@johnbb](https://discourse.julialang.org/u/johnbb)\
**Post date:** [January 13, 2020, 9:38am UTC](https://discourse.julialang.org/t/how-to-apply-an-activation-function-to-a-subset-of-output-units/33202/3 "2020-01-13T09:38:16Z")

</div>

Thanks, but it does not seem to work for arrays of dimension larger than one. In my case Flux expects 2-dimensional arrays (number of outputs x batch size). That is, the function should be applied to all but the first row.

I also consider to circumvent the problem by adding a sufficiently high number to the targets/responses such that the transformation can be applied to all output variables.

---

<div class="post-metadata">

**Author:** ![johnbb](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnbb/32/34233_2.png) [@johnbb](https://discourse.julialang.org/u/johnbb)\
**Post date:** [February 20, 2023, 11:19am UTC](https://discourse.julialang.org/t/how-to-apply-an-activation-function-to-a-subset-of-output-units/33202/4 "2023-02-20T11:19:54Z")

</div>

I have since the previous posting three years ago avoided the problem by adding a sufficiently high number to the targets whenever needed, but I would now like to keep the targets as they are.

Is there a better way than the following?

```julia
s(x::AbstractArray{Float32,2}) = vcat(x[[1], :], softplus(x[2:end, :]))
Chain(...., Dense(10, 10), s)

```

---

<div class="post-metadata">

**Author:** ![ToucheSir](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/touchesir/32/14411_2.png) [@ToucheSir](https://discourse.julialang.org/u/ToucheSir)\
**Post date:** [February 21, 2023, 12:34am UTC](https://discourse.julialang.org/t/how-to-apply-an-activation-function-to-a-subset-of-output-units/33202/5 "2023-02-21T00:34:19Z")

</div>

Looks pretty clean to me. You could save an allocation with `x[1:1, :]` instead of `x[[1], :]` and by converting the indexing to views (though I’m not sure about GPU compatibility there), but that’s very minor.