# Why does binarycrossentropy needs an index in a denoising autoencoder?

**URL:** <https://discourse.julialang.org/t/why-does-binarycrossentropy-needs-an-index-in-a-denoising-autoencoder/41013>\
**Category:** Machine Learning\
**Tags:** flux\
**Created:** [June 8, 2020, 7:20pm UTC](https://discourse.julialang.org/t/why-does-binarycrossentropy-needs-an-index-in-a-denoising-autoencoder/41013 "2020-06-08T19:20:08Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![swuyts](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/swuyts/32/12518_2.png) [@swuyts](https://discourse.julialang.org/u/swuyts)\
**Post date:** [June 8, 2020, 7:20pm UTC](https://discourse.julialang.org/t/why-does-binarycrossentropy-needs-an-index-in-a-denoising-autoencoder/41013/1 "2020-06-08T19:20:08Z")

</div>

Hi there,

I am still somewhat a rookie in Julia and Flux and I have problem understanding what is going on when I switch between `crossentropy` and `binarycrossentropy` loss functions.

I coded the following simple denoising autoencoder:

```julia
using Flux, Random

data = rand(2000,100)
data_corrupted = copy(data)

# Corrupt data
for sample_index in 1:size(data)[2]
        # Create random indices
        rng = MersenneTwister(1234)
        indices = findall(bitrand(rng, 2000) .> 0)

                # Change values at indices to 0
                for i in 1:size(indices)[1]
                        data_corrupted[indices[i], sample_index] = 0
                end
end

# Partition into batches of 10
data = [data[:, i:min(i+10-1,size(data, 2))] for i in 1:10:size(data, 2)]
data_corrupted = [data_corrupted[:, i:min(i+10-1,size(data_corrupted, 2))] for i in 1:10:size(data_corrupted, 2)]

# Define model
encoder = Dense(2000, 50, σ)
decoder = Dense(50, 2000, σ)
m = Chain(encoder, decoder)

# Defining the loss function
loss(x, y) = Flux.crossentropy(m(x), y)

# Defining the optimiser
opt = ADAM()

# Train
Flux.train!(loss, params(m), zip(data_corrupted, data), opt)

```

This runs fine.

But if I then change the loss function to:

```julia
loss(x, y) = Flux.binarycrossentropy(m(x), y)

```

I get the following error:

```julia
ERROR: LoadError: MethodError: no method matching eps(::Array{Float32,2})
Closest candidates are:
  eps(!Matched::Dates.Time) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Dates/src/types.jl:387
  eps(!Matched::Dates.Date) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Dates/src/types.jl:386
  eps(!Matched::Dates.DateTime) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Dates/src/types.jl:385
 ...

```

However if I change the loss to what has been suggested [here](https://github.com/FluxML/Flux.jl/issues/850):

```julia
loss(x, y) = Flux.binarycrossentropy(m(x)[1], y[1])

```

The model trains without any problem.

I have a hard time understanding why I need this indexing for the `binarycrossentropy`, while I do not need it for the `crossentropy`. I understand that the `eps` function requires an array of dim 1, but I am confused as whether it will now calculate the loss only on the first batch instead of on all data.

Any insights are very welcome!

Many thanks,  
Sander

---

<div class="post-metadata">

**Author:** ![contradict](https://avatars.discourse-cdn.com/v4/letter/c/ac91a4/32.png) [@contradict](https://discourse.julialang.org/u/contradict)\
**Post date:** [June 8, 2020, 9:48pm UTC](https://discourse.julialang.org/t/why-does-binarycrossentropy-needs-an-index-in-a-denoising-autoencoder/41013/2 "2020-06-08T21:48:05Z")

</div>

The loss function needs to return a [scalar](https://fluxml.ai/Flux.jl/v0.10/training/training/#Loss-Functions-1).

From the definitions of [`crossentropy`](https://fluxml.ai/Flux.jl/stable/models/layers/#Flux.crossentropy) and [`binarycrossentropy`](https://fluxml.ai/Flux.jl/stable/models/layers/#Flux.binarycrossentropy), `crossentropy` includes a sum over the model output, while `binarycrossentropy` expects a scalar model output.

Try broadcasting `binarycrossentropy` over the output and summing like this:

```julia
loss(x, y) = sum(Flux.binarycrossentropy.(m(x), y)) 

```

---

<div class="post-metadata">

**Author:** ![swuyts](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/swuyts/32/12518_2.png) [@swuyts](https://discourse.julialang.org/u/swuyts)\
**Post date:** [June 9, 2020, 7:12am UTC](https://discourse.julialang.org/t/why-does-binarycrossentropy-needs-an-index-in-a-denoising-autoencoder/41013/3 "2020-06-09T07:12:44Z")

</div>

That makes a whole lot of sense and works like a charm. Thank you for pointing this out to me!