Flux bilinear upsampling

Specific Domains Machine Learning
1

What is the best way to use Flux to create a bilinear upsampling layer? Something analogous to tensorflow bilinear upsample.

I appreciate your insight in advance.

1 Like
2

Decided to go for it.

1 Like
3

The implementation effort has moved to this pull request.

I got the forward part working for both CPU and GPU, but gradient doesn’t work in the GPU case (it does in the CPU case). It seems hard to debug because Juno’s debugger crashes when I try to get into the gradient call. Who here has more experience getting gradients through with CuArrays?

x = Float32.([1 2; 3 4])[:,:,:,:]
x_c = CuArray(x)

c = BilinearUpsample2d((2,2))
c_c = gpu(c) #don't think this does anything, since there are no arrays stored

o = c(x)
o_c = c(x_c)

g = gradient(x -> sum(c(x)), x)[1]
g_c = gradient(x -> sum(c(x)), x_c)[1]
@Juno.run gradient(x -> sum(c(x)), x_c) #crashes on my setup

The error it gives is “unrecognized isdefined node $(QuoteNode(Float32))”, can’t find anything on it

My setup is Julia 1.4.1, Flux 0.10.5, Atom 1.46.0

Any ideas?

4

Hi, I cannot directly help with your problem, but maybe we can find another solution in separating CPU and GPU code. I have translated pytorch’s implementation of bilinear upsampling to julia and it seems to work more or less, even with fractional upsampling. I just dont have experience with zygote and adjoints. You can have a look at the implementation here and try it out with the following code. The forward pass looks good, but I havent checked the backward pass.

using FileIO
using ImageView  # ZZZzzzz
using Colors
using BenchmarkTools
f = download("https://upload.wikimedia.org/wikipedia/en/e/ed/Nyan_cat_250px_frame.PNG")
nyan = load(f)

imshow(nyan)

nyan_nchw = reshape(reinterpret(UInt8, nyan),1,3,250,250)
nyan_whcn = permutedims(nyan_nchw, [3,4,2,1]) .|> Float32
nyan_gpu = CuArray(nyan_whcn)

nyan_large = upsample_bilinear(nyan_gpu, pi, pi)

nyan_large_cpu = Array(nyan_large)[:,:,:,1]
nyan_colored = view(reinterpret(RGB{Float32}, permutedims(nyan_large_cpu, [3,1,2])),1,:,:)  # what a mess

imshow(nyan_colored/255)

# or plain arrays:
n = 64
s = 4
checkerboard = zeros(Float32, n, n)
for x in 1:2s:n-s
    for y in 1:2s:n-s
        checkerboard[y:y+s-1, x:x+s-1] .= 1
    end
end
imshow(checkerboard)

checkerboard_gpu = CuArray(reshape(checkerboard,n,n,1,1))

res = Array(upsample_bilinear(checkerboard_gpu, 2, 2))

imshow(res[:,:,1,1])
imshow(round.(res[:,:,1,1]))

Maybe we you can compare this to your results and have a look at the backward pass.

EDIT: I just saw this, which also looks interesting!