This is probably not responsible for the entire 2x speed difference, but I have noticed Flux.Conv layers are not type-stable, which I guess would reduce performance. Opened a github issue here and a thread here.
Because of the lack of response I am unsure whether this type-instability is actually a big deal though, I am quite new to Julia.