[3,2,1] is a Vector{Int64}, and the basic Flux layers are only guaranteed to work with floating point types. That you can pass an int array into Dense is a happy coincidence, and even then not guaranteed to work in perpetuity. Try Float32[3,2,1] or allocating a float array by any other means instead, and the RNNs should work as expected.
As for this:
This is probably not doing what you want or expect of it. Broadcasting over an array applies the broadcasted function element-wise, so this is equivalent to [my_model(1), my_model(2), my_model(3)]. See Recurrence ยท Flux for a guide on how to properly use sequences with Flux RNNs.