Agreed, it looks like a bug:
julia> SVD = svd(A)
LinearAlgebra.SVD{Float64, Float64, Matrix{Float64}}
U factor:
3×2 Matrix{Float64}:
1.00000e+00 0.00000e+00
0.00000e+00 1.00000e+00
0.00000e+00 0.00000e+00
singular values:
2-element Vector{Float64}:
1.00000e+00
1.00000e+00
Vt factor:
2×2 Matrix{Float64}:
1.00000e+00 0.00000e+00
0.00000e+00 1.00000e+00
julia> (SVD.Vt * Diagonal(SVD.S)) * SVD.U' * A
2×2 Matrix{Float64}:
1.00000e+00 0.00000e+00
0.00000e+00 1.00000e+00