OK, I found the solution myself now.
Basically this: Ordering of axes & dimensions · Issue #173 · JuliaImages/juliaimages.github.io · GitHub
So apparently the x and y axes of the warp function are swapped.
Therefore creating the linear transformation using img1_points[:, [2,1]] and img2_points[:, [2,1]] works.