Do empty!(ax) instead of empty!(ax.scene), the latter kind of pulls the rug out from under the Axis by removing some connections needed for the interactions
Do empty!(ax) instead of empty!(ax.scene), the latter kind of pulls the rug out from under the Axis by removing some connections needed for the interactions