Actually, keys are for the normal case easy, but you need to have a public key. I have
$ ls ~/.ssh
id_rsa id_rsa.pub known_hosts
and everything works smoothly without any prompting. If I move id_rsa.pub I get prompted about the private key (with correct default) and then about public key (with no default).