Yes, that is how ssh-agent words. It isn’t nice, but it is how it works for everyone.
ssh-agent
(I am not a fan of ssh-agent)