One option might be to not include a Manifest.toml in your git repository. Most Julia library packages do not include a Manifest.toml in the git repository, in part for this reason.
If you really want to test using a Manifest.toml, you could have a distinct Manifest.toml for each Julia minor version within separate project folders that include the main package that lacks the Manifest.toml. During different CI runs, you would use Pkg.activate to choose a different Manifest.toml based on the Julia minor version.