The problem with this mixing of atomic/non-atomic operations on the same types/data is that you loose all benefit of statically ensured atomicness in the first place. If you allow both to happen in principle on the same data, you open yourself/your code up to the possibility of some non-atomic code running during the atomic portion, making a mess of things (which doesn’t have to be initiated by you, even if the code you’ve written would not mess with things. A user library loading your code could just as well mess with the data in an unsafe manner). The solution here is to seperate these two concepts: Work on atomic types when requiring atomics and moving the data to non-atomic types when not requiring them. I suspect this moving is more expensive than just keeping the atomics around though.
There is a sort of middle ground in the form of lock-free programming, but that only applies to algorithms, not data as you’re proposing.