I had a similar desire. The @with construct is fine for coding but not very convenient for interactive work. My thought was not to have a mutable value but rather to enter and leave scope with separate commands, something like
julia> @newscope x => 2.5
julia> # some interactive coding
julia> @endscope
Based on the answers to my question How do ScopedValues actually work? I gather that scope changes are implemented in core Julia and thus something like this cannot be implemented in a package. But perhaps it is doable in principle?