If you just want to check whether a method call will be “compiled away”, i.e. evaluated at compile-time if its arguments’ values are known at compile-time, looking at compiled code e.g. @code_llvm is way more reliable than timing.
I think you were confused by exactly what “compiled away” means in this context. When you do @btime f(123456789), you’re not benchmarking f(123456789), you’re actually benchmarking temp() = f(123456789). f(123456789) is called in the local scope of a temporary method, so when the method is compiled, the compiler decided to “compile away” the f(123456789) call to a constant value. This decision also depends on what f does, like f(x) = rand(x) could never be compiled away.
When you interpolate @btime f($123456789), you benchmark something like temp(x) = f(x). The top method call temp(123456789) is not compiled away, it uses the compiled code temp(::Int) to take in a runtime integer and calculate stuff at runtime. So it actually represents what happens when f(123456789) is evaluated at runtime.
Honestly I have no idea why @btime is designed that way, it’s a source of a lot of confusion especially because the unorthodox usage of $ in a macro call isn’t even a guarantee of preventing the benchmark from compiling the method call away. Just seems better if none of the arguments are ever assumed constant and we can elect to write our own temporary methods with constant arguments. I’m guessing it’s actually some difficult design problem involving the benchmark loop being run at global scope rather than the macro call scope.