Yes!
I have also learned a lot from you and @c42f. Thank you so much.
What I wonder now is why Ref{Int} is so slow here.
Found the answer in another post.
After specifying the Ref{Int} fields as Base.RefValue{Int}, I got almost the same performance gains (but still slower than mutable struct + Int).
=== Using compiler ===
10.557 s (583631370 allocations: 19.09 GiB)