Actually never mind, I figured it out.
Turns out this link does explain it after all 
If you make plots using svg graphics then font size is exact and you can save the image as png with any resolution.
For example to make 6.5" by 3" figure with fontsize 12:
using CairoMakie
CairoMakie.activate!(type = "svg")
size_inches = (6.5, 3)
size_pt = 72 .* size_inches
fig=Figure(resolution= size_pt, fontsize=12)
ax = Axis(fig[1, 1], title="Figure", xlabel="xlabel", ylabel="ylabel")
lines!(ax,[1,2,3,4,5],[1,2,3,4,5], label="line")
save("figure.png", fig, px_per_unit = 10)
#you can change px_per_unit to change resolution but relative sizes of font vs figure will stay the same.
EDIT: as pointed out by @jules, CairoMakie.activate!(type = "svg") actually doesn’t matter and as long as you set figure size (e.g., 6.5" by 3") using code below, paste the figure in word (or any text editor) and rescale it to exactly figure size (e.g., 6.5" by 3") then font will be exactly correct. You can also increase px_per_unit or pt_per_unit to increase resolution of your .png figure if it appears pixelated but that will not change the font size of image rescale to figure size (e.g., 6.5" by 3").
size_inches = (6.5, 3)
size_pt = 72 .* size_inches