I found the answer in some Issue which skipped my first 100 search attempts…
https://github.com/JuliaPlots/Plots.jl/issues/2596#issuecomment-616054537
You should explicitly include a font that is known to GR() like this:
import Plots
using Plots
plot(exp.(0:0.01:4), fontfamily="helvetica"))
savefig("test.svg")
and voila, text is text in your svg.