This comment is informative on why Base.parse doesn’t do what Meta.parse can, I’ll just copy the text too:
In general, parse(T, string) isn’t related to Julia syntax. parse(Int, "1_000") doesn’t work either, even though 1_000 is a valid Julia integer literal, whereas parse(DateTime, "2022-04-26T15:15") works even though 2022-04-26T15:15 is not Julia syntax. What we accept in parse is more about what we’re likely to find in external data sources (e.g. CSV files).
Note that the returned object is a Float32, not a Float64, which is what the OP is asking about - Parsing a Float32 formatted as a string representing as a Float64.
There is a related discussion on github, and your issue seemingly still does not work out of the box.
You could use, e.g., package Parsers.jl which does the job.
julia> x = Parsers.parse(Float32, "1.23f-4")
0.000123f0