How is a non-standard literal different from calling its corresponding macro?

Internals & Design
,
1 
julia> x = "whoops"
"whoops"

julia> @raw_str("$x") # huh...
ERROR: UndefVarError: x not defined
Stacktrace:
 [1] top-level scope
   @ REPL[48]:1

julia> "$x", raw"$x" # expected
("whoops", "\$x")

julia> @macroexpand @raw_str("$x") # interpolates from definition scope, hence error
:("$(Base.x)")

julia> @macroexpand "$x" # interpolates from call scope
:("$(x)")

julia> @macroexpand raw"$x" # literal appears to evaluate first
"\$x"

The example shows the discrepancy. The non-standard literal expression even evaluates despite @macroexpand usually preventing that. I want to know what’s going on because this seems important for custom parsing and perhaps interpolation.

It doesn’t seem like @raw_str("$x") is simply doing "$x" first then applying raw_str. If that were the case, I would expect @raw_str("$x") to do the same thing as "$x" because macro raw_str(s); s; end actually does (closest to) nothing to the text.

2 Likes
2

I think you’re just confused about how “$x” is displayed / represented, raw_str macro can’t be simpler:

julia> macro my_str(x)
           x
       end
@my_str (macro with 1 method)

julia> my"$x"
"\$x"

“$x” with interpolation is similar to “a\nb”, this is just how you write things. If you need a raw “$” in your string, you need to either write “\$x”, or, use raw_str . Similar to if you need a literal backslash in your string, you need to write:

julia> print("\\")
\

in other words, the point of @raw_str is just so you don’t have to type \ all the time.

julia> raw"a\b"
"a\\b"
3

I’ve also stumbled upon this problem. Indeed, non-standard string literals parse differently. You can verify this by quoting the ns-string and the macro call:

julia> :(raw"good ol' string") == :(@raw_str "good ol' string")
true

julia> :(raw"$x") == :(@raw_str "$x")
false

EDIT:

The non-standard literal expression even evaluates despite @macroexpand usually preventing that.

It’s more complicated than that. In Julia, literals cannot be quoted. This means that you can, for example, write macros that only take numeric literals:

julia> macro is_num_literal(n::Number) true end;

julia> macro is_num_literal(n) false end;

julia> @is_num_literal 1
true

julia> x = 1;

julia> @is_num_literal x
false

But, strings are weird. Strings with interpolations can be quoted, and the result is an Expr with a string head!

julia> :"good ol' string" |> dump
String "good ol' string"

julia> :"$x" |> dump
Expr
  head: Symbol string
  args: Array{Any}((1,))
    1: Symbol x

All this to say that @macroexpand really didn’t evaluate anything, but since ns-strings cancel interpolations, then the expansion returns a literal string (not a string Expr).

1 Like
4
Trying `dump` like you did, 
julia> dump(:( raw"$x" ))
Expr
  head: Symbol macrocall
  args: Array{Any}((3,))
    1: Symbol @raw_str
    2: LineNumberNode
      line: Int64 1
      file: Symbol REPL[4]
    3: String "\$x"

julia> dump(:( @raw_str "$x" ))
Expr
  head: Symbol macrocall
  args: Array{Any}((3,))
    1: Symbol @raw_str
    2: LineNumberNode
      line: Int64 1
      file: Symbol REPL[5]
    3: Expr
      head: Symbol string
      args: Array{Any}((1,))
        1: Symbol x

it looks like :(@raw_str "$x") contains :"$x" except macro hygiene rules look for the x in the definition scope instead of the call scope. On the other hand, :( raw"$x" ) is equivalent to :(@raw_str "\$x"). You can interpolate the raw string without manual escapes :(@raw_str $(raw"$x")). You get an equivalent result without interpolation :(@raw_str raw"$x") because the parser acts on literals before macros expand, but there will be two @raw_str macro calls. Before macros expand, the parser makes $ do interpolation for standard string literals (there’s no $ character remaining in the expression) and replaces '$' with the equivalent '\$' for non-standard string literals. Given a name like blah"$x", it’ll make up a corresponding @blah_str symbol, even if the macro doesn’t exist.

I can’t control parsing at all, but at least I’ll know to expect that $ does not do interpolation in non-standard string literals, leaving whatever it does up to the macro. I’ll leave this open because I don’t know what other decisions that the parser makes before macro expansion, depending on whether a string literal is standard or non-standard.

1 Like
5

I think the rules are relatively straightforward:

"something" is a string literal

"something_with_$interpolation" is syntax sugar for string("something_with_", interpolation)

@some_macro "something_with_$interpolation" is syntax sugar for @some_macro string("something_with_", interpolation)

prefix"something_with_$interpolation" is syntax sugar for @prefix_str "something_with_\$interpolation", so all $s are escaped

@prefix_str "something_with_$interpolation" does not get special parser treatment so like example 3 it is syntax sugar for @prefix_str string("something_with_", interpolation). One can argue whether the parser should give @prefix_str macros in their macro form special treatment as well, but it doesn’t. I guess that makes sense as you can reassign names, so you could call the same macro differently depending on whether you give it a name with _str at the end or not.

2 Likes
6

Not true, the string call enters the macro as an expression, it’s not yet evaluated to make 1 input string. I don’t know if the parser does a string call or something equivalent to it.

julia> macro headof(ex) Expr(:quote, ex.head) end
@headof (macro with 1 method)

julia> @headof "something_with_$interpolation"
:string

julia> @headof string("something_with_", interpolation)
:call

The parser does not do a string call for $-interpolated standard literals, I type pirated the primary callee method @eval Base function print_to_string(xs...) to insert 1 space after every argument. While interpolated standard literals do get those extra spaces (and messes up REPL text display), the spaces are not in the expression, so it seems to happen later in eval.

1 Like
7

Eh yeah sorry I was thinking about the Expr(:string not the string function call. But they seem to be equivalent. Expr(:string probably just exists so that macros know that the user has written the interpolation syntax.

julia> @generated function test(v::Val)
           if v === Val{true}
               quote
                   x = 1
                   $(Expr(:string, "a string and a ", :x))
               end
           elseif v === Val{false}
               quote
                   x = 1
                   "a string and a $x"
               end
           end
       end
test (generic function with 1 method)

julia> @code_warntype test(Val(true))
MethodInstance for test(::Val{true})
  from test(v::Val) @ Main REPL[17]:1
Arguments
  #self#::Core.Const(test)
  v::Core.Const(Val{true}())
Locals
  x::Int64
Body::String
1 ─      (x = 1)
│   %2 = Base.string("a string and a ", x::Core.Const(1))::String
└──      return %2


julia> @code_warntype test(Val(false))
MethodInstance for test(::Val{false})
  from test(v::Val) @ Main REPL[17]:1
Arguments
  #self#::Core.Const(test)
  v::Core.Const(Val{false}())
Locals
  x::Int64
Body::String
1 ─      (x = 1)
│   %2 = Base.string("a string and a ", x::Core.Const(1))::String
└──      return %2
1 Like
8

From my understanding, which is mostly based on this JuliaSyntax file, the only things that can be escaped in non-standard strings are the string delimiters themselves: \" (if string), \` (if cmd). Everything else is treated as a regular character.

1 Like
9

I’m still not fully understanding how the raw string flags are determined in the new parser, but it’s a lot more readable, showing the divide between standard string and raw string (really everything else) and how parsing works there. Also found this old stackoverflow post “How does a non-standard string literal avoid a syntax error generated by a standard string literal?” about the current/old parser showing the same divide. I’ll close this thread, thanks for chipping in, everybody.

1 Like