How do I implement recursive interpolation ($) for string macros?

General Usage
, , ,
1

Could Meta.parse grow an option to re-use $ interpolation semantics within a string macro?

I’m trying to replicate functionality of Mike Bostock’s Hyper Text Literal only ported to Julia (HypertextLiteral.jl). Specifically, our goal is to permit any sort of Julia expression, but where our string macro provides context-sensitive escaping. As an example of what we mean…

julia> htl"""<ul>$([htl"<li>$x</li>" for x in ["a", "c&d"]])</ul>"""
HTML{String}("<ul><li>a</li><li>c&amp;d</li></ul>")

We can nest here because the outer string macro uses triple quotes, letting the inner string macro provide customized interpretation for $. The problem is, this only works one level. While normal string interpolation works any number of levels, it doesn’t work with string macros. Let’s explain. In the trivial case of recursion, even with a regular string macro syntax, there is an immediate syntax error before the macro is even invoked.

julia> macro my_str(expr::String) expr end
@my_str (macro with 1 method)

julia> my"one$("two")"
ERROR: syntax: cannot juxtapose string literal

We can get closer by using triple quotes, at least the call gets to our macro.

julia> my"""one$("two")"""
"one\$(\"two\")"

If you use Meta.parse for substrings marked with $ this buys you one level of nesting, however, parse still exhibits the same limitation as above. We’ve verified this with our draft implementation. This code searches to find occurrences of $ and then uses Meta.parse to convert this sub-expression. In particular, the opening example presented actually works.

There are work-rounds to this lack of nesting. We can use functions that return fragments. We can use let construct. We could also implement hypertext literal as a @htl() macro which won’t cause this issue with the grammar. Am I missing any other approach, saving clever parsing and translation that tries to emulate Julia but is otherwise brittle?

Even so. It’d be nice to have recursive interpolation for string macros. I think it’s acceptable that the outermost call would have to be triple string. However, it’d be great if subsequent usage of the string macro with single quotes could have interpolation conversion. What if Meta.parse added an option to permit recursion in this case? Anyone could then let custom macros re-use the built-in recursive interpolation parse logic.

2

How about use something other than $ for hypertext interpolation and leave $ alone? You could even pun with ¢. I realize this is a different thing than the original language, but it sure sidesteps a lot of problems. This does mean you have to decide how to parse nested versions of ¢ yourself, maybe an advantage maybe not.

Edit:
This looks like a nice tool, I look forward to using this however you end up getting it to work!

3

@contradict – for string macros, the $ character is not treated with special behavior (it’s not about picking another character). For this particular case, I wish string macros also implemented interpolation, identical to regular strings; however, there are other string macro use cases (such as regex) where automatic interpolation is inappropriate. Hence the current behavior seems reasoned, it’s just unfortunate that interpolation works only for regular strings and not for user-defined string types via macros. I’m simply curious if there is any way that this parser behavior could be made to work, in a future iteration of Julia.

4

Is it fair to summarize your issue by these examples?

julia> "one$("two")" # works
"onetwo"

julia> macro my_str(x); x; end
@my_str (macro with 1 method)

julia> my"one$("two")" # doesn't work
ERROR: syntax: cannot juxtapose string literal
Stacktrace:
 [1] top-level scope
   @ none:1

That’s a fair complaint particularly because you want to mimic Julia code inside the string.

The tricky thing is that apparently you want the parser to recognize this for parsing, but then not assign any semantic meaning to it: it should still pass through the raw string so you can parse it. Then, where htl syntax dictates it, you parse it as julia code.

There is all kinds of corner cases for syntax-within-a-syntax like this. For example, does every $ in the string represent Julia interpolation? Or are there places (e.g. embedded javascript using jquery) where it doesn’t? If so, should the user escape it? (And if you need escaping anyway, why not make escaping the solution to your original problem?) What if you call another string macro inside the interpolation? What if that string macro also defines its own string parsing?

If you go down this road, I think you’ll soon end up wanting to make Julia’s top-level parser pluggable to support this “properly”. Perl actually has that feature; you can define tokens after which the parser gives control of the source code bytes stream to a user callback(!). It’s hard to see how to fit that into Julia, but I think it’s where your problem statement inevitably leads.

FWIW I’m not a fan of the idea that Meta.parse should support dialects of Julia syntax with keyword arguments. It adds mental overhead for everyone who needs to deal with Julia syntax for very specific benefits in corner cases.

So I’m afraid that there’s no better answer than just dealing with the various ways we handle nested quotes already: """ and \, and nested escaping if necessary.

Off-topic: you might be interested in (my) HAML.jl which is doing some of the same $-shenanigans here and some context-aware escaping here. The @haml_str macro has exactly the same issue so please believe me when I say I 100% sympathize.

5

Thank you for your thoughts and response, @tkluck

Actually no – I wish to keep Julia’s exact parsing semantics with regard to $. Hence, there are no corner cases because it’s not a syntax within a syntax. Instead, I wish to assign to the parsed expression tree different semantics. Indeed, I believe that mechanism could be (mostly) implemented using a function macro, @htl(...) with two known downsides: (a) it’d be more characters, the canonical case would be @htl("...") rather than just htl"...", and (b) the abstract syntax tree provided to macros doesn’t provide $ as a recognizable token (see #38501).

The deeper problem with my request is that there are other string macro use cases, such as regular expressions, that really don’t want the Julia syntax or semantics. Hence, the string macros don’t include interpolation feature (which can be hugely valuable). It’s a reasonable choice.

Right. I think you’re correct that it’s the wrong place. That said @htl("...") is much nosier than htl"...", especially when frequently used in a recursive manner as you would be when making an HTML template. Thanks.

This is exactly that “syntax within a syntax” approach; conversely, keeping Julia syntax and expression semantics is important. Ideally, one would be able to take any Julia expression and simply prefix relevant strings with htl to get context sensitive escaping.

Yea. I think it’s about having helpful documentation that explains common cases, outlines how nested interpolation won’t work with this syntax, etc. Once #38501 is addressed, then a macro function equivalent could be added as an alternative.