# How do I implement recursive interpolation ($) for string macros?

**URL:** <https://discourse.julialang.org/t/how-do-i-implement-recursive-interpolation-for-string-macros/50444>\
**Category:** General Usage\
**Tags:** question, macros, recursion, string-interpolation\
**Created:** [November 19, 2020, 3:11pm UTC](https://discourse.julialang.org/t/how-do-i-implement-recursive-interpolation-for-string-macros/50444 "2020-11-19T15:11:54Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![cce](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cce/32/460_2.png) [@cce](https://discourse.julialang.org/u/cce)\
**Post date:** [November 19, 2020, 3:11pm UTC](https://discourse.julialang.org/t/how-do-i-implement-recursive-interpolation-for-string-macros/50444/1 "2020-11-19T15:11:54Z")

</div>

_Could `Meta.parse` grow an option to re-use `$` interpolation semantics within a string macro?_

I’m trying to replicate functionality of Mike Bostock’s [Hyper Text Literal](https://github.com/observablehq/htl) only ported to Julia ([HypertextLiteral.jl](https://github.com/clarkevans/HypertextLiteral.jl)). Specifically, our goal is to permit any sort of Julia expression, but where our string macro provides context-sensitive escaping. As an example of what we mean…

```julia
julia> htl"""<ul>$([htl"<li>$x</li>" for x in ["a", "c&d"]])</ul>"""
HTML{String}("<ul><li>a</li><li>c&amp;d</li></ul>")

```

We can nest here because the outer string macro uses triple quotes, letting the inner string macro provide customized interpretation for `$`. The problem is, this only works one level. While normal string interpolation works any number of levels, it doesn’t work with string macros. Let’s explain. In the trivial case of recursion, even with a regular string macro syntax, there is an immediate syntax error before the macro is even invoked.

```julia
julia> macro my_str(expr::String) expr end
@my_str (macro with 1 method)

julia> my"one$("two")"
ERROR: syntax: cannot juxtapose string literal

```

We can get closer by using triple quotes, at least the call gets to our macro.

```julia
julia> my"""one$("two")"""
"one\$(\"two\")"

```

If you use `Meta.parse` for substrings marked with `$` this buys you one level of nesting, however, `parse` still exhibits the same limitation as above. We’ve verified this with our [draft implementation](https://github.com/clarkevans/HypertextLiteral.jl/blob/9dd771eaf9f59d58fc2efbc148c075390ec53975/src/HypertextLiteral.jl). This code searches to find occurrences of `$` and then uses `Meta.parse` to convert this sub-expression. In particular, the opening example presented actually works.

There are work-rounds to this lack of nesting. We can use functions that return fragments. We can use `let` construct. We could also implement hypertext literal as a `@htl()` macro which won’t cause this issue with the grammar. Am I missing any other approach, saving clever parsing and translation that tries to emulate Julia but is otherwise brittle?

Even so. It’d be nice to have recursive interpolation for string macros. I think it’s acceptable that the outermost call would have to be triple string. However, it’d be great if subsequent usage of the string macro with single quotes could have interpolation conversion. What if `Meta.parse` added an option to permit recursion in this case? Anyone could then let custom macros re-use the built-in recursive interpolation parse logic.

---

<div class="post-metadata">

**Author:** ![contradict](https://avatars.discourse-cdn.com/v4/letter/c/ac91a4/32.png) [@contradict](https://discourse.julialang.org/u/contradict)\
**Post date:** [November 19, 2020, 4:19pm UTC](https://discourse.julialang.org/t/how-do-i-implement-recursive-interpolation-for-string-macros/50444/2 "2020-11-19T16:19:59Z")

</div>

How about use something other than `$` for hypertext interpolation and leave `$` alone? You could even pun with ¢. I realize this is a different thing than the original language, but it sure sidesteps a lot of problems. This does mean you have to decide how to parse nested versions of ¢ yourself, maybe an advantage maybe not.

Edit:  
This looks like a nice tool, I look forward to using this however you end up getting it to work!

---

<div class="post-metadata">

**Author:** ![cce](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cce/32/460_2.png) [@cce](https://discourse.julialang.org/u/cce)\
**Post date:** [November 19, 2020, 8:03pm UTC](https://discourse.julialang.org/t/how-do-i-implement-recursive-interpolation-for-string-macros/50444/3 "2020-11-19T20:03:20Z")

</div>

@contradict – for string macros, the `$` character is not treated with special behavior (it’s not about picking another character). For this particular case, I wish string macros also implemented interpolation, identical to regular strings; however, there are other string macro use cases (such as regex) where automatic interpolation is inappropriate. Hence the current behavior seems reasoned, it’s just unfortunate that interpolation works only for regular strings and not for user-defined string types via macros. I’m simply curious if there is any way that this parser behavior could be made to work, in a future iteration of Julia.

---

<div class="post-metadata">

**Author:** ![tkluck](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkluck/32/15769_2.png) [@tkluck](https://discourse.julialang.org/u/tkluck)\
**Post date:** [November 20, 2020, 1:04am UTC](https://discourse.julialang.org/t/how-do-i-implement-recursive-interpolation-for-string-macros/50444/4 "2020-11-20T01:04:44Z")

</div>

Is it fair to summarize your issue by these examples?

```julia
julia> "one$("two")" # works
"onetwo"

julia> macro my_str(x); x; end
@my_str (macro with 1 method)

julia> my"one$("two")" # doesn't work
ERROR: syntax: cannot juxtapose string literal
Stacktrace:
 [1] top-level scope
   @ none:1

```

That’s a fair complaint particularly because you want to mimic Julia code inside the string.

The tricky thing is that apparently you want the parser to _recognize_ this for parsing, but then _not_ assign any semantic meaning to it: it should still pass through the raw string so you can parse it. Then, where htl syntax dictates it, you parse it as julia code.

There is all kinds of corner cases for syntax-within-a-syntax like this. For example, does _every_ `$` in the string represent Julia interpolation? Or are there places (e.g. embedded javascript using jquery) where it doesn’t? If so, should the user escape it? (And if you need escaping anyway, why not make escaping the solution to your original problem?) What if you call _another_ string macro inside the interpolation? What if that string macro also defines its own string parsing?

If you go down this road, I think you’ll soon end up wanting to make Julia’s top-level parser pluggable to support this “properly”. [Perl actually has that feature](https://perldoc.perl.org/perlapi#PL_keyword_plugin); you can define tokens after which the parser gives control of the source code bytes stream to a user callback(!). It’s hard to see how to fit that into Julia, but I think it’s where your problem statement inevitably leads.

FWIW I’m not a fan of the idea that `Meta.parse` should support dialects of Julia syntax with keyword arguments. It adds mental overhead for everyone who needs to deal with Julia syntax for very specific benefits in corner cases.

So I’m afraid that there’s no better answer than just dealing with the various ways we handle nested quotes already: `"""` and `\`, and nested escaping if necessary.

Off-topic: you might be interested in (my) [`HAML.jl`](https://github.com/tkluck/HAML.jl) which [is doing some of the same `$`-shenanigans here](https://github.com/tkluck/HAML.jl/blob/master/src/SourceTools.jl#L153) and [some context-aware escaping here](https://github.com/tkluck/HAML.jl/blob/master/src/Escaping.jl). The `@haml_str` macro has exactly the same issue so please believe me when I say I 100% sympathize.

---

<div class="post-metadata">

**Author:** ![cce](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cce/32/460_2.png) [@cce](https://discourse.julialang.org/u/cce)\
**Post date:** [November 20, 2020, 2:57am UTC](https://discourse.julialang.org/t/how-do-i-implement-recursive-interpolation-for-string-macros/50444/5 "2020-11-20T02:57:49Z")

</div>

Thank you for your thoughts and response, @tkluck

> [@tkluck](#):
>
> The tricky thing is that apparently you want the parser to _recognize_ this for parsing, but then _not_ assign any semantic meaning to it: it should still pass through the raw string so you can parse it. Then, where htl syntax dictates it, you parse it as julia code.

Actually no – I wish to keep Julia’s exact parsing semantics with regard to `$`. Hence, there are no corner cases because it’s not a syntax within a syntax. Instead, I wish to assign to the parsed expression tree different semantics. Indeed, I believe that mechanism could be (mostly) implemented using a function macro, `@htl(...)` with two known downsides: (a) it’d be more characters, the canonical case would be `@htl("...")` rather than just `htl"..."`, and (b) the abstract syntax tree provided to macros doesn’t provide `$` as a recognizable token (see [#38501](https://github.com/JuliaLang/julia/issues/38501)).

The deeper problem with my request is that there are other string macro use cases, such as regular expressions, that really don’t want the Julia syntax or semantics. Hence, the string macros don’t include interpolation feature (which can be hugely valuable). It’s a reasonable choice.

> [@tkluck](#):
>
> FWIW I’m not a fan of the idea that `Meta.parse` should support dialects of Julia syntax with keyword arguments. It adds mental overhead for everyone who needs to deal with Julia syntax for very specific benefits in corner cases.

Right. I think you’re correct that it’s the wrong place. That said `@htl("...")` is much nosier than `htl"..."`, especially when frequently used in a recursive manner as you would be when making an HTML template. Thanks.

> [@tkluck](#):
>
> Off-topic: you might be interested in (my) [`HAML.jl`](https://github.com/tkluck/HAML.jl) which [is doing some of the same `$` -shenanigans here](https://github.com/tkluck/HAML.jl/blob/master/src/SourceTools.jl#L153) and [some context-aware escaping here](https://github.com/tkluck/HAML.jl/blob/master/src/Escaping.jl). The `@haml_str` macro has exactly the same issue so please believe me when I say I 100% sympathize.

This is exactly that “syntax within a syntax” approach; conversely, keeping Julia syntax and expression semantics is important. Ideally, one would be able to take any Julia expression and simply prefix relevant strings with `htl` to get context sensitive escaping.

> [@tkluck](#):
>
> So I’m afraid that there’s no better answer than just dealing with the various ways we handle nested quotes already: `"""` and `\` , and nested escaping if necessary.

Yea. I think it’s about having helpful documentation that explains common cases, outlines how nested interpolation won’t work with this syntax, etc. Once [#38501](https://github.com/JuliaLang/julia/issues/38501) is addressed, then a macro function equivalent could be added as an alternative.