# Non-legal UTF-8 filenames (in e.g. TOML or JSON)

**URL:** <https://discourse.julialang.org/t/non-legal-utf-8-filenames-in-e-g-toml-or-json/126214>\
**Category:** Internals & Design\
**Created:** [February 23, 2025, 5:33pm UTC](https://discourse.julialang.org/t/non-legal-utf-8-filenames-in-e-g-toml-or-json/126214 "2025-02-23T17:33:27Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![Palli](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/palli/32/3380_2.png) [@Palli](https://discourse.julialang.org/u/Palli)\
**Post date:** [February 23, 2025, 5:33pm UTC](https://discourse.julialang.org/t/non-legal-utf-8-filenames-in-e-g-toml-or-json/126214/1 "2025-02-23T17:33:28Z")

</div>

> <https://github.com/ziglang/zig/issues/15552#issuecomment-1699950384>
>
> Now that ZON files (Zig Object Notation) are a thing, they should be documented …in langref
> ZON files described on #14523

> A major flaw with json is its inability to store non-UTF8 strings which inadvertently makes them unreliable for storing unix filenames (here a filename is defined as any sequence of bytes except `NUL` and `/`, including invalid UTF-8).  
> […]  
> Some prior art on this would be Rust’s [`raw string literal`](https://doc.rust-lang.org/reference/tokens.html#raw-string-literals) notation and `qsn` which is a format based on those ideas. It is documented quite nicely here, along with good general coverage of this topic: [http://www.oilshell.org/release/latest/doc/qsn.html](http://www.oilshell.org/release/latest/doc/qsn.html)

> ZON strings are no different from Zig string literals in this context, so the answer is: yes, ZON strings store any sequence of bytes, agnostic to their interpretation. The literal in file must itself be UTF-8 encoded, because Zig sources - and by extension ZON - are always UTF-8 encoded, but you can represent any byte sequence which is _not_ valid UTF-8 by using simple [escape sequences](https://ziglang.org/documentation/master/#Escape-Sequences) (the main relevant one here being `\xNN`).

```julia
shell> touch \xff

julia> touch("\xff") # The former didn't work, see:

```

-rw-rw-r-- 1 pharaldsson pharaldsson 0 feb 23 17:23 xff  
-rw-rw-r-- 1 pharaldsson pharaldsson 0 feb 23 17:24 ‘’$‘\377’

I can open this file, but lets say I want to serialize the filename itself, or if Pkg has to handle such (it it even a possibility of it happening, if generated from within Julia? I suppose you can make such a Package/module).

You can do `escape_string(filename)` and it’s seemingly always needed to be careful, unless your code/Julia API already7 does it. You can do it redundantly with no ill effect, only a bit of expansion, it’s just then you must unescape too twice, we the users need to know if needed or not.

Does it happen by default for [filenames] in JSON packages? Probably not, it would mean you need to know if potential illegal UTF-8 (always, for non-validated strings, and filenames can’t be validated). It seems to me Pkg (using TOML) doesn’t escape (or only in once place), since it doesn’t deal with filenames. It’s unclear it TOML does it for you (for other contexts). Python and C++ have good path/filesystems APIs, and there’s talk of similar in Julia, without just naked strings, and then likely should handle escaping for you. Strictly speaking the escaping is need for all Julia strings… unless when they are known to be validated UTF-8.

On Windows with UTF-16, are all even-numbered byte sequences also valid? And even odd, like the file above?

> <https://stackoverflow.com/questions/75107763/why-binary-mode-when-reading-writing-toml-in-python>

I was looking into ZON encoding and if we should possibly support it (it’s not just for Zig or its packages system):

> <https://github.com/ziglang/zig/pull/14523>
>
> Note: the actual diff is much smaller; I moved parse logic from one file to anot…her, but it is unmodified except for an additional function.
> 
> \## Description
> 
> This branch introduces Zig Object Notation which can be parsed using the standard library. The same function used to parse zig files is augmented to also parse zon files, using an enum to toggle between the two modes. I considered using a separate implementation to parse zon files, however, I made the decision to reuse existing logic for the following reasons:
> 
> \* Shared logic is nice for maintenance
> \* Shared logic keeps the binary size of the compiler smaller
> \* Reusing the logic for rendering the AST is nice. We get \`zig fmt\` for free.
> \* It also lets us reuse a bunch of error reporting logic
> 
> Next, this branch switches to \`build.zig.zon\` instead of \`build.zig.ini\` for the manifest file and enhances the relevant error reporting.
> 
> This is the last remaining planned breaking change to the package manager, other than these two:
> \* #14286
> \* #14307
> 
> These will not be quite as invasive breakages, so, after this branch is merged, it starts to become more feasible for people to jump in and start using this new feature of zig.
> 
> Closes #14290.
> 
> \## Follow-up work to be done:
> \* \[x\] #14530
> \* \[x\] #14531
> \* \[\] #14532
> \* \[\] #14534
> \* \[\] #5039

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [February 23, 2025, 7:20pm UTC](https://discourse.julialang.org/t/non-legal-utf-8-filenames-in-e-g-toml-or-json/126214/2 "2025-02-23T19:20:52Z")

</div>

> [@Palli](#):
>
> How do I serialize the filename, and get it escaped?

`show(io, string)` will write it in escaped form (with quotation marks) or `print(io, escape_string(string))` if you don’t want quotation marks.

> [@Palli](#):
>
> Does it happen by default for TOML, and JSON packages?

No, because `\x` escaping is not allowed in these formats as I understand it.

- JSON.jl just writes the raw non-UTF8 bytes to the string. It can parse these back, but JSON parsers in other languages may give an error. e.g. `JSON.parse(JSON.json(Dict("foo" => "\xff")))` works.
- TOML.jl [throws an error](https://github.com/JuliaLang/TOML.jl/blob/44aab3c023323587680ab8d8c7ef478bf78c4c0c/src/print.jl#L9-L11) if you try to output strings with non-UTF8 bytes.

Of course, you could always use a different serialization format. The simplest is to save it as an array of byte values by outputting `codeunits(str)` instead of `str` — `Dict("foo" => codeunits("\xff"))` works just fine with both JSON and TOML.

---

<div class="post-metadata">

**Author:** ![Palli](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/palli/32/3380_2.png) [@Palli](https://discourse.julialang.org/u/Palli)\
**Post date:** [February 23, 2025, 9:02pm UTC](https://discourse.julialang.org/t/non-legal-utf-8-filenames-in-e-g-toml-or-json/126214/3 "2025-02-23T21:02:12Z")

</div>

> [@stevengj](#):
>
> `\x` escaping is not allowed in these formats

I assumed, but you’re right, JSON (and TOML) doesn’t support arbitrary byte-strings, and thus _NOT_ all filenames, without help like QSN add-on:

[https://www.oilshell.org/release/0.9.9/doc/qsn.html](https://www.oilshell.org/release/0.9.9/doc/qsn.html)

> QSN (“quoted string notation”) is a data format for byte strings […]
> 
> It’s an adaptation of Rust’s string literal syntax with a few use cases:
> 
> - To print filenames to a terminal. Printing arbitrary bytes to a terminal is bad, so programs like coreutils already have [informal QSN-like formats](https://www.gnu.org/software/coreutils/quotes.html).
> - To exchange data between different programs, like JSON or UTF-8. Note that JSON can’t express arbitrary byte strings.
> - To solve the “framing problem” over pipes. QSN represents newlines like \n, so literal newlines can be used to delimit records. Oil uses QSN because it’s well-defined and parsable.
> 
> It’s both human- and machine-readable. Any programming language or tool that understands JSON should also understand QSN.

It’s a fucking annoying problem, either you output illegal UTF-8, by implicit agreement, violating the JSON standard, or you add QSNm that all don’t know of.

You want to support all possible filenames, at least in some situations, think implementing a backup system. You do not want to ignore those files or worse throw an error, and not back up any files… But even if you support on Unix with QSN, it’s unclear to me you could restore on Windows. I.e. is illegal UTF-8 mappable to some legal or illegal UTF-16, and then transportable back.

```julia
julia> length("\xff")
1

julia> length("\uff") # Not same as above
1

julia> sizeof("\uff")
2

julia> bitstring('\uff')
"11000011101111110000000000000000"

```

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [February 23, 2025, 9:08pm UTC](https://discourse.julialang.org/t/non-legal-utf-8-filenames-in-e-g-toml-or-json/126214/4 "2025-02-23T21:08:10Z")

</div>

> [@Palli](#):
>
> It’s a \*\*\*\*\*\* annoying problem, either you output illegal UTF-8, by implicit agreement, violating the JSON standard, or you add QSNm that all don’t know of.

If you have an arbitrary sequence of bytes, it’s arguably not a string at all, especially from the perspective of the format. You should just store it as a `Vector{UInt8}` via `codeunits(str)`.

---

<div class="post-metadata">

**Author:** ![Palli](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/palli/32/3380_2.png) [@Palli](https://discourse.julialang.org/u/Palli)\
**Post date:** [February 23, 2025, 9:13pm UTC](https://discourse.julialang.org/t/non-legal-utf-8-filenames-in-e-g-toml-or-json/126214/5 "2025-02-23T21:13:00Z")

</div>

> [@stevengj](#):
>
> If you have an arbitrary sequence of bytes, it’s arguably not a string at all, especially from the perspective of the format.

Can be argued yes, Unicode-wise, but is (still) currently in Julia a `String`.

> [@stevengj](#):
>
> You should just store it as a `Vector{UInt8}` via `codeunits(str)`.

That doesn’t help at all, or only makes it explicit you intent to store illegal too. It stores the same byte-string, and still unclear to users, you to store in JSON.

---

<div class="post-metadata">

**Author:** ![nhz2](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nhz2/32/44428_2.png) [@nhz2](https://discourse.julialang.org/u/nhz2)\
**Post date:** [February 23, 2025, 9:29pm UTC](https://discourse.julialang.org/t/non-legal-utf-8-filenames-in-e-g-toml-or-json/126214/6 "2025-02-23T21:29:01Z")

</div>

It is common to use base64 encoding if you want to store data in JSON.

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [February 23, 2025, 9:33pm UTC](https://discourse.julialang.org/t/non-legal-utf-8-filenames-in-e-g-toml-or-json/126214/7 "2025-02-23T21:33:00Z")

</div>

> [@Palli](#):
>
> That doesn’t help at all, or only makes it explicit you intent to store illegal too. It stores the same byte-string, and still unclear to users, you to store in JSON.

If you have `filename = [0x66, 0x6f, 0x6f, 0x2e, 0x74, 0x78, 0x74]` it’s pretty clear IMO, but in general a JSON file is not completely self-documenting — in general you need some documentation anyway to tell people what the fields mean.

And it solves the problem of storing an arbitary filename in a JSON file without violating the standard, and allowing any conforming implementation in any language to read it.

---

<div class="post-metadata">

**Author:** ![Palli](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/palli/32/3380_2.png) [@Palli](https://discourse.julialang.org/u/Palli)\
**Post date:** [February 23, 2025, 9:39pm UTC](https://discourse.julialang.org/t/non-legal-utf-8-filenames-in-e-g-toml-or-json/126214/8 "2025-02-23T21:39:20Z")

</div>

Right and very valid for contents of (some) files, but bloating, and bad for filenames, making them unreadable. Then QSN is better, and arguable for data too. “[Ascii85](https://en.wikipedia.org/wiki/Ascii85) (also called Base85)” is in most cases a better alternative for binary data, just Base64 more supported, including in Julia (would be easy to support both).

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [February 23, 2025, 9:43pm UTC](https://discourse.julialang.org/t/non-legal-utf-8-filenames-in-e-g-toml-or-json/126214/9 "2025-02-23T21:43:00Z")

</div>

If you’re worried about this, I would just give an error on `!isvalid(filename)` — at least in situations where you need to store the filename in a human-readable format like JSON — and tell users to rename their files. Non-Unicode-compatible filenames are likely to cause other problems (e.g. they won’t be portable across filesystems in general), and there’s no real need to support them in a typical application AFAICT (aside from low-level OS utilities).

---

<div class="post-metadata">

**Author:** ![Palli](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/palli/32/3380_2.png) [@Palli](https://discourse.julialang.org/u/Palli)\
**Post date:** [February 23, 2025, 9:50pm UTC](https://discourse.julialang.org/t/non-legal-utf-8-filenames-in-e-g-toml-or-json/126214/10 "2025-02-23T21:50:28Z")

</div>

If you construct a filename, with a (future) type for it, then it might be valid to reject non-legal UTF-8 strings (and even QSN-encoding such, by default). But if you e.g. read files from your directly, we must support all files there, including non-UTF-8 (at least I feel like, by default, as I explained with my backup example).

I marked my comment with a solution, for QSN (seems least bad, no solution seems prefect), that I propose Julia would adopt, when you cast to a String.

It can be input into this proposal:

> [@Designing a Paths Julep](https://discourse.julialang.org/t/designing-a-paths-julep/124335):
>
> Prompted by [a gripe I made on Slack](https://julialang.slack.com/archives/C67TK21LJ/p1732097833044289), thoughts I’ve had on paths have coalesed into the start of a Julep and a prototpye. At this stage, I’d love to get wider input on the idea and help shoring it up into a solid proposal for Julia (a Julep). That HackMD document is commentable and editable to anyone logged in, and there’s a basic (and likely lightly buggy) prototype that can be played with. It would be great to have a little working group of sorts to take this to a finished proposal. @jakobn…

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [February 23, 2025, 10:00pm UTC](https://discourse.julialang.org/t/non-legal-utf-8-filenames-in-e-g-toml-or-json/126214/11 "2025-02-23T22:00:25Z")

</div>

> [@Palli](#):
>
> But if you e.g. read files from your directly, we must support all files there, including non-UTF-8 (at least I feel like, by default, as I explained with my backup example).

You’re not just reading filenames. You’re storing them in a configuration file. You get to choose the format of your configuration file. You can:

- Tell users that your configuration file doesn’t support non-Unicode filenames.
- Store filenames in JSON or TOML as byte arrays or some other custom encoding.
- Use a file format (not JSON or TOML) that allows strings that contain arbitrary bytes. Or use non-standard JSON (don’t worry about parsing it from anything but JSON.jl).

> [@Palli](#):
>
> I marked my comment with a solution, for QSN (seems least bad, no solution seems prefect), that I propose Julia would adopt, when you cast to a String.

I don’t know what change in Julia you are proposing — Julia `String` already supports arbitrary byte streams, and Julia’s filesystem APIs work just fine with non-UTF8 filenames too. Your problem is with the JSON and TOML file formats, which are not part of Julia and are not controlled by Julia, and for those you have the three options above.