# Efficiency of parsing ASCII vs. Unicode
**URL:** https://discourse.julialang.org/t/efficiency-of-parsing-ascii-vs-unicode/108727
**Category:** Performance
**Tags:** unicode
**Created:** [January 12, 2024, 1:15pm UTC](https://discourse.julialang.org/t/efficiency-of-parsing-ascii-vs-unicode/108727 "2024-01-12T13:15:55Z")
**Posts on this page:** 11
**Page:** 1
### Author:  [@greatpet](https://discourse.julialang.org/u/greatpet)
#### Post date: [January 12, 2024, 1:15pm UTC](https://discourse.julialang.org/t/efficiency-of-parsing-ascii-vs-unicode/108727/1 "2024-01-12T13:15:55Z")
What about performance implications for parsing ASCII text files? Consider the following function which checks if a file has an equal number of left brackets and right brackets:
```julia
function file_has_balanced_brackets(file)
s = read(file, String)
counter = 0
for c in s
if c == '('
counter += 1
elseif c == ')'
counter -= 1
end
end
return (counter == 0)
end
```
Since characters in Julia have 4-byte sizes, rather than 1 byte for ASCII, is there a performance penalty for the code above?
---
### Author:  [@sijo](https://discourse.julialang.org/u/sijo)
#### Post date: [January 12, 2024, 1:41pm UTC](https://discourse.julialang.org/t/efficiency-of-parsing-ascii-vs-unicode/108727/2 "2024-01-12T13:41:07Z")
> [@greatpet](#):
>
> Since characters in Julia have 4-byte sizes, rather than 1 byte for ASCII, is there a performance penalty for the code above?
ASCII characters are only 1 byte in UTF-8.
---
### Author:  [@greatpet](https://discourse.julialang.org/u/greatpet)
#### Post date: [January 12, 2024, 1:50pm UTC](https://discourse.julialang.org/t/efficiency-of-parsing-ascii-vs-unicode/108727/3 "2024-01-12T13:50:40Z")
> [@sijo](#):
>
> ASCII characters are only 1 byte in UTF-8.
See this:
```julia
julia> sizeof('a')
4
julia> sizeof(UInt8('a'))
1
```
That’s why I’m wondering if I’m losing performance by using `Char` literals like `'a'`, rather than e.g. the more verbose `UInt8('a')` when parsing ASCII files.
---
### Author:  [@sijo](https://discourse.julialang.org/u/sijo)
#### Post date: [January 12, 2024, 1:54pm UTC](https://discourse.julialang.org/t/efficiency-of-parsing-ascii-vs-unicode/108727/4 "2024-01-12T13:54:42Z")
Ah sorry I misunderstood. I don’t think working with 32-bit values is typically slower than 8-bit on common architectures but certainly in some contexts it must be less efficient… But the penalty will probably be with the string type (UTF-8) rather than the character type, for example `length(str)` is inefficient. In Julia if that’s a problem and you know you don’t need UTF-8, you can easily use another string type from LegacyStrings.jl.
---
### Author:  [@stevengj](https://discourse.julialang.org/u/stevengj)
#### Post date: [January 12, 2024, 1:58pm UTC](https://discourse.julialang.org/t/efficiency-of-parsing-ascii-vs-unicode/108727/5 "2024-01-12T13:58:25Z")
> [@greatpet](#):
>
> That’s why I’m wondering if I’m losing performance by using `Char` literals like `'a'`, rather than e.g. the more verbose `UInt8('a')` when parsing ASCII files.
High-performance parsers like CSV.jl often work with bytes (“code units” of strings) rather than decoding into Unicode characters, and similarly high-performance string processing often works with [`codeunits(str)`](https://docs.julialang.org/en/v1/base/strings/#Base.codeunits). Functions like `findnext` also have special optimizations when searching for a `Char` in a `String` that convert it to a raw byte first. The UTF-8 representation allows you to work with bytes efficiently for searching/matching operations.
---
### Author:  [@Mason](https://discourse.julialang.org/u/Mason)
#### Post date: [January 12, 2024, 2:09pm UTC](https://discourse.julialang.org/t/efficiency-of-parsing-ascii-vs-unicode/108727/6 "2024-01-12T14:09:52Z")
Just to expand on @stevengj’s excellent answer @greatpet, there’s a difference between characters on their own, and characters in a string. So even though `'a'` takes up 8 bytes, the string `"abcdefgh"` also only stores `8` bytes because of the encoding strategy that’s standard for working with unicode text.
That is, an ascii character only takes up one byte when it is a part of a string.
Here’s a demo:
```julia
julia> sizeof(['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h'])
32
```
vs
```julia
julia> sizeof("abcdefgh")
8
```
Now it is however true that working with strings statically known to be pure ASCII can still have performance benefits because there’s less branching and indirection in how it’s dealt with, but it’s memory footprint is actually not one of the differences.
---
### Author:  [@stevengj](https://discourse.julialang.org/u/stevengj)
#### Post date: [January 12, 2024, 2:16pm UTC](https://discourse.julialang.org/t/efficiency-of-parsing-ascii-vs-unicode/108727/7 "2024-01-12T14:16:04Z")
> [@Mason](#):
>
> Now it is however that working with strings statically known to be pure ASCII can still have performance benefits
For many operations, it doesn’t matter whether they are pure ASCII. Suppose that you have the string `s = "🐻 + (🐨 + 🐼)"` and you want to find the character `'('` = U+0028. Then you can search the bytes, given by `codeunits(s)`, for the byte `0x28`:
```julia
julia> codeunits(s)
20-element Base.CodeUnits{UInt8, String}:
0xf0
0x9f
0x90
0xbb
0x20
0x2b
0x20
0x28
0xf0
0x9f
0x90
0xa8
0x20
0x2b
0x20
0xf0
0x9f
0x90
0xbc
0x29
julia> findfirst(==(0x28), codeunits(s))
8
```
In fact, this is what `findfirst(==('('), s)` does internally.
The point is that when you are parsing files, you are mostly looking for either ASCII characters like `','` or `'\n'`, in which case you can search bytes, or for substrings, in which case you can also look for substrings of bytes. You rarely need to decode things into Unicode characters for many common file formats.
---
### Author:  [@Mason](https://discourse.julialang.org/u/Mason)
#### Post date: [January 12, 2024, 2:20pm UTC](https://discourse.julialang.org/t/efficiency-of-parsing-ascii-vs-unicode/108727/8 "2024-01-12T14:20:37Z")
Yes, of course, but there’s still circumstances where parsing pure ascii strings can be faster than parsing full on UTF-8 strings from what I understand.
But yes, I agree that typically there is not a significant different, especially not one we should care about when talking about the parsing performance of julia.
From what I understand, the difference between pure ASCII and general unicode strings ends up being more of a problem for quasi-numerical things like when bioinformatics people use them to encode DNA sequences (but of course in that circumstance you’re typically better off using a more specialized encoding than ASCII)
---
### Author:  [@greatpet](https://discourse.julialang.org/u/greatpet)
#### Post date: [January 12, 2024, 4:33pm UTC](https://discourse.julialang.org/t/efficiency-of-parsing-ascii-vs-unicode/108727/9 "2024-01-12T16:33:30Z")
> [@stevengj](#):
>
> ```julia
> julia> findfirst(==(0x28), codeunits(s))
> 8
>
> ```
As I don’t remember the ASCII coding by heart, I wish “Char macros” existed in a way similar to string macros, so I could write this instead:
```julia
findfirst(==(uint8'('), codeunits(s))
```
Currently I would write
```julia
findfirst(==(UInt8('(')), codeunits(s))
```
knowing that the compiler will carry out the trivial constant folding. This unfortunately make the code a little more verbose, as the `UInt8` conversion appears everywhere in my source file.
Another minor inconvenience is that the `Base` functions for classifying characters, such as `isspace` and `isletter`, only accept `Char` but not `UInt8` arguments, so I have to cook up my own versions like this one:
```julia
function isspace_uint8(c::UInt8)
c in (UInt8('\t'), UInt8('\n'), UInt8('\v'), UInt8('\f'), UInt8('\r'), UInt8(' '))
end
```
Alternatively, I could define
```julia
isspace_uint8(c::UInt8) = isspace(Char(c))
```
but this doesn’t seem ideal for performance.
---
### Author:  [@stevengj](https://discourse.julialang.org/u/stevengj)
#### Post date: [January 12, 2024, 4:39pm UTC](https://discourse.julialang.org/t/efficiency-of-parsing-ascii-vs-unicode/108727/10 "2024-01-12T16:39:02Z")
> [@greatpet](#):
>
> As I don’t remember the ASCII coding by heart, I wish “Char macros” existed in a way similar to string macros, so I could write this instead:
Yes, that has been raised before. Unfortunately it’s not possible with `'` syntax because of the way `x'y'` bar is parsed as `x' * y'`. This was discussed in [custom character literal macros? · Issue #26305 · JuliaLang/julia · GitHub](https://github.com/JuliaLang/julia/issues/26305)
However, it’s perfectly possible to define a string macro that returns a `UInt8`. For example JuliaSyntax.jl defines a [`macro u8_str`](https://github.com/JuliaLang/JuliaSyntax.jl/blob/a6f2d1580f7bbad11822033e8c83e607aa31f100/src/utils.jl#L53-L58) do precisely this: `u8"x"` returns `UInt8('x')`.
> [@greatpet](#):
>
> Another minor inconvenience is that the `Base` functions for classifying characters, such as `isspace` and `isletter`, only accept `Char` but not `UInt8` arguments, so I have to cook up my own versions like this one:
One possibility that has come up from time to time is having a new `ASCIIChar` subtype of `AbstractChar` that allows you to introduce new methods that dispatch on this.
---
### Author:  [@DNF](https://discourse.julialang.org/u/DNF)
#### Post date: [January 12, 2024, 5:42pm UTC](https://discourse.julialang.org/t/efficiency-of-parsing-ascii-vs-unicode/108727/11 "2024-01-12T17:42:51Z")
> [@greatpet](#):
>
> Since characters in Julia have 4-byte sizes, rather than 1 byte for ASCII, is there a performance penalty for the code above?
This sounds like an argument for short variable names😉
In this context, at least, `α` beats `alpha`.