# Flaw in Regex support for String
**URL:** https://discourse.julialang.org/t/flaw-in-regex-support-for-string/9667
**Category:** Internals & Design
**Tags:** strings, regex
**Created:** [March 12, 2018, 7:24pm UTC](https://discourse.julialang.org/t/flaw-in-regex-support-for-string/9667 "2018-03-12T19:24:44Z")
**Posts on this page:** 19
**Page:** 2
### Author:  [@ScottPJones](https://discourse.julialang.org/u/ScottPJones)
#### Post date: [April 8, 2018, 9:55pm UTC](https://discourse.julialang.org/t/flaw-in-regex-support-for-string/9667/21 "2018-04-08T21:55:33Z")
No. It’s a much more serious problem (the issue the documentation you pointed out just talks about Regexes being mutated, but that can be worked around by creating the Regex objects directly and not using the r"…" macro).
What I noticed was the following (in pcre.jl):
```julia
const JIT_STACK = RefValue{Ptr{Cvoid}}(C_NULL)
const MATCH_CONTEXT = RefValue{Ptr{Cvoid}}(C_NULL)
```
Those would need to be allocated on a per-thread basis, or use a lock to control access, and can’t easily be worked around at the moment.
---
### Author:  [@bkamins](https://discourse.julialang.org/u/bkamins)
#### Post date: [April 8, 2018, 10:35pm UTC](https://discourse.julialang.org/t/flaw-in-regex-support-for-string/9667/22 "2018-04-08T22:35:34Z")
A normal approach would be to put them in an array which should be relatively simple as when ` __init__ ` in PCRE is called we know the number of threads and this number cannot change later.
---
### Author:  [@ScottPJones](https://discourse.julialang.org/u/ScottPJones)
#### Post date: [April 8, 2018, 10:37pm UTC](https://discourse.julialang.org/t/flaw-in-regex-support-for-string/9667/23 "2018-04-08T22:37:52Z")
Yes, exactly. I’m not saying that it wouldn’t be simple, just that it’s another gotcha if you try to use threads at the moment.
Julia really needs a simple way (as other people such as Steven Johnson have asked for) to create something as thread-local storage, and make sure that accessing those variables is fast.
---
### Author:  [@nalimilan](https://discourse.julialang.org/u/nalimilan)
#### Post date: [April 9, 2018, 8:02am UTC](https://discourse.julialang.org/t/flaw-in-regex-support-for-string/9667/24 "2018-04-09T08:02:43Z")
> [@ScottPJones](#):
>
> I’m not sure I understand your question - the regex must be valid UTF (8 or 16), if you have the UTF compile flag set (which also means that the haystack is considered to be UTF, and will be checked for validity unless you put the NO\_CHECK\_UTF flag on for the match).
I meant when passing `NO_CHECK_UTF` for the match but not for the regex compilation (or for both since it doesn’t make a difference if the regex is valid).
> [@ScottPJones](#):
>
> Are you aware of other libraries that don’t follow the Unicode org recommendations, that allow processing of invalid strings in the way you want? Note:
>
> Boost security notice from 2013
>
> Rust handling of invalid UTF-8
>
> IETF RFC3629 security recommendations
>
> (and numerous others)
>
> I think the problems with doing so are fairly well-known among people who do a lot of work with string processing / conversions / encoding.
AFAICT these links refer to issues where invalid UTF-8 strings are incorrectly considered as valid, which is clearly problematic. In Julia `String` provides no guarantees that its contents are valid UTF-8, so people should call `isvalid` or use a validated string type if they rely on it for safety or correctness.
---
### Author:  [@ScottPJones](https://discourse.julialang.org/u/ScottPJones)
#### Post date: [April 9, 2018, 12:28pm UTC](https://discourse.julialang.org/t/flaw-in-regex-support-for-string/9667/25 "2018-04-09T12:28:37Z")
> [@nalimilan](#):
>
> AFAICT these links refer to issues where invalid UTF-8 strings are incorrectly considered as valid, which is clearly problematic.
There are also other issues listed in the [Unicode Technical Report on security considerations](https://www.unicode.org/reports/tr36/).
> [@nalimilan](#):
>
> In Julia String provides no guarantees that its contents are valid UTF-8, so people should call isvalid or use a validated string type if they rely on it for safety or correctness.
That would be bad for performance (I’ve shown that with my Strs.jl package - it’s faster to do validity checking upfront and then be able to use optimizations not possible if you don’t know the strings are valid).
It’s also like saying that you should put a `checkbounds` call before every array access, if you care about safety or correctness, instead of the approach of allowing you to use `@inbounds` when you are sure that the access is valid (which is the same philosophy that the PCRE library takes with providing the NO\_CHECK\_UTF flag).
That seems to be rather a double standard - Julia checks validity for array access (unless you are sure and specifically turn it off), but requires you to check at run-time for string validity if you care about safety and correctness.
---
### Author:  [@ScottPJones](https://discourse.julialang.org/u/ScottPJones)
#### Post date: [April 13, 2018, 1:56pm UTC](https://discourse.julialang.org/t/flaw-in-regex-support-for-string/9667/26 "2018-04-13T13:56:10Z")
Besides the well-known security issues with allowing invalid sequences in UTF-8, UTF-16, or UTF-32 encodings, there’s a big performance hit by not checking validity up front, both because the validity check can be a lot faster,
(even without SIMD instructions, but with SIMD instructions, which PCRE’s validity checks don’t use, it can be quite a bit faster), and then further, because the processing is a lot simpler if you know you can’t have short sequences, and just looking at the first byte (or word for UTF-16), you know you can access the next byte(s) or word.
The PCRE2 code will definitely read up to 5 bytes past the end of the string if you input an invalid UTF-8 sequence such as `\xfd` at the end.
I frankly don’t think that the PCRE2 maintainers would be interested in trying to go against the IETF, W3C, and Unicode org recommendations to allow for invalid sequences, as has been suggested on GitHub.
Here are some other examples of bad behavior with the current incorrect setting NO\_UTF\_CHECK for compile and match:
```julia
julia> findfirst(r"\x80", "aas;ldfjasdlkffoo abc\xc2\x80")
22:22
julia> findfirst(r"\xc2", "aas;ldfjasdlkffoo abc\xc2\x80")
julia> findfirst(r"\C\xbf", "aas;ldfjasdlkffoo abc\xfd\xbf\xbf\xbf\xbf\xbf")
ERROR: LoadError: PCRE JIT error: no more memory
Stacktrace:
[1] error at ./error.jl:33 [inlined]
[2] jit_compile at ./pcre.jl:109 [inlined]
[3] compile(::Regex) at ./regex.jl:56
[4] Regex(::String, ::UInt32, ::UInt32) at ./regex.jl:30
[5] Regex(::String) at ./regex.jl:51
[6] @r_str(::LineNumberNode, ::Module, ::Any, ::Vararg{Any,N} where N) at ./regex.jl:83
in expression starting at REPL[14]:1
```
---
### Author:  [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)
#### Post date: [April 13, 2018, 2:15pm UTC](https://discourse.julialang.org/t/flaw-in-regex-support-for-string/9667/27 "2018-04-13T14:15:56Z")
All of these examples involve invalid regexes.
---
### Author:  [@ScottPJones](https://discourse.julialang.org/u/ScottPJones)
#### Post date: [April 13, 2018, 7:14pm UTC](https://discourse.julialang.org/t/flaw-in-regex-support-for-string/9667/28 "2018-04-13T19:14:58Z")
> [@StefanKarpinski](#):
>
> All of these examples involve invalid regexes.
And? So is the example in [#26796](https://github.com/JuliaLang/julia/issues/26796#issue-313882003), and you seemed to be calling for the ability to search for illegal sequences there.
---
### Author:  [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)
#### Post date: [April 13, 2018, 7:25pm UTC](https://discourse.julialang.org/t/flaw-in-regex-support-for-string/9667/29 "2018-04-13T19:25:54Z")
Allowing invalid Unicode in substring search is very different than allowing invalid Unicode in regular expressions.
---
### Author:  [@bkamins](https://discourse.julialang.org/u/bkamins)
#### Post date: [April 13, 2018, 7:59pm UTC](https://discourse.julialang.org/t/flaw-in-regex-support-for-string/9667/30 "2018-04-13T19:59:11Z")
👍 for checking pattern for validity on construction `Regex` in base (as I understand that this is the conclusion what should be done) and I guess packages can do whatever is sensible within their ecosystem. I will make a PR ([https://github.com/JuliaLang/julia/pull/26802/files](https://github.com/JuliaLang/julia/pull/26802/files)) so that it can be discussed there.
---
### Author:  [@ScottPJones](https://discourse.julialang.org/u/ScottPJones)
#### Post date: [April 13, 2018, 9:32pm UTC](https://discourse.julialang.org/t/flaw-in-regex-support-for-string/9667/31 "2018-04-13T21:32:12Z")
The example I gave with the overlong sequence at the end (or near enough to the end) happens no matter whether the regex is valid or not, and can still cause an access violation.
---
### Author:  [@Palli](https://discourse.julialang.org/u/Palli)
#### Post date: [May 2, 2018, 10:59pm UTC](https://discourse.julialang.org/t/flaw-in-regex-support-for-string/9667/32 "2018-05-02T22:59:06Z")
> [@ScottPJones](#):
>
> it’s faster to do validity checking upfront and then be able to use optimizations not possible if you don’t know the strings are valid).
>
> […] That seems to be rather a double standard
It’s not always faster. That is if you get away with no validity checking. Say if I implemented grep, or wc or line count.
To keep this on topic, I read Perl had the best UTF-8 (may have said Unicode) support, and when I checked it read in illegal bytes. I.e. didn’t check for BOM and do anything with it, just read as illegal UTF-8 bytes. So is it likely PCRE (made for/used by Perl? Or only compatible with?) allows illegal? Isn’t Perl as lax as Julia?
On arrays, yes sometimes you access them only just once more or never… but Julia is made for heavy repeated access to them. The bounds checks (without annotations) can often be optimized by the compiler outside of the loop, so I see not “double standard”. For heavy text processing on same string, it’s like the case when checking can’t be optimized away. Then you need to annotate. Julia could cache in a “for\_sure\_\_legal\_UTF-8” bit kept with strings, so checking really only happens once.
---
### Author:  [@ScottPJones](https://discourse.julialang.org/u/ScottPJones)
#### Post date: [May 3, 2018, 3:23am UTC](https://discourse.julialang.org/t/flaw-in-regex-support-for-string/9667/33 "2018-05-03T03:23:25Z")
Besides performance issues, there are a lot of well known security issues with allowing invalid UTF-8 sequences.
---
### Author:  [@sdanisch](https://discourse.julialang.org/u/sdanisch)
#### Post date: [May 4, 2018, 3:45pm UTC](https://discourse.julialang.org/t/flaw-in-regex-support-for-string/9667/34 "2018-05-04T15:45:07Z")
Security issues, because of the existence of an unchecked string type? Or because of using it in an security sensitive contexts? The latter would be much less of an argument against julias current behaviour - especially since no one is against having a package like CheckedAndSecureStrings.jl 😉
---
### Author:  [@ScottPJones](https://discourse.julialang.org/u/ScottPJones)
#### Post date: [May 4, 2018, 4:38pm UTC](https://discourse.julialang.org/t/flaw-in-regex-support-for-string/9667/35 "2018-05-04T16:38:30Z")
When you can have strings that are faster, generally take less space, are easier to use, _and_ eliminate some of the known security issues (there are of course others that you really need to program defensively for, see the Unicode security recommendations), why would you even _want_ to use Base `String` and `Char`? 😉
Edit: These days, how can you know when some function will or will not be used in a security sensitive context?
This came up when discussing the hashing function used for strings in Julia (MurmurHash3), people said that it was important to make the `Dict` type safer against DOS attacks. Julia is going to be used for the new airplane navigation / collision avoidance stuff, right? Seems to me that being more security conscious would be a good thing.
---
### Author:  [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)
#### Post date: [May 4, 2018, 8:19pm UTC](https://discourse.julialang.org/t/flaw-in-regex-support-for-string/9667/36 "2018-05-04T20:19:19Z")
Please bring up actual security issues as you find them.
---
### Author:  [@ScottPJones](https://discourse.julialang.org/u/ScottPJones)
#### Post date: [May 4, 2018, 8:54pm UTC](https://discourse.julialang.org/t/flaw-in-regex-support-for-string/9667/37 "2018-05-04T20:54:58Z")
I’ve pointed these out more than once: here are some well known ones (again!):
[UTF-8 Exploits](http://unicode.org/reports/tr36/#UTF-8_Exploit)
---
### Author:  [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)
#### Post date: [May 4, 2018, 9:08pm UTC](https://discourse.julialang.org/t/flaw-in-regex-support-for-string/9667/38 "2018-05-04T21:08:19Z")
None of those are actual exploitable issues in Julia.
---
### Author:  [@ScottPJones](https://discourse.julialang.org/u/ScottPJones)
#### Post date: [May 4, 2018, 10:12pm UTC](https://discourse.julialang.org/t/flaw-in-regex-support-for-string/9667/39 "2018-05-04T22:12:53Z")
Of course they are. People have written web applications in Julia, so any of the exploits that take advantage of invalid strings. For example:
> Process A performs security checks, but does not check for non-shortest forms.
> Process B accepts the byte sequence from process A, and transforms it into UTF-16 while interpreting non-shortest forms.
> The UTF-16 text may then contain characters that should have been filtered out by process A.
If Process A is written in Julia, and passes through bad byte sequences, but because the Julia comparison operators won’t be able to do the security checks correctly, if process B (or maybe on the same process, in the OS, a library call, a database connection, or something written in a JVM based language) would be vulnerable.
Note that there are outstanding PRs to try to fix a few of the comparison problems in Julia already.
[Previous page](https://discourse.julialang.org/t/flaw-in-regex-support-for-string/9667.md?page=1)