# Bug in isvalid with an overlong UTF-8 encoded vector or string

**URL:** <https://discourse.julialang.org/t/bug-in-isvalid-with-an-overlong-utf-8-encoded-vector-or-string/15290>\
**Category:** Internals & Design\
**Created:** [September 21, 2018, 3:28pm UTC](https://discourse.julialang.org/t/bug-in-isvalid-with-an-overlong-utf-8-encoded-vector-or-string/15290 "2018-09-21T15:28:32Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![ScottPJones](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/scottpjones/32/146_2.png) [@ScottPJones](https://discourse.julialang.org/u/ScottPJones)\
**Post date:** [September 21, 2018, 3:28pm UTC](https://discourse.julialang.org/t/bug-in-isvalid-with-an-overlong-utf-8-encoded-vector-or-string/15290/1 "2018-09-21T15:28:32Z")

</div>

The code (at strings/string.jl:168 on version 1.0) correctly returns false for 2 byte and 4 byte overlong sequences, but is broken for 3 byte overlong sequences.

```julia
julia> isvalid(String, UInt8[0xf0,128,128,128])
false

julia> isvalid(String, UInt8[0xe0,128,128])
true

julia> isvalid(String, UInt8[0xc0,128])
false

```

---

<div class="post-metadata">

**Author:** ![StefanKarpinski](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stefankarpinski/32/24_2.png) [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)\
**Post date:** [September 21, 2018, 7:30pm UTC](https://discourse.julialang.org/t/bug-in-isvalid-with-an-overlong-utf-8-encoded-vector-or-string/15290/2 "2018-09-21T19:30:40Z")

</div>

Thanks for the bug report. Issue filed: [invalid bug for three-byte characters · Issue #29311 · JuliaLang/julia · GitHub](https://github.com/JuliaLang/julia/issues/29311).

---

<div class="post-metadata">

**Author:** ![ScottPJones](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/scottpjones/32/146_2.png) [@ScottPJones](https://discourse.julialang.org/u/ScottPJones)\
**Post date:** [September 21, 2018, 8:27pm UTC](https://discourse.julialang.org/t/bug-in-isvalid-with-an-overlong-utf-8-encoded-vector-or-string/15290/3 "2018-09-21T20:27:49Z")

</div>

If nobody beats me to it, I’ll fix it (but will need somebody to create the PR).  
Thanks for your response!  
Was good to see you all at the Meetup this week!

---

<div class="post-metadata">

**Author:** ![ScottPJones](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/scottpjones/32/146_2.png) [@ScottPJones](https://discourse.julialang.org/u/ScottPJones)\
**Post date:** [September 21, 2018, 10:07pm UTC](https://discourse.julialang.org/t/bug-in-isvalid-with-an-overlong-utf-8-encoded-vector-or-string/15290/4 "2018-09-21T22:07:50Z")

</div>

If somebody wants to pick it up, I have a fix (with tests!) on my fork:

> **[GitHub - ScottPJones/julia at spj/fixinvalid](https://github.com/ScottPJones/julia/tree/spj/fixinvalid)**
>
> spj/fixinvalid

Now, I have to hang my head in shame, because when I fixed some other bugs in UTF-8 validation 3 years ago,  
(such as detecting UTF-16 surrogates present in UTF-8), I missed a check (for the overlong 3-byte case).  
[https://github.com/JuliaLang/julia/issues/11141](https://github.com/JuliaLang/julia/issues/11141)

Hopefully some nice person can pull this fix in!

Thanks,  
Scott