# Valid chars

**URL:** <https://discourse.julialang.org/t/valid-chars/21489>\
**Category:** Offtopic\
**Tags:** question, strings\
**Created:** [March 5, 2019, 10:10am UTC](https://discourse.julialang.org/t/valid-chars/21489 "2019-03-05T10:10:52Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![yakir12](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yakir12/32/297_2.png) [@yakir12](https://discourse.julialang.org/u/yakir12)\
**Post date:** [March 5, 2019, 10:10am UTC](https://discourse.julialang.org/t/valid-chars/21489/1 "2019-03-05T10:10:52Z")

</div>

Disclaimer: I’m totally abusing the fact that there are people here that really know what they’re doing when it comes to strings, this has little to do with Julia.

When working with databases that store strings, sometimes the string might include a character that is invalid, that is, it fails `isvalid`. I have some control over the input to these databases. So I could limit the user’s input to exclude certain characters, all in the interest of keeping the data valid for the modest cost of user expressiveness. **How would you best limit a user’s input to guarantee the string is valid?**

Thanks!

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [March 5, 2019, 10:20am UTC](https://discourse.julialang.org/t/valid-chars/21489/2 "2019-03-05T10:20:56Z")

</div>

Not sure what you are asking about, since `isvalid` tests bytes in a string from a particular offset. Valid strings can fail for `isvalid` from certain positions, as shown in the docstring (`?isvalid`).

---

<div class="post-metadata">

**Author:** ![yakir12](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yakir12/32/297_2.png) [@yakir12](https://discourse.julialang.org/u/yakir12)\
**Post date:** [March 5, 2019, 10:35am UTC](https://discourse.julialang.org/t/valid-chars/21489/3 "2019-03-05T10:35:55Z")

</div>

I’m trying to build some kind MWE to show you what I mean but it’s kind of hard. Anyway, one of the strings includes a degree symbol that gets read as ‘\xb0’. This seems to be an invalid character.

It is entirely possible that I’m misusing `isvalid` here!

---

<div class="post-metadata">

**Author:** ![yakir12](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yakir12/32/297_2.png) [@yakir12](https://discourse.julialang.org/u/yakir12)\
**Post date:** [March 5, 2019, 10:43am UTC](https://discourse.julialang.org/t/valid-chars/21489/4 "2019-03-05T10:43:04Z")

</div>

[Here](https://www.dropbox.com/s/10b8yodzm8eq3kv/tmp.json?dl=0) is an example of an invalid piece of text.

To make it even simpler, here’s a piece of code to download, read, and test the example:

```julia
link = "https://www.dropbox.com/s/10b8yodzm8eq3kv/tmp.json?dl=0"
run(`wget -O tmp.json $link`)
txt = read("tmp.json", String)
@assert all(isvalid, txt) # this fails

```

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [March 5, 2019, 10:49am UTC](https://discourse.julialang.org/t/valid-chars/21489/5 "2019-03-05T10:49:21Z")

</div>

That just looks like malformed UTF8, ie the file is not valid UTF8-encoding. Looks something like Latin-1, if the `0xb0` is `°` (the degree sign).

It would be best to establish the valid format (whoever generated the file should know), or failing that, convert heuristically and/or reject invalid strings.

---

<div class="post-metadata">

**Author:** ![yakir12](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yakir12/32/297_2.png) [@yakir12](https://discourse.julialang.org/u/yakir12)\
**Post date:** [March 5, 2019, 10:55am UTC](https://discourse.julialang.org/t/valid-chars/21489/6 "2019-03-05T10:55:28Z")

</div>

> [@Tamas\_Papp](#):
>
> It would be best to establish the valid format

Great! So in an ideal world, the validity of the UTF8-encoding of the strings input into the database would first be checked before it got accepted. This sounds straight forward, but I feel it might be more complicated than that… Any pitfalls I should be warned about? And btw, in Julia, is there any way to test for the validity of the UTF8 encoding of a string?

Right now I do the latter (with `isvalid`):

> [@Tamas\_Papp](#):
>
> reject invalid strings

---

<div class="post-metadata">

**Author:** ![Sukera](https://avatars.discourse-cdn.com/v4/letter/s/ce7236/32.png) [@Sukera](https://discourse.julialang.org/u/Sukera)\
**Post date:** [March 5, 2019, 11:03am UTC](https://discourse.julialang.org/t/valid-chars/21489/7 "2019-03-05T11:03:15Z")

</div>

In Julia, strings are stored as UTF-8, so `isvalid(String, s)` where `s` is read in by e.g. `s = readline(<path>)` should be checking whether the read string is valid UTF-8. You could also read the string in as `UInt8` and do `isvalid(String, s)` on that, going by the help page.

```julia
help?> isvalid
#...
isvalid(T, value) -> Bool

Returns true if the given value is valid for that type. 
Types currently can be either AbstractChar or String. 
Values for AbstractChar can be of type AbstractChar or UInt32. 
Values for String can be of that type, or Vector{UInt8}.
#...

julia> s = "Hello"
"Hello"

julia> UInt8[s...]
5-element Array{UInt8,1}:
   0x48   
   0x65
   0x6c
   0x6c
   0x6f                                                 

julia> isvalid(String, ans)
true                                                  

```