# Regex on byte vector

**URL:** <https://discourse.julialang.org/t/regex-on-byte-vector/47754>\
**Category:** General Usage\
**Tags:** regex\
**Created:** [October 5, 2020, 12:19am UTC](https://discourse.julialang.org/t/regex-on-byte-vector/47754 "2020-10-05T00:19:06Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![kernelmethod](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kernelmethod/32/35480_2.png) [@kernelmethod](https://discourse.julialang.org/u/kernelmethod)\
**Post date:** [October 5, 2020, 12:19am UTC](https://discourse.julialang.org/t/regex-on-byte-vector/47754/1 "2020-10-05T00:19:06Z")

</div>

Hey there! In Python, I can apply a regular expression to a byte string using something like

```python
import re
patt = re.compile(b"^ab+c$")
patt.match(b"abbbc")

```

Is there a way of doing something similar in Julia? I know I can do something like

```julia
my_occursin!(patt::Regex, x::Vector{UInt8}) = occursin(patt, String(x))

```

but this isn’t quite the same because `String(x)` mutates `x`:

```julia
julia> x = Vector{UInt8}("abc")
3-element Array{UInt8,1}:
 0x61
 0x62
 0x63

julia> String(x)
"abc"

julia> x
UInt8[]

```

Is there a way to apply a regex to a `Vector{UInt8}` that doesn’t require mutating the byte vector?

Thanks!

---

<div class="post-metadata">

**Author:** ![lmiq](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lmiq/32/18314_2.png) [@lmiq](https://discourse.julialang.org/u/lmiq)\
**Post date:** [October 5, 2020, 12:37am UTC](https://discourse.julialang.org/t/regex-on-byte-vector/47754/2 "2020-10-05T00:37:20Z")

</div>

> [@kernelmethod](#):
>
> `my_occursin!(patt::Regex, x::Vector{UInt8}) = occursin(patt, String(x))`

I don’t know if there is a better solution, but in the worst case, you can copy `x`:

```julia
my_occursin!(patt::Regex, x::Vector{UInt8}) = occursin(patt, copy(String(x)))

```

Actually you can use a view:

```julia
julia> x = Vector{UInt8}("abbbc")
5-element Array{UInt8,1}:
 0x61
 0x62
 0x62
 0x62
 0x63

julia> occursin(Regex("^ab..c"),String(@view x[1:end]))
true

julia> x
5-element Array{UInt8,1}:
 0x61
 0x62
 0x62
 0x62
 0x63

```

---

<div class="post-metadata">

**Author:** ![kernelmethod](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kernelmethod/32/35480_2.png) [@kernelmethod](https://discourse.julialang.org/u/kernelmethod)\
**Post date:** [October 5, 2020, 12:47am UTC](https://discourse.julialang.org/t/regex-on-byte-vector/47754/3 "2020-10-05T00:47:47Z")

</div>

The `view`-based idea is good, but I think that under the hood it’s also just creating a copy of `x`: [https://github.com/JuliaLang/julia/blob/55a6dab76329b693f0fab372b1a80289bff01a90/base/strings/string.jl#L51](https://github.com/JuliaLang/julia/blob/55a6dab76329b693f0fab372b1a80289bff01a90/base/strings/string.jl#L51)

---

<div class="post-metadata">

**Author:** ![lmiq](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lmiq/32/18314_2.png) [@lmiq](https://discourse.julialang.org/u/lmiq)\
**Post date:** [October 5, 2020, 12:49am UTC](https://discourse.julialang.org/t/regex-on-byte-vector/47754/4 "2020-10-05T00:49:49Z")

</div>

Indeed, the help entry for String is explicit in suggesting the copy, with some reasons given.

```julia
help?> String
search: String string StringIndexError Cstring Cwstring bitstring SubString include_string setrounding unsafe_string AbstractString

  String(v::AbstractVector{UInt8})

  Create a new String object from a byte vector v containing UTF-8 encoded characters. If v is Vector{UInt8} it will be truncated to
  zero length and future modification of v cannot affect the contents of the resulting string. To avoid truncation use
  String(copy(v)).

  When possible, the memory of v will be used without copying when the String object is created. This is guaranteed to be the case
  for byte vectors returned by take! on a writable IOBuffer and by calls to read(io, nb). This allows zero-copy conversion of I/O
  data to strings. In other cases, Vector{UInt8} data may be copied, but v is truncated anyway to guarantee consistent behavior.

  ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

  String(s::AbstractString)

  Convert a string to a contiguous byte array representation encoded as UTF-8 bytes. This representation is often appropriate for
  passing strings to C.

```

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [October 5, 2020, 2:49am UTC](https://discourse.julialang.org/t/regex-on-byte-vector/47754/5 "2020-10-05T02:49:43Z")

</div>

> [@kernelmethod](#):
>
> Is there a way to apply a regex to a `Vector{UInt8}` that doesn’t require mutating the byte vector?

Certainly the low-level PCRE library supports this, so it would be possible to implement by replicating or generalizing some of the code in Base.

(This has come up a few times; I’ve often thought that it might be useful to put together a `StringView` type that wraps around any `a::AbstractVector{UInt8}` with `stride(a,1) == 1` and exposes the regex methods etcetera.)

---

<div class="post-metadata">

**Author:** ![kernelmethod](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kernelmethod/32/35480_2.png) [@kernelmethod](https://discourse.julialang.org/u/kernelmethod)\
**Post date:** [October 5, 2020, 5:09am UTC](https://discourse.julialang.org/t/regex-on-byte-vector/47754/6 "2020-10-05T05:09:38Z")

</div>

> [@stevengj](#):
>
> Certainly the low-level PCRE library supports this, so it would be possible to implement by replicating or generalizing some of the code in Base.
> 
> (This has come up a few times; I’ve often thought that it might be useful to put together a `StringView` type that wraps around any `a::AbstractVector{UInt8}` with `stride(a,1) == 1` and exposes the regex methods etcetera.)

Sure, I’ll try out this approach 🙂 If I get anywhere with it I’ll open a PR, since I imagine it’s probably generally useful functionality that more people could do with.

---

<div class="post-metadata">

**Author:** ![kernelmethod](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kernelmethod/32/35480_2.png) [@kernelmethod](https://discourse.julialang.org/u/kernelmethod)\
**Post date:** [October 9, 2020, 4:44am UTC](https://discourse.julialang.org/t/regex-on-byte-vector/47754/7 "2020-10-09T04:44:11Z")

</div>

I’ve opened up an issue ([https://github.com/JuliaLang/julia/issues/37956](https://github.com/JuliaLang/julia/issues/37956)) in case anybody is interested in taking this on and/or tracking its progress.

---

<div class="post-metadata">

**Author:** ![mgkuhn](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mgkuhn/32/6276_2.png) [@mgkuhn](https://discourse.julialang.org/u/mgkuhn)\
**Post date:** [October 10, 2020, 8:45pm UTC](https://discourse.julialang.org/t/regex-on-byte-vector/47754/8 "2020-10-10T20:45:42Z")

</div>

I think this feature request is a symptom of a far more wide-ranging problem that should be solved instead: namely that Julia stdlib still lacks a version of `String` suitable for processing binary and other non-UTF-8 string data, with all the facilities that `String` offers, including, but by far not limited to, regular expressions. I would therefore suggest that instead feature request [#37979](https://github.com/JuliaLang/julia/issues/37979) is a more generic solution the the same problem, namely making the entire `AbstractString` API (including regex) easily available for processing byte sequences where Unicode is not of interest, by adding a binary/byte/basic-latin sibling of `String` which could be called `BString` . Basically a 1 byte = 1 character version of `String` without a UTF-8 decoder running behind the scenes all the time. `Vector{UInt8}` (mutable) seems more a workaround for the lack of an immutable binary string type.

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [October 11, 2020, 2:32am UTC](https://discourse.julialang.org/t/regex-on-byte-vector/47754/9 "2020-10-11T02:32:29Z")

</div>

> [@mgkuhn](#):
>
> Basically a 1 byte = 1 character version of `String`

What does this mean? ASCII? What is a “character” if it is not a Unicode codepoint?

Besides, the PCRE regex library used by Julia expects UTF-8.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [October 11, 2020, 8:18am UTC](https://discourse.julialang.org/t/regex-on-byte-vector/47754/10 "2020-10-11T08:18:00Z")

</div>

> [@mgkuhn](#):
>
> adding a binary/byte/basic-latin sibling of `String` which could be called `BString`

I am not sure why this cannot just be an additional package. Cf [Kristoffer’s suggestion](https://github.com/JuliaLang/julia/issues/37979#issuecomment-706664234) in the issue you opened.

That said, what’s the use case for

> [@mgkuhn](#):
>
> a 1 byte = 1 character version of `String`

besides ASCII? EBCDIC or historical 8-bit codepages? Why not just convert those to UTF8?

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [November 10, 2020, 9:30pm UTC](https://discourse.julialang.org/t/regex-on-byte-vector/47754/11 "2020-11-10T21:30:28Z")

</div>

> [@stevengj](#):
>
> > [@kernelmethod](#):
> >
> > Is there a way to apply a regex to a `Vector{UInt8}` that doesn’t require mutating the byte vector?
> 
> Certainly the low-level PCRE library supports this, so it would be possible to implement by replicating or generalizing some of the code in Base.

Update: I created a draft package for this:

> **[GitHub - JuliaStrings/StringViews.jl: String-like views of arbitrary Julia...](https://github.com/JuliaStrings/StringViews.jl)**
>
> String-like views of arbitrary Julia byte arrays. Contribute to JuliaStrings/StringViews.jl development by creating an account on GitHub.