Regex on byte vector

Hey there! In Python, I can apply a regular expression to a byte string using something like

import re
patt = re.compile(b"^ab+c$")

Is there a way of doing something similar in Julia? I know I can do something like

my_occursin!(patt::Regex, x::Vector{UInt8}) = occursin(patt, String(x))

but this isn’t quite the same because String(x) mutates x:

julia> x = Vector{UInt8}("abc")
3-element Array{UInt8,1}:

julia> String(x)

julia> x

Is there a way to apply a regex to a Vector{UInt8} that doesn’t require mutating the byte vector?


I don’t know if there is a better solution, but in the worst case, you can copy x:

my_occursin!(patt::Regex, x::Vector{UInt8}) = occursin(patt, copy(String(x)))

Actually you can use a view:

julia> x = Vector{UInt8}("abbbc")
5-element Array{UInt8,1}:

julia> occursin(Regex("^ab..c"),String(@view x[1:end]))

julia> x
5-element Array{UInt8,1}:

The view-based idea is good, but I think that under the hood it’s also just creating a copy of x:

Indeed, the help entry for String is explicit in suggesting the copy, with some reasons given.

help?> String
search: String string StringIndexError Cstring Cwstring bitstring SubString include_string setrounding unsafe_string AbstractString


  Create a new String object from a byte vector v containing UTF-8 encoded characters. If v is Vector{UInt8} it will be truncated to
  zero length and future modification of v cannot affect the contents of the resulting string. To avoid truncation use

  When possible, the memory of v will be used without copying when the String object is created. This is guaranteed to be the case
  for byte vectors returned by take! on a writable IOBuffer and by calls to read(io, nb). This allows zero-copy conversion of I/O
  data to strings. In other cases, Vector{UInt8} data may be copied, but v is truncated anyway to guarantee consistent behavior.



  Convert a string to a contiguous byte array representation encoded as UTF-8 bytes. This representation is often appropriate for
  passing strings to C.

1 Like

Certainly the low-level PCRE library supports this, so it would be possible to implement by replicating or generalizing some of the code in Base.

(This has come up a few times; I’ve often thought that it might be useful to put together a StringView type that wraps around any a::AbstractVector{UInt8} with stride(a,1) == 1 and exposes the regex methods etcetera.)


Sure, I’ll try out this approach :slightly_smiling_face: If I get anywhere with it I’ll open a PR, since I imagine it’s probably generally useful functionality that more people could do with.

I’ve opened up an issue ( in case anybody is interested in taking this on and/or tracking its progress.


I think this feature request is a symptom of a far more wide-ranging problem that should be solved instead: namely that Julia stdlib still lacks a version of String suitable for processing binary and other non-UTF-8 string data, with all the facilities that String offers, including, but by far not limited to, regular expressions. I would therefore suggest that instead feature request #37979 is a more generic solution the the same problem, namely making the entire AbstractString API (including regex) easily available for processing byte sequences where Unicode is not of interest, by adding a binary/byte/basic-latin sibling of String which could be called BString . Basically a 1 byte = 1 character version of String without a UTF-8 decoder running behind the scenes all the time. Vector{UInt8} (mutable) seems more a workaround for the lack of an immutable binary string type.

What does this mean? ASCII? What is a β€œcharacter” if it is not a Unicode codepoint?

Besides, the PCRE regex library used by Julia expects UTF-8.


I am not sure why this cannot just be an additional package. Cf Kristoffer’s suggestion in the issue you opened.

That said, what’s the use case for

besides ASCII? EBCDIC or historical 8-bit codepages? Why not just convert those to UTF8?

Update: I created a draft package for this: