Regex on byte vector

General Usage
1

Hey there! In Python, I can apply a regular expression to a byte string using something like

import re
patt = re.compile(b"^ab+c$")
patt.match(b"abbbc")

Is there a way of doing something similar in Julia? I know I can do something like

my_occursin!(patt::Regex, x::Vector{UInt8}) = occursin(patt, String(x))

but this isn’t quite the same because String(x) mutates x:

julia> x = Vector{UInt8}("abc")
3-element Array{UInt8,1}:
 0x61
 0x62
 0x63

julia> String(x)
"abc"

julia> x
UInt8[]

Is there a way to apply a regex to a Vector{UInt8} that doesn’t require mutating the byte vector?

Thanks!

2

I don’t know if there is a better solution, but in the worst case, you can copy x:

my_occursin!(patt::Regex, x::Vector{UInt8}) = occursin(patt, copy(String(x)))

Actually you can use a view:

julia> x = Vector{UInt8}("abbbc")
5-element Array{UInt8,1}:
 0x61
 0x62
 0x62
 0x62
 0x63

julia> occursin(Regex("^ab..c"),String(@view x[1:end]))
true

julia> x
5-element Array{UInt8,1}:
 0x61
 0x62
 0x62
 0x62
 0x63
2 Likes
3

The view-based idea is good, but I think that under the hood it’s also just creating a copy of x: https://github.com/JuliaLang/julia/blob/55a6dab76329b693f0fab372b1a80289bff01a90/base/strings/string.jl#L51

4

Indeed, the help entry for String is explicit in suggesting the copy, with some reasons given.

help?> String
search: String string StringIndexError Cstring Cwstring bitstring SubString include_string setrounding unsafe_string AbstractString

  String(v::AbstractVector{UInt8})

  Create a new String object from a byte vector v containing UTF-8 encoded characters. If v is Vector{UInt8} it will be truncated to
  zero length and future modification of v cannot affect the contents of the resulting string. To avoid truncation use
  String(copy(v)).

  When possible, the memory of v will be used without copying when the String object is created. This is guaranteed to be the case
  for byte vectors returned by take! on a writable IOBuffer and by calls to read(io, nb). This allows zero-copy conversion of I/O
  data to strings. In other cases, Vector{UInt8} data may be copied, but v is truncated anyway to guarantee consistent behavior.

  ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

  String(s::AbstractString)

  Convert a string to a contiguous byte array representation encoded as UTF-8 bytes. This representation is often appropriate for
  passing strings to C.
1 Like
5

Certainly the low-level PCRE library supports this, so it would be possible to implement by replicating or generalizing some of the code in Base.

(This has come up a few times; I’ve often thought that it might be useful to put together a StringView type that wraps around any a::AbstractVector{UInt8} with stride(a,1) == 1 and exposes the regex methods etcetera.)

6 Likes
6

Sure, I’ll try out this approach :slightly_smiling_face: If I get anywhere with it I’ll open a PR, since I imagine it’s probably generally useful functionality that more people could do with.

7

I’ve opened up an issue (https://github.com/JuliaLang/julia/issues/37956) in case anybody is interested in taking this on and/or tracking its progress.

3 Likes
8

I think this feature request is a symptom of a far more wide-ranging problem that should be solved instead: namely that Julia stdlib still lacks a version of String suitable for processing binary and other non-UTF-8 string data, with all the facilities that String offers, including, but by far not limited to, regular expressions. I would therefore suggest that instead feature request #37979 is a more generic solution the the same problem, namely making the entire AbstractString API (including regex) easily available for processing byte sequences where Unicode is not of interest, by adding a binary/byte/basic-latin sibling of String which could be called BString . Basically a 1 byte = 1 character version of String without a UTF-8 decoder running behind the scenes all the time. Vector{UInt8} (mutable) seems more a workaround for the lack of an immutable binary string type.

9

What does this mean? ASCII? What is a “character” if it is not a Unicode codepoint?

Besides, the PCRE regex library used by Julia expects UTF-8.

2 Likes
10

I am not sure why this cannot just be an additional package. Cf Kristoffer’s suggestion in the issue you opened.

That said, what’s the use case for

besides ASCII? EBCDIC or historical 8-bit codepages? Why not just convert those to UTF8?

11

Update: I created a draft package for this:

2 Likes