# Regular expressions returning offsets in bytes not characters

**URL:** https://discourse.julialang.org/t/regular-expressions-returning-offsets-in-bytes-not-characters/4676
**Category:** General Usage
**Tags:** question, regex
**Created:** [July 6, 2017, 3:26am UTC](https://discourse.julialang.org/t/regular-expressions-returning-offsets-in-bytes-not-characters/4676 "2017-07-06T03:26:33Z")
**Posts on this page:** 10
**Page:** 1

<div class="post-metadata">

### Author: ![cstook](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cstook/32/3731_2.png) [@cstook](https://discourse.julialang.org/u/cstook)
#### Post date: [July 6, 2017, 3:26am UTC](https://discourse.julialang.org/t/regular-expressions-returning-offsets-in-bytes-not-characters/4676/1 "2017-07-06T03:26:33Z")

</div>

Hello,

Regular expressions seems to be returning offsets in bytes not characters. Is this the intended behavior? Is there a way to get the offsets in characters?

```julia
julia> m1 = match(r"(3).*(5)"ix,"123a56789")
RegexMatch("3a5", 1="3", 2="5")

julia> print(m1.offsets)
[3,5]
julia> m2 = match(r"(3).*(5)"ix,"123α56789")
RegexMatch("3α5", 1="3", 2="5")

julia> print(m2.offsets)
[3,6]
julia>

```

Thanks

---

<div class="post-metadata">

### Author: ![ScottPJones](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/scottpjones/32/146_2.png) [@ScottPJones](https://discourse.julialang.org/u/ScottPJones)
#### Post date: [July 6, 2017, 3:35am UTC](https://discourse.julialang.org/t/regular-expressions-returning-offsets-in-bytes-not-characters/4676/2 "2017-07-06T03:35:33Z")

</div>

That’s just the way it was designed.  
If you want offsets of the Unicode codepoints, you could use the `LegacyStrings` package, and use the `UTF32String` type.

---

<div class="post-metadata">

### Author: ![cstook](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cstook/32/3731_2.png) [@cstook](https://discourse.julialang.org/u/cstook)
#### Post date: [July 6, 2017, 4:55am UTC](https://discourse.julialang.org/t/regular-expressions-returning-offsets-in-bytes-not-characters/4676/3 "2017-07-06T04:55:49Z")

</div>

OK. Thanks.

---

<div class="post-metadata">

### Author: ![johann.spies](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johann.spies/32/8805_2.png) [@johann.spies](https://discourse.julialang.org/u/johann.spies)
#### Post date: [July 6, 2017, 6:47am UTC](https://discourse.julialang.org/t/regular-expressions-returning-offsets-in-bytes-not-characters/4676/4 "2017-07-06T06:47:49Z")

</div>

> [@cstook](#):
>
> m1 = match(r"(3).\*(5)"ix,“123a56789”)

Or maybe (I am not sure if this is what you want):

```julia
julia> m1 = match(r"(?<first>3).*(?<second>5)"ix,"123a56789")
RegexMatch("3a5", first="3", second="5")

julia> m1.offsets
2-element Array{Int64,1}:
 3
 5

julia> m1[:first]
"3"

julia> m1[:second]
"5"

```

Regards  
Johann

---

<div class="post-metadata">

### Author: ![nalimilan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nalimilan/32/147_2.png) [@nalimilan](https://discourse.julialang.org/u/nalimilan)
#### Post date: [July 6, 2017, 7:48am UTC](https://discourse.julialang.org/t/regular-expressions-returning-offsets-in-bytes-not-characters/4676/5 "2017-07-06T07:48:00Z")

</div>

You don’t need to use `UTF32String` nor LegacyStrings. Maybe you can tell us more about what you want to do?

String indices are in bytes in Julia (at least for the default `String` type) because that’s the only efficient way of accessing a character in variable length encodings like UTF-8 or UTF-16. Counting characters requires iterating over the string from its beginning.

If what you really need is the number of characters before the first match, you can just do something like `length(s[1:m1.offsets[1]])` or (a bit more efficient) `length(SubString(s, 1, m1.offsets[1]))`. But beware that “character” is a subtle notion, which does not necessarily correspond to Unicode codepoints. See `graphemes` if what you need is the user-perceived number of characters.

---

<div class="post-metadata">

### Author: ![waldyrious](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/waldyrious/32/80_2.png) [@waldyrious](https://discourse.julialang.org/u/waldyrious)
#### Post date: [July 6, 2017, 9:30am UTC](https://discourse.julialang.org/t/regular-expressions-returning-offsets-in-bytes-not-characters/4676/6 "2017-07-06T09:30:25Z")

</div>

> [@nalimilan](#):
>
> See `graphemes` if what you need is the user-perceived number of characters.

Docs link for convenience: [https://docs.julialang.org/en/stable/stdlib/strings/#Base.UTF8proc.graphemes](https://docs.julialang.org/en/stable/stdlib/strings/#Base.UTF8proc.graphemes)

---

<div class="post-metadata">

### Author: ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)
#### Post date: [July 6, 2017, 12:37pm UTC](https://discourse.julialang.org/t/regular-expressions-returning-offsets-in-bytes-not-characters/4676/7 "2017-07-06T12:37:37Z")

</div>

> [@cstook](#):
>
> Regular expressions seems to be returning offsets in bytes not characters. Is this the intended behavior? Is there a way to get the offsets in characters?

You can get the offset in characters from the `ind2chr` function.

The reason that they return the offsets in bytes (“code units” of the underlying UTF-8 encoding) is this is how Julia `String` is indexed, so byte offsets are usually the most useful thing to know (e.g. to extract substrings from the original string).

```julia
julia> s = "123α56789"
"123α56789"

julia> m2 = match(r"(3).*(5)"ix, s)
RegexMatch("3α5", 1="3", 2="5")

julia> m2.offsets
2-element Array{Int64,1}:
 3
 6

julia> s[m2.offsets]
"35"

julia> ind2chr.(s, m2.offsets)
2-element Array{Int64,1}:
 3
 5

```

---

<div class="post-metadata">

### Author: ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)
#### Post date: [July 6, 2017, 12:41pm UTC](https://discourse.julialang.org/t/regular-expressions-returning-offsets-in-bytes-not-characters/4676/8 "2017-07-06T12:41:37Z")

</div>

> [@nalimilan](#):
>
> `length(s[1:m1.offsets[1]])` or (a bit more efficient) `length(SubString(s, 1, m1.offsets[1]))`

Don’t do this. Use `ind2chr`.

But you’re right that you may want to use `graphemes`, e.g. `length(graphemes(SubString(s, 1, m2.offsets[2])))`, if you want to count user-perceived characters.

---

<div class="post-metadata">

### Author: ![cstook](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cstook/32/3731_2.png) [@cstook](https://discourse.julialang.org/u/cstook)
#### Post date: [July 6, 2017, 10:27pm UTC](https://discourse.julialang.org/t/regular-expressions-returning-offsets-in-bytes-not-characters/4676/9 "2017-07-06T22:27:14Z")

</div>

I was using a mixture of `length` and `match.offsets` to compute indices into strings. This was very bad. Switching to using only `match.offsets` fixed the problem.

Thank everyone for your help.

---

<div class="post-metadata">

### Author: ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)
#### Post date: [July 7, 2017, 1:19pm UTC](https://discourse.julialang.org/t/regular-expressions-returning-offsets-in-bytes-not-characters/4676/10 "2017-07-07T13:19:47Z")

</div>

> [@cstook](#):
>
> I was using a mixture of `length` and `match.offsets` to compute indices into strings.

Never use `length(s)` for indexing strings; use `endof(s)` instead.
