# What is the Interface of AbstractString?

**URL:** <https://discourse.julialang.org/t/what-is-the-interface-of-abstractstring/8937>\
**Category:** General Usage\
**Created:** [February 9, 2018, 6:59am UTC](https://discourse.julialang.org/t/what-is-the-interface-of-abstractstring/8937 "2018-02-09T06:59:22Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![samoconnor](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/samoconnor/32/1802_2.png) [@samoconnor](https://discourse.julialang.org/u/samoconnor)\
**Post date:** [February 9, 2018, 6:59am UTC](https://discourse.julialang.org/t/what-is-the-interface-of-abstractstring/8937/1 "2018-02-09T06:59:22Z")

</div>

The Interfaces documentation does not cover AbstractString: [https://docs.julialang.org/en/latest/manual/interfaces](https://docs.julialang.org/en/latest/manual/interfaces)

What is the minimal set of methods that must be defined for a custom AbstractString type?  
Is there a good example of a custom string type somewhere, or should I just follow the pattern of SubString:  
[https://github.com/JuliaLang/julia/blob/master/base/strings/substring.jl](https://github.com/JuliaLang/julia/blob/master/base/strings/substring.jl)

---

<div class="post-metadata">

**Author:** ![NaOH](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/naoh/32/5632_2.png) [@NaOH](https://discourse.julialang.org/u/NaOH)\
**Post date:** [February 9, 2018, 4:05pm UTC](https://discourse.julialang.org/t/what-is-the-interface-of-abstractstring/8937/2 "2018-02-09T16:05:16Z")

</div>

There are more string related packages here:

- [https://github.com/JuliaString](https://github.com/JuliaString)

But I haven’t found a detailed answer there, these packages use string macros.

---

<div class="post-metadata">

**Author:** ![nalimilan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nalimilan/32/147_2.png) [@nalimilan](https://discourse.julialang.org/u/nalimilan)\
**Post date:** [February 9, 2018, 7:12pm UTC](https://discourse.julialang.org/t/what-is-the-interface-of-abstractstring/8937/3 "2018-02-09T19:12:40Z")

</div>

It’ now documented in `?AbstractString`: [https://github.com/JuliaLang/julia/blob/16620b62443f0ffd3aa5fa58174b5d66a28cc646/base/strings/basic.jl#L3-L41](https://github.com/JuliaLang/julia/blob/16620b62443f0ffd3aa5fa58174b5d66a28cc646/base/strings/basic.jl#L3-L41). Should probably be added to interfaces too.

---

<div class="post-metadata">

**Author:** ![samoconnor](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/samoconnor/32/1802_2.png) [@samoconnor](https://discourse.julialang.org/u/samoconnor)\
**Post date:** [February 9, 2018, 11:14pm UTC](https://discourse.julialang.org/t/what-is-the-interface-of-abstractstring/8937/4 "2018-02-09T23:14:24Z")

</div>

Thanks Milan,

The `?AbstractString` docstring doesn’t seem to have a list of required methods, but there is a section commented `## required string functions ##` in the source just after the doc string. From this I conclude that the required methods are:

- `ncodeunits(s::MyAbstractString) -> Int`
- `codeunit(s::MyAbstractString, i::Integer) -> Union{UInt8, UInt16, UInt32}`
- `isvalid(s::MyAbstractString, i::Integer) -> Bool`
- `next(s::MyAbstractString, i::Integer) -> Tuple{Char, Int}`
- `length(s::MyAbstractString)` or define Base.IteratorSize(::MyAbstractString) = Base.SizeUnknown()

Based on looking at `substring.jl` I guess that it is probably good to specialise some of these functions as well (where the implementation permits): `getindex`, `cmp`, `pointer`, `unsafe_convert`, `nextind`, `thisind`.

I’m experimenting with `struct JSONString{S <: AbstractString} j::S; i::Int` where `j` is the whole JSON text and `i` is the offset of a `"` delimited JSON string value. The intention is to do parsing of JSON escape sequences in-line in `codeunit` so that JSON string values can be used in-place without copying or un-escaping.

I’m hoping that this makes field name lookup string comparisons fast because most comparisons will fail early without ever unescaping the full string (the scanner sets a flag for strings with no escaping so that these can shortcut to a direct `memcmp`).

I’m also hoping for performance gains in the case where a large JSON text is opened, a few values are tweaked, and then it is saved again. In this case there is no need to ever un-escape or re-escape all the unchanged values.

[https://github.com/samoconnor/jsonhack/blob/master/src/AbstractString.jl#L27-L39](https://github.com/samoconnor/jsonhack/blob/master/src/AbstractString.jl#L27-L39)

---

<div class="post-metadata">

**Author:** ![samoconnor](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/samoconnor/32/1802_2.png) [@samoconnor](https://discourse.julialang.org/u/samoconnor)\
**Post date:** [February 9, 2018, 11:48pm UTC](https://discourse.julialang.org/t/what-is-the-interface-of-abstractstring/8937/5 "2018-02-09T23:48:34Z")

</div>

Reading in a bit more detail I see that…

> In order for `isvalid(s, i)` to be an O(1) function, the encoding of `s` must be [self-synchronizing](https://en.wikipedia.org/wiki/Self-synchronizing_code) this is a basic assumption of Julia’s generic string support.

This mightn’t be doable in general for JSON strings, which can have escape sequences up to 6 characters long: `\uXXXX`. But I’m hopeful that I can still come up with something efficient for sequential access in strings with occasional escape sequences.

---

<div class="post-metadata">

**Author:** ![StefanKarpinski](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stefankarpinski/32/24_2.png) [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)\
**Post date:** [February 10, 2018, 12:11am UTC](https://discourse.julialang.org/t/what-is-the-interface-of-abstractstring/8937/6 "2018-02-10T00:11:14Z")

</div>

6 is O(1)

---

<div class="post-metadata">

**Author:** ![samoconnor](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/samoconnor/32/1802_2.png) [@samoconnor](https://discourse.julialang.org/u/samoconnor)\
**Post date:** [February 10, 2018, 12:19am UTC](https://discourse.julialang.org/t/what-is-the-interface-of-abstractstring/8937/7 "2018-02-10T00:19:39Z")

</div>

True. The wiki page about self-synchronizing codes talks about “the symbol stream formed by a portion of one code word, is not a valid code word”, which seems not to be the case for JSON escaping. `"["\uD834\uDd1e"]`. If my index points to the second `''` I have to backtrack to the first `'\'` to see that it is not a valid index. But this is still a constant amount of work.

---

<div class="post-metadata">

**Author:** ![StefanKarpinski](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stefankarpinski/32/24_2.png) [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)\
**Post date:** [February 10, 2018, 12:48am UTC](https://discourse.julialang.org/t/what-is-the-interface-of-abstractstring/8937/8 "2018-02-10T00:48:48Z")

</div>

I plan on documenting this more thoroughly if I can ever get the Pkg3 stuff merged and working on master… it’s a real beast to change all of code loading and package resolution and installation. But it’s getting there.

---

<div class="post-metadata">

**Author:** ![samoconnor](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/samoconnor/32/1802_2.png) [@samoconnor](https://discourse.julialang.org/u/samoconnor)\
**Post date:** [February 10, 2018, 5:21am UTC](https://discourse.julialang.org/t/what-is-the-interface-of-abstractstring/8937/9 "2018-02-10T05:21:16Z")

</div>

I have `JSON.String <: AbstractString` mostly working now.

I’m seeing something unexpected. I have instrumented `next` to print out its inputs and outputs.  
When i compare a `JSON.String` to a `Base.String` with no common prefix, I expect to see `next` called just once. However, it seems that `cmp(a::AbstractString, b::AbstractString)` calls `next` one more time than it needs to:

```julia
julia> using LazyJSON
julia> json = """\"Foo\\u1234Bar\""""
julia> j = LazyJSON.parse(json)
julia> typeof(j)
LazyJSON.String{String}
julia> j.s
"\"Foo\\u1234Bar\""
julia> j.i
1

julia> j == "XYZ"
next(::JSON.String, 2) -> F, 2
next(::JSON.String, 3) -> o, 3
false

julia> j == "Food"
next(::JSON.String, 2) -> F, 2
next(::JSON.String, 3) -> o, 3
next(::JSON.String, 4) -> o, 4
next(::JSON.String, 10) -> ሴ, 10
next(::JSON.String, 11) -> B, 11
false

julia> j == "FooሴBar"
next(::JSON.String, 2) -> F, 2
next(::JSON.String, 3) -> o, 3
next(::JSON.String, 4) -> o, 4
next(::JSON.String, 10) -> ሴ, 10
next(::JSON.String, 11) -> B, 11
next(::JSON.String, 12) -> a, 12
next(::JSON.String, 13) -> r, 13
true

```

---

<div class="post-metadata">

**Author:** ![samoconnor](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/samoconnor/32/1802_2.png) [@samoconnor](https://discourse.julialang.org/u/samoconnor)\
**Post date:** [February 10, 2018, 6:16am UTC](https://discourse.julialang.org/t/what-is-the-interface-of-abstractstring/8937/10 "2018-02-10T06:16:49Z")

</div>

Example with escape sequences in field names, 992 bytes allocated vs 37k for JSON.jl:

````julia
julia> json = """
           {
               "Image\\t Tab": {
                   "Width": 800,
                   "Height": 600,
                   "Title": "View from 15th Floor",
                   "Thumb\\nail": {
                       "\\ud83e\\udd16 Url": "http://www.example.com/image/481989943",
                       "Height": 125,
                       "Width": 100
                   },
                   "Animated" : false,
                   "IDs": [116, 943, 234, 38793]
               }
           }
       """
julia> function f(json)
           v = LazyJSON.parse(json)
           v["Image\t Tab"]["Thumb\nail"]["🤖 Url"]
       end

julia> @time f(json)
  0.000023 seconds (28 allocations: 992 bytes)
"http://www.example.com/image/481989943"```

````

vs JSON.jl

```julia
julia> function f(json)
           v = JSON.parse(json)
           v["Image\t Tab"]["Thumb\nail"]["🤖 Url"]
       end
julia> @time f(json)
  0.000554 seconds (294 allocations: 36.844 KiB)
"http://www.example.com/image/481989943"

```

Example without escapes, only 8 allocations:

```julia
julia> function f(json)
           v = LazyJSON.parse(json)
           v["Image"]["Thumbnail"]["Url"]
       end

julia> @time f(json)
  0.000006 seconds (8 allocations: 288 bytes)
"http://www.example.com/image/481989943"

```

---

<div class="post-metadata">

**Author:** ![samoconnor](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/samoconnor/32/1802_2.png) [@samoconnor](https://discourse.julialang.org/u/samoconnor)\
**Post date:** [February 10, 2018, 6:22am UTC](https://discourse.julialang.org/t/what-is-the-interface-of-abstractstring/8937/11 "2018-02-10T06:22:59Z")

</div>

But, here is the fun part…

With a 1MB input file, the lazy `AbstractString` parser takes 2% of the time and 384 bytes vs 7MB.

```julia
$ du -sh ec2-2016-11-15.normal.json
1012K	ec2-2016-11-15.normal.json

```

```julia
j = String(read("ec2-2016-11-15.normal.json"))
julia> function f(json)
           v = JSON.parse(json)
           v["shapes"]["scope"]["enum"][1]
       end
julia> @time f(j)
  0.066773 seconds (66.43 k allocations: 7.087 MiB)
"Availability Zone"

```

```julia
julia> function f(json)
           v = LazyJSON.parse(json)
           v["shapes"]["scope"]["enum"][1]
       end
julia> @time f(j)
  0.001392 seconds (12 allocations: 384 bytes)
"Availability Zone"

```

---

<div class="post-metadata">

**Author:** ![ScottPJones](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/scottpjones/32/146_2.png) [@ScottPJones](https://discourse.julialang.org/u/ScottPJones)\
**Post date:** [February 10, 2018, 2:37pm UTC](https://discourse.julialang.org/t/what-is-the-interface-of-abstractstring/8937/12 "2018-02-10T14:37:41Z")

</div>

I’m just wrapping up some rather severe rewriting of the parser in `JSON.jl` (I started it before I saw Jacob Quinn’s `JSON2.jl` and before you posted your `LazyJSON.jl`).  
I’m hoping that maybe we can combine some of our efforts, and have a really kick-\*ss JSON parser for Julia! 😀

---

<div class="post-metadata">

**Author:** ![ScottPJones](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/scottpjones/32/146_2.png) [@ScottPJones](https://discourse.julialang.org/u/ScottPJones)\
**Post date:** [February 10, 2018, 4:49pm UTC](https://discourse.julialang.org/t/what-is-the-interface-of-abstractstring/8937/13 "2018-02-10T16:49:58Z")

</div>

> [@samoconnor](#):
>
> What is the minimal set of methods that must be defined for a custom AbstractString type?
> 
> Is there a good example of a custom string type somewhere

I would like to think that this is probably the most complete custom string type: [GitHub - JuliaString/Strs.jl: String support package for Julia](https://github.com/JuliaString/Strs.jl), it is meant to provide complete replacements for both the `String` and `Char` types.

When I am done, I’ll try to document what string functionality needed to be implemented, both for a minimal set, and for a complete replacement.

---

<div class="post-metadata">

**Author:** ![samoconnor](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/samoconnor/32/1802_2.png) [@samoconnor](https://discourse.julialang.org/u/samoconnor)\
**Post date:** [February 11, 2018, 5:44am UTC](https://discourse.julialang.org/t/what-is-the-interface-of-abstractstring/8937/14 "2018-02-11T05:44:05Z")

</div>

I’ve cleaned up my jsonhack repo and added some documentation and moved it to here: [https://github.com/samoconnor/LazyJSON.jl](https://github.com/samoconnor/LazyJSON.jl)

---

<div class="post-metadata">

**Author:** ![samoconnor](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/samoconnor/32/1802_2.png) [@samoconnor](https://discourse.julialang.org/u/samoconnor)\
**Post date:** [February 21, 2018, 12:24pm UTC](https://discourse.julialang.org/t/what-is-the-interface-of-abstractstring/8937/15 "2018-02-21T12:24:58Z")

</div>

I’ve recently implemented another `AbstractString`. This time backed by an `IOBuffer`. I noticed a few rough edges:

- The `Base.thisind(::String, i)` function just calls `Base._thisind_str(s, i)` (note: no type on `s`). This is conveniant because I can implement `Base.thisind(ms::MyString, i)` as `Base._thisind_str(ms, i)`. However, it would be easier if the `Base.thisind(::String, i)` was changed to `Base.thisind(::AbstractString, i)`, then I wouldn’t need a specialisation at all.

- The `Base.next(s::String, i::Int)` is not just a wrapper for `_next`. So I’ve had to copy/paste it and `Base.next_continued(s::String, ...)` into `MyStrings.jl`. It would be nice if `Base.next(::String, ...)` was widened to `Base.next(s::AbstractString, ...)`

If `thisind` and `next` were tweaked like this, then the custom `AbstractString` would be simply:

```julia
Base.codeunit(s::IOString) = UInt8
Base.ncodeunits(s::IOString) = s.buf.size
Base.codeunit(s::IOString, i::Integer) = s.buf.data[i]
Base.pointer(s::IOString) = pointer(s.buf.data)
Base.pointer(s::IOString, i) = pointer(s.buf.data, i)

Base.convert(::Type{Base.String}, s::IOString) =
    unsafe_string(pointer(s), ncodeunits(s))

```

Reference: [https://github.com/samoconnor/LazyJSON.jl/blob/master/src/IOStrings.jl](https://github.com/samoconnor/LazyJSON.jl/blob/master/src/IOStrings.jl)

---

<div class="post-metadata">

**Author:** ![nalimilan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nalimilan/32/147_2.png) [@nalimilan](https://discourse.julialang.org/u/nalimilan)\
**Post date:** [February 21, 2018, 12:50pm UTC](https://discourse.julialang.org/t/what-is-the-interface-of-abstractstring/8937/16 "2018-02-21T12:50:25Z")

</div>

Your proposal relies on the assumption that all `AbstractString` types use UTF-8, but that’s not part of the interface.

---

<div class="post-metadata">

**Author:** ![samoconnor](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/samoconnor/32/1802_2.png) [@samoconnor](https://discourse.julialang.org/u/samoconnor)\
**Post date:** [September 28, 2018, 4:36am UTC](https://discourse.julialang.org/t/what-is-the-interface-of-abstractstring/8937/17 "2018-09-28T04:36:15Z")

</div>

I have an `SubString`-like `AbstractString` subtype where it would be convenient for the indexes to be the same as an underlying string; i.e. `isvalid(s, 1) == false` and `firstindex(s) != 1` ([LazyHTTP.jl](https://github.com/JuliaWeb/HTTP.jl/blob/so/lazy/src/LazyHTTP.jl)).

I would like to know: Is it valid for `firstindex` to return \> 1 for an `AbstractString` subtype? @StefanKarpinski ? @nalimilan ?

The `AbstractString` spec hints at sub-string types that use the same indexes as an underlying string:  
[https://github.com/JuliaLang/julia/blob/master/base/strings/basic.jl#L31-L32](https://github.com/JuliaLang/julia/blob/master/base/strings/basic.jl#L31-L32)

But, the definition of `first(keys(::AbstractString))` implies that `isvalid(::AbstractString, 1)` is always `true`:  
[https://github.com/JuliaLang/julia/blob/master/base/strings/basic.jl#L536-L538](https://github.com/JuliaLang/julia/blob/master/base/strings/basic.jl#L536-L538)  
However, the `iterate` method above uses `firstindex` to set the start `state`, which suggests that it might be ok to define `firstindex(::MyStringSubtype) != 1` and `isvalid(::MyStringSubtype, 1) == false`

[https://github.com/JuliaLang/julia/blob/master/base/strings/basic.jl#L158-L158](https://github.com/JuliaLang/julia/blob/master/base/strings/basic.jl#L158-L158)

Are the places that assume `isvalid(s, 1) == true` bugs? (i.e. hard-coded `1` instead of calling `firstindex(s)`)  
Or is `isvalid(s, 1) == true` intended to be an invariant property of the `AbstractString` interface?

---

<div class="post-metadata">

**Author:** ![nalimilan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nalimilan/32/147_2.png) [@nalimilan](https://discourse.julialang.org/u/nalimilan)\
**Post date:** [September 28, 2018, 7:05am UTC](https://discourse.julialang.org/t/what-is-the-interface-of-abstractstring/8937/18 "2018-09-28T07:05:09Z")

</div>

AFAIK `AbstractString` implementations are supposed to have indices starting at `1`. You’re free to use any value for the iteration state, though. (The excerpt of the docs you quote refers to what happens when passing out of bounds indices to `nextind`, which is a different issue.)

---

<div class="post-metadata">

**Author:** ![samoconnor](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/samoconnor/32/1802_2.png) [@samoconnor](https://discourse.julialang.org/u/samoconnor)\
**Post date:** [September 28, 2018, 8:35am UTC](https://discourse.julialang.org/t/what-is-the-interface-of-abstractstring/8937/19 "2018-09-28T08:35:17Z")

</div>

Thanks @nalimilan.  
Regarding using non index values for iteration, I can see at least one place where it is assumed that an index is a valid iteration state:  
[https://github.com/JuliaLang/julia/blob/master/base/strings/basic.jl#L161-L164](https://github.com/JuliaLang/julia/blob/master/base/strings/basic.jl#L161-L164)

---

<div class="post-metadata">

**Author:** ![nalimilan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nalimilan/32/147_2.png) [@nalimilan](https://discourse.julialang.org/u/nalimilan)\
**Post date:** [September 28, 2018, 8:44am UTC](https://discourse.julialang.org/t/what-is-the-interface-of-abstractstring/8937/20 "2018-09-28T08:44:46Z")

</div>

Good catch. Though it can be considered as a reasonable and convenient fallback, as one can easily override `getindex` if needed.

[Next page](https://discourse.julialang.org/t/what-is-the-interface-of-abstractstring/8937.md?page=2)