# \[ANN\]: WIP Strs.jl package ready for alpha review and testing

**URL:** <https://discourse.julialang.org/t/ann-wip-strs-jl-package-ready-for-alpha-review-and-testing/8087>\
**Category:** Community\
**Created:** [January 1, 2018, 5:14am UTC](https://discourse.julialang.org/t/ann-wip-strs-jl-package-ready-for-alpha-review-and-testing/8087 "2018-01-01T05:14:12Z")\
**Posts on this page:** 16\
**Page:** 1

<div class="post-metadata">

**Author:** ![ScottPJones](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/scottpjones/32/146_2.png) [@ScottPJones](https://discourse.julialang.org/u/ScottPJones)\
**Post date:** [January 1, 2018, 5:14am UTC](https://discourse.julialang.org/t/ann-wip-strs-jl-package-ready-for-alpha-review-and-testing/8087/1 "2018-01-01T05:14:12Z")

</div>

I’d like to announce the availability for review and comment of my (very WIP!) Strs.jl package,

Please check it out, it’s still rather rough, a lot of code hasn’t been optimized, more needs to be made to use traits instead of having things hard coded, many of my ideas for this I haven’t even started to implement (optional substrings, cached hash values, cached UTF-8 or UTF-16 or raw bytes)

[https://github.com/JuliaString/Strs.jl](https://github.com/JuliaString/Strs.jl)

(I still need to make the logo I want for the JuliaString org 🙂 3 concentric circles, of the “Julia” colors, with ASCII/Latin1 text in the other, BMP (probably math, Japanese, maybe Hindo stuff), and finally non-BMP (a few emojis) in the center).

Thanks, and Happy New Year!

---

<div class="post-metadata">

**Author:** ![xiaodai](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xiaodai/32/15937_2.png) [@xiaodai](https://discourse.julialang.org/u/xiaodai)\
**Post date:** [January 1, 2018, 5:18am UTC](https://discourse.julialang.org/t/ann-wip-strs-jl-package-ready-for-alpha-review-and-testing/8087/2 "2018-01-01T05:18:24Z")

</div>

Should faster string sorts got to JuliaString or SortingAlgorithms.jl?

---

<div class="post-metadata">

**Author:** ![kevin.squire](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kevin.squire/32/62_2.png) [@kevin.squire](https://discourse.julialang.org/u/kevin.squire)\
**Post date:** [January 1, 2018, 7:03am UTC](https://discourse.julialang.org/t/ann-wip-strs-jl-package-ready-for-alpha-review-and-testing/8087/3 "2018-01-01T07:03:30Z")

</div>

I would vote for SortingAlgorithms. Scott, correct me if I’m wrong, but it seems that Strs.jl is meant to be used in lieu of strings in Base. Ideally, a radix-based string sort would be written generally enough that it would work for any type of string, including strings in Base and any additional String types defined in packages (including Strs.jl).

Of course, the basic idea of radix sort sorting using the binary/numerical representation of characters probably conflicts with the notion of most non-ascii character encodings, especially UTF8…

Cheers,  
Kevin

---

<div class="post-metadata">

**Author:** ![xiaodai](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xiaodai/32/15937_2.png) [@xiaodai](https://discourse.julialang.org/u/xiaodai)\
**Post date:** [January 1, 2018, 7:24am UTC](https://discourse.julialang.org/t/ann-wip-strs-jl-package-ready-for-alpha-review-and-testing/8087/4 "2018-01-01T07:24:09Z")

</div>

Yeah radixsort is useful for grouping. Actually grouping is my primary use case so at some point my attention will switch to radixgroup for fast group by operations.

I think given one of the goals of Strs.jl is O(1) access to characters (bytes would be ok too). I think the sorting method for Strs.jl might need to be specialised for Strs.jl. Not sure yet.

---

<div class="post-metadata">

**Author:** ![ScottPJones](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/scottpjones/32/146_2.png) [@ScottPJones](https://discourse.julialang.org/u/ScottPJones)\
**Post date:** [January 1, 2018, 11:52am UTC](https://discourse.julialang.org/t/ann-wip-strs-jl-package-ready-for-alpha-review-and-testing/8087/5 "2018-01-01T11:52:30Z")

</div>

The [https://github.com/JuliaString](https://github.com/JuliaString) organization is meant to hold packages for better string support in Julia, just as there are organizations for bioinformatics ([BioJulia · GitHub](https://github.com/BioJulia)) and plotting in Julia ([https://github.com/JuliaPlots](https://github.com/JuliaPlots)).

[https://github.com/JuliaCollections/SortingAlgorithms.jl](https://github.com/JuliaCollections/SortingAlgorithms.jl) is a nice package in the [https://github.com/JuliaCollections](https://github.com/JuliaCollections) organization, which has a number of other interesting packages as well.  
You could either submit a PR to add your sorting code to the `SortingAlgorithms.jl` package, or I imagine as a separate package to `JuliaCollections` (whatever the owners etc. of the org prefer)

The other packages in `JuliaString` (such as [GitHub - JuliaString/StringLiterals.jl: Implement improved string literals with Swift-style syntax for interpolation, hex, & unicode characters, plus C & Python style formatting and Unicode, HTML, LaTeX, and Emoji entities](https://github.com/JuliaString/StringLiterals.jl), [https://github.com/JuliaString/ICU.jl](https://github.com/JuliaString/ICU.jl), and [https://github.com/JuliaString/Format.jl](https://github.com/JuliaString/Format.jl)) do not (currently!) use my new `Strs.jl` package, they are meant to be useful as much as possible no matter what `AbstractString` type you are using.

Your faster radix sorting for strings would fit in well (I think in either `JuliaCollections` or `JuliaString`, TBH), even if you don’t add optimizations for specific `Str` types.  
I’d love to see fast sorting for the “direct indexed” string types in `Strs.jl`, such as `BinaryStr`, or any of the ones currently implemented (except for `UTF8Str` and `UTF16Str`, those are multi codeunit encodings of Unicode).

If you wish to submit a (MIT licensed) package for faster string sorting to `JuliaString`, that would be great.

---

<div class="post-metadata">

**Author:** ![ScottPJones](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/scottpjones/32/146_2.png) [@ScottPJones](https://discourse.julialang.org/u/ScottPJones)\
**Post date:** [January 1, 2018, 11:59am UTC](https://discourse.julialang.org/t/ann-wip-strs-jl-package-ready-for-alpha-review-and-testing/8087/6 "2018-01-01T11:59:01Z")

</div>

> [@kevin.squire](#):
>
> Of course, the basic idea of radix sort sorting using the binary/numerical representation of characters probably conflicts with the notion of most non-ascii character encodings, especially UTF8…

I don’t think you’d want to be doing sorting so much on the encoded forms, but rather on the code points, so only multi- codeunit encodings, such as `UTF-8` (i.e. `String`, `LegacyStrings.UTF8String`, or `UTF8Str`), or `UTF-16` (`LegacyStrings.UTF16String` or `UTF16Str`), would be an issue, and I think you would still just do the sort on the 8-bit or 16-bit code units (the result for UTF-8 should be the same as if you sorted by UTF-32, anyway).

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [January 1, 2018, 1:57pm UTC](https://discourse.julialang.org/t/ann-wip-strs-jl-package-ready-for-alpha-review-and-testing/8087/7 "2018-01-01T13:57:20Z")

</div>

> [@ScottPJones](#):
>
> and I think you would still just do the sort on the 8-bit or 16-bit code units

For UTF-8, sorting on code points and code units gives the same result, but for UTF-16, it does not; [some small transformations are required](https://ssl.icu-project.org/docs/papers/utf16_code_point_order.html).

---

<div class="post-metadata">

**Author:** ![ScottPJones](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/scottpjones/32/146_2.png) [@ScottPJones](https://discourse.julialang.org/u/ScottPJones)\
**Post date:** [January 1, 2018, 2:37pm UTC](https://discourse.julialang.org/t/ann-wip-strs-jl-package-ready-for-alpha-review-and-testing/8087/8 "2018-01-01T14:37:15Z")

</div>

I’m well aware of that, but actually some software requires sorting by the UTF-16 code points (usually for compatibility reasons because software originally written for Unicode 1.x sorts them by the 16-bit values, and indices in databases are in that order). I knew how to do a Unicode 2.0 compatible sort on UTF-16 long before that [https://ssl.icu-project.org/docs/papers/utf16\_code\_point\_order.html](https://ssl.icu-project.org/docs/papers/utf16_code_point_order.html) came out - it’s fairly obvious, but for most purposes, if you want to sort for human use, you should follow the recommendations and use locale specific collation sequences, after normalization.

Having both sorting with the transformations (which, IMO, should be the default for directly dealing with UTF-16 encoded data) and without (i.e. compatible also with CESU-8 sorting order) is useful (but is the sort of thing that I’d put in a `StrExtensions.jl` package - I don’t think direct operations on any multi-code unit representations should really be in the base `Strs.jl` package, probably only simple conversions, but I’ll sort that out later.

---

<div class="post-metadata">

**Author:** ![ScottPJones](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/scottpjones/32/146_2.png) [@ScottPJones](https://discourse.julialang.org/u/ScottPJones)\
**Post date:** [January 1, 2018, 2:45pm UTC](https://discourse.julialang.org/t/ann-wip-strs-jl-package-ready-for-alpha-review-and-testing/8087/9 "2018-01-01T14:45:30Z")

</div>

BTW, I do hope you can take the time to thoroughly review the code, and would appreciate any suggestions.

---

<div class="post-metadata">

**Author:** ![ScottPJones](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/scottpjones/32/146_2.png) [@ScottPJones](https://discourse.julialang.org/u/ScottPJones)\
**Post date:** [January 2, 2018, 11:15am UTC](https://discourse.julialang.org/t/ann-wip-strs-jl-package-ready-for-alpha-review-and-testing/8087/10 "2018-01-02T11:15:37Z")

</div>

@Tamas_Papp I just saw your [https://github.com/tpapp/ByteParsers.jl](https://github.com/tpapp/ByteParsers.jl) package (while investigating your [GitHub - tpapp/julia-repl: Run an inferior Julia REPL in a terminal inside Emacs](https://github.com/tpapp/julia-repl) package, which as an inveterate Emacs user, is of great interest to me! 🤓) - I think it could benefit from using `Strs.jl` instead of (or in addition to) `Vector{UInt8}`.

I see that your code currently makes the string be converted to a `Vector{UInt8}`, (which may or may not do any real work, depending on the type of the `AbstractString`). That would not be necessary with my `Str` types.  
You would also not be limited to ASCII characters for the delimiter, something that came up recently (see: [Unicode related error when reading a .csv - #3 by IljaK91](https://discourse.julialang.org/t/unicode-related-error-when-reading-a-csv/7132/3)) with somebody trying to use `CSV.jl`, because a Python program exported strings with Unicode quotes, i.e.:

```julia
‘“’: Unicode U+201c (category Pi: Punctuation, initial quote)
’”’: Unicode U+201d (category Pf: Punctuation, final quote)

```

You could use any of the `CodeUnitSingle` `Str` types (currently all of them except for `UTF8Str` and `UTF16Str`), such as `UniStr`, `BinaryStr`, `RawByteStr`, `RawWordStr`, `RawCharStr`, etc.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [January 2, 2018, 11:19am UTC](https://discourse.julialang.org/t/ann-wip-strs-jl-package-ready-for-alpha-review-and-testing/8087/11 "2018-01-02T11:19:12Z")

</div>

Thanks for the suggestion. I meant `ByteParsers.jl` as temporary measure, and strictly for ASCII data (fortunately, most of the social science data I get is plain vanilla ASCII). But I will keep an eye on your package and revisit the issue once `v0.7` is out.

---

<div class="post-metadata">

**Author:** ![ScottPJones](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/scottpjones/32/146_2.png) [@ScottPJones](https://discourse.julialang.org/u/ScottPJones)\
**Post date:** [January 11, 2018, 1:07pm UTC](https://discourse.julialang.org/t/ann-wip-strs-jl-package-ready-for-alpha-review-and-testing/8087/12 "2018-01-11T13:07:57Z")

</div>

Just merged in a bunch of fixes, now passes `runtests` 😀

@jeff.bezanson Even though this is still WIP, please take a look!

Green means that it is more than 5% faster than `String`, the number is the ratio of time for `String` divided by time for that `Str` encoding for that test.

Also I have some early benchmarking results, on a few functions!

 ![Telugu](https://global.discourse-cdn.com/julialang/original/3X/2/8/2845dcb4223ade607829c85af7b69aafe72bfee2.png)  
 ![Hungarian](https://global.discourse-cdn.com/julialang/original/3X/2/d/2df2d058d4104a41f7f0807901534b93695f455c.png)  
 ![Japanese](https://global.discourse-cdn.com/julialang/original/3X/5/c/5cc4631cd6a98e7bb0f2d9b945979ce63246a2e5.png)  
 ![Russian](https://global.discourse-cdn.com/julialang/original/3X/f/9/f9f304dd9b0f69f8d802d1a27b5dbd0d1a7a2a11.png)  
 ![Swedish](https://global.discourse-cdn.com/julialang/original/3X/f/7/f7c1abca58d8887475951e5b5d2d6854981ac0ea.png)  
 ![ASCII-Dict](https://global.discourse-cdn.com/julialang/original/3X/e/e/ee933a7df510071516565b1df026205d0427ff85.png)  
 ![Chinese](https://global.discourse-cdn.com/julialang/original/3X/0/1/011ee064342e1b8fece9a0611de7e3e71d868c43.png)

---

<div class="post-metadata">

**Author:** ![amrods](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/amrods/32/2543_2.png) [@amrods](https://discourse.julialang.org/u/amrods)\
**Post date:** [January 11, 2018, 1:34pm UTC](https://discourse.julialang.org/t/ann-wip-strs-jl-package-ready-for-alpha-review-and-testing/8087/13 "2018-01-11T13:34:33Z")

</div>

I get the following error in 0.6.2

```julia
julia> using Strs
INFO: Precompiling module Strs.
WARNING: could not import Base.thisind into Strs
ERROR: LoadError: LoadError: UndefVarError: Nothing not defined
Stacktrace:
 [1] macro expansion at /Users/amrods/.julia/v0.6/Strs/src/types.jl:159 [inlined]
 [2] anonymous at ./<missing>:?
 [3] include_from_node1(::String) at /Applications/Julia-0.6.app/Contents/Resources/julia/lib/julia/sys.dylib:?
 [4] include(::String) at /Applications/Julia-0.6.app/Contents/Resources/julia/lib/julia/sys.dylib:?
 [5] include_from_node1(::String) at /Applications/Julia-0.6.app/Contents/Resources/julia/lib/julia/sys.dylib:?
 [6] include(::String) at /Applications/Julia-0.6.app/Contents/Resources/julia/lib/julia/sys.dylib:?
 [7] anonymous at ./<missing>:2
while loading /Users/amrods/.julia/v0.6/Strs/src/types.jl, in expression starting on line 156
while loading /Users/amrods/.julia/v0.6/Strs/src/Strs.jl, in expression starting on line 82
ERROR: Failed to precompile Strs to /Users/amrods/.julia/lib/v0.6/Strs.ji.
Stacktrace:
 [1] compilecache(::String) at /Applications/Julia-0.6.app/Contents/Resources/julia/lib/julia/sys.dylib:?
 [2] _require(::Symbol) at /Applications/Julia-0.6.app/Contents/Resources/julia/lib/julia/sys.dylib:?
 [3] require(::Symbol) at /Applications/Julia-0.6.app/Contents/Resources/julia/lib/julia/sys.dylib:?

```

---

<div class="post-metadata">

**Author:** ![ScottPJones](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/scottpjones/32/146_2.png) [@ScottPJones](https://discourse.julialang.org/u/ScottPJones)\
**Post date:** [January 11, 2018, 2:46pm UTC](https://discourse.julialang.org/t/ann-wip-strs-jl-package-ready-for-alpha-review-and-testing/8087/14 "2018-01-11T14:46:39Z")

</div>

I haven’t ported it to v0.6 yet, that’s planned, shortly

---

<div class="post-metadata">

**Author:** ![ScottPJones](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/scottpjones/32/146_2.png) [@ScottPJones](https://discourse.julialang.org/u/ScottPJones)\
**Post date:** [January 24, 2018, 5:37pm UTC](https://discourse.julialang.org/t/ann-wip-strs-jl-package-ready-for-alpha-review-and-testing/8087/15 "2018-01-24T17:37:03Z")

</div>

It is now working on v0.6.2 (and latest master!)

---

<div class="post-metadata">

**Author:** ![ScottPJones](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/scottpjones/32/146_2.png) [@ScottPJones](https://discourse.julialang.org/u/ScottPJones)\
**Post date:** [April 3, 2018, 12:58pm UTC](https://discourse.julialang.org/t/ann-wip-strs-jl-package-ready-for-alpha-review-and-testing/8087/16 "2018-04-03T12:58:49Z")

</div>

Just a heads up, the [Strs.jl](https://github.com/JuliaString/Strs.jl) package is now passing most all of the String unit tests on v0.6.2 (missing a few tests that were only for new methods in v0.7, such as length, nextind, prevind with extra parameters) and on latest v0.7, so it is no longer so “WIP”.

For most of the performance tests, it is uniformly faster than `String` (work still needs to be done for Regex,  
since the Base Regex support doesn’t support abstract strings, and would need to be extended to be able to handle 16-bit and 32-bit codeunits), frequently 2-10x, sometimes as much as 1000x faster in my testing  
(which are rather extensive, testing a large set of files, with String and various Str encodings, on 19 different string/character functions currently).

I would very much appreciate any additional benchmarks, so that `Str` and `Chr` can be compared to `String` and `Char` with a wide variety of use cases.