# String compression on windows
**URL:** https://discourse.julialang.org/t/string-compression-on-windows/69585
**Category:** General Usage
**Tags:** question, strings, json
**Created:** [October 11, 2021, 8:05pm UTC](https://discourse.julialang.org/t/string-compression-on-windows/69585 "2021-10-11T20:05:44Z")
**Posts on this page:** 18
**Page:** 1
### Author:  [@Impressium](https://discourse.julialang.org/u/Impressium)
#### Post date: [October 11, 2021, 8:05pm UTC](https://discourse.julialang.org/t/string-compression-on-windows/69585/1 "2021-10-11T20:05:44Z")
Hi all 🙂
I’m forced to use windows. Is there any string compression pkg that can be used on windows? (like [https://github.com/ararslan/Shoco.jl](https://github.com/ararslan/Shoco.jl))
Thank you!
---
### Author:  [@giordano](https://discourse.julialang.org/u/giordano)
#### Post date: [October 11, 2021, 8:10pm UTC](https://discourse.julialang.org/t/string-compression-on-windows/69585/2 "2021-10-11T20:10:55Z")
@ararslan what does
> The Shoco C library does not work on Windows due to lack of C99 support, which means that this package has the same restriction.
mean? 🤔
---
### Author:  [@gbaraldi](https://discourse.julialang.org/u/gbaraldi)
#### Post date: [October 11, 2021, 8:57pm UTC](https://discourse.julialang.org/t/string-compression-on-windows/69585/3 "2021-10-11T20:57:40Z")
Looks like both the library and the package aren’t mantained, [https://github.com/siara-cc/Unishox](https://github.com/siara-cc/Unishox). This looks like a decent substitute, it’s pure C and it works with unicode, I’m gonna try a local binarybuilder build to write a wrapper, the api seems to be 2 functions
---
### Author:  [@Impressium](https://discourse.julialang.org/u/Impressium)
#### Post date: [October 11, 2021, 9:03pm UTC](https://discourse.julialang.org/t/string-compression-on-windows/69585/4 "2021-10-11T21:03:18Z")
That sounds awesome.
---
### Author:  [@Impressium](https://discourse.julialang.org/u/Impressium)
#### Post date: [October 11, 2021, 9:04pm UTC](https://discourse.julialang.org/t/string-compression-on-windows/69585/5 "2021-10-11T21:04:44Z")
Maybe I have to describe my use case. I have a really large JSON string that I want to compress.
---
### Author:  [@Oscar\_Smith](https://discourse.julialang.org/u/Oscar_Smith)
#### Post date: [October 11, 2021, 9:09pm UTC](https://discourse.julialang.org/t/string-compression-on-windows/69585/6 "2021-10-11T21:09:12Z")
Rather than compression json, you might want to consider using a more efficient format in the first place. Binary data formats are likely going to be smaller and faster than compressed json.
---
### Author:  [@gbaraldi](https://discourse.julialang.org/u/gbaraldi)
#### Post date: [October 11, 2021, 9:09pm UTC](https://discourse.julialang.org/t/string-compression-on-windows/69585/7 "2021-10-11T21:09:24Z")
In that case [https://github.com/JuliaIO/TranscodingStreams.jl](https://github.com/JuliaIO/TranscodingStreams.jl) is probably your best bet, shoco and apparently unishox, are for short strings. Also what Oscar said
---
### Author:  [@Impressium](https://discourse.julialang.org/u/Impressium)
#### Post date: [October 11, 2021, 10:21pm UTC](https://discourse.julialang.org/t/string-compression-on-windows/69585/8 "2021-10-11T22:21:50Z")
I know but I can’t decide that…
---
### Author:  [@ararslan](https://discourse.julialang.org/u/ararslan)
#### Post date: [October 19, 2021, 4:58pm UTC](https://discourse.julialang.org/t/string-compression-on-windows/69585/9 "2021-10-19T16:58:41Z")
I… don’t entirely remember. Best I recall, Shoco doesn’t support Windows because it relies on C99 features that aren’t available on Windows, though maybe that’s just a limitation of MSVC (I don’t know) and maybe support for a 22 year old standard has since been added. I’ve not thought about Shoco in a long time. 😅
---
### Author:  [@Palli](https://discourse.julialang.org/u/Palli)
#### Post date: [October 19, 2021, 5:41pm UTC](https://discourse.julialang.org/t/string-compression-on-windows/69585/10 "2021-10-19T17:41:35Z")
I don’t know of better than Shoco, that I knew of and Unishox I just learned of, except for my own _idea_. I intend to implement it in Julia, basically it’s coding bigraphs (or tigraphs), not single letters, for compressing two letters into a byte (or by a third, 3 into 1 byte, or more likely 6 into 2), giving direct index (most often) and faster sorting.
If you see such a library please let me know, then I don’t have to do it. Or if you want to implement yourself or collaborate.
I see Portuguese compresses the most down to 40% (except for Chinese and Japanese), a bit more than my bigram idea (while losing direct indexing). Do you need to support mostly one specific language, or few, then which? My bigram idea is almost trivial to code, trigram more involved, do you really need down to 33%?
I have lots of memory compression ideas that have higher priority, e.g. for numbers. I choose the priority based on what excites me most at the time. If you want direct indexing into a string then UTF-8 is already a problem, TranscodingStreams.jl would also be a problem, and it seems also the two short-string compression methods already mentioned.
Binary formats are more compact (for e.g. numbers), but I believe they (most of them at least) use UTF-8 still uncompressed, for the text parts in JSON.
---
### Author:  [@ararslan](https://discourse.julialang.org/u/ararslan)
#### Post date: [October 21, 2021, 11:01pm UTC](https://discourse.julialang.org/t/string-compression-on-windows/69585/11 "2021-10-21T23:01:44Z")
> [@Impressium](#):
>
> (like [GitHub - ararslan/Shoco.jl: Julia wrapper for the shoco string compression C library](https://github.com/ararslan/Shoco.jl))
FYI @Impressium, I’ve updated Shoco to work with Julia 1.6 and it now supports Windows. If you end up using it, please open issues if you run into any problems!
---
### Author:  [@gbaraldi](https://discourse.julialang.org/u/gbaraldi)
#### Post date: [November 3, 2021, 9:34pm UTC](https://discourse.julialang.org/t/string-compression-on-windows/69585/12 "2021-11-03T21:34:01Z")
I’ve just released [https://github.com/gbaraldi/Unishox.jl](https://github.com/gbaraldi/Unishox.jl) which is the exact same thing except it looks like Unishox is getting more development than Shoco. I have no idea about differences in performance.
---
### Author:  [@SortofDamocles](https://discourse.julialang.org/u/SortofDamocles)
#### Post date: [February 10, 2022, 6:38pm UTC](https://discourse.julialang.org/t/string-compression-on-windows/69585/13 "2022-02-10T18:38:21Z")
There appears to be a bug in Unishox. I have a rather nasty string `S` (at the end of the post). The first time I tried to use Unishox to compress it, I got:
```julia
julia> Unishox.decompress(Unishox.compress(S))
ERROR: BoundsError: attempt to access 1218-element Vector{Int8} at index [1:2519]
Stacktrace:
[1] throw_boundserror(A::Vector{Int8}, I::Tuple{UnitRange{Int64}})
@ Base .\abstractarray.jl:651
[2] checkbounds
@ .\abstractarray.jl:616 [inlined]
[3] getindex
@ .\array.jl:807 [inlined]
[4] decompress(s::String)
@ Unishox C:\Users\rbuckalew\.julia\packages\Unishox\dahtN\src\Unishox.jl:40
[5] top-level scope
@ REPL[35]:1
```
After a restart, the next time I tried the same commands but my Julia session simply crashed.
Here is the string that led to this behavior:
```julia
julia> S
"[[[[[[[[[[[-3]]]]], [[[[[-2]]]]]], [[[[[[-6]]]]], [[[[[-5]]]]]]], [[[[[[[-3]]]]], [[[[[-2]]]]]], [[[[[[-3]]]]], [[[[[-2]]]]]]], [[[[[[[4]]]]], [[[[[4]]]]]], [[[[[[7]]]]], [[[[[4]]]]]]]], [[[[[[[[-3]]]]], [[[[[0]]]]]], [[[[[[-3]]]]], [[[[[-1]]]]]]], [[[[[[[-6]]]]], [[[[[0]]]]]], [[[[[[-6]]]]], [[[[[-4]]]]]]], [[[[[[[-7]]]]], [[[[[-4]]]]]], [[[[[[1]]]]], [[[[[-2]]]]]]]], [[[[[[[[2]]]]], [[[[[5]]]]]], [[[[[[2]]]]], [[[[[1]]]]]]], [[[[[[[-1]]]]], [[[[[2]]]]]], [[[[[[-1]]]]], [[[[[-2]]]]]]], [[[[[[[-2]]]]], [[[[[-2]]]]]], [[[[[[0]]]]], [[[[[0]]]]]]]]], [[[[[[[[[1]]]]], [[[[[2]]]]]], [[[[[[1]]]]], [[[[[2]]]]]]], [[[[[[[-1]]]]]], [[[[[[-1]]]]]]]], [[[[[[[[-3]]]]], [[[[[4]]]]]], [[[[[[-1]]]]], [[[[[-2]]]]]]], [[[[[[[1]]]]]], [[[[[[-3]]]]]]]], [[[[[[[[-2]]]]], [[[[[5]]]]]], [[[[[[0]]]]], [[[[[-1]]]]]]], [[[[[[[-2]]]]]], [[[[[[-6]]]]]]]]], [[[[[[[[[-1]]]]], [[[[[0]]]]]], [[[[[[2]]]]], [[[[[0]]]]]]], [[[[[[[-3]]]]]], [[[[[[-6]]]]]]]], [[[[[[[[-6]]]]], [[[[[4]]]]]], [[[[[[2]]]]], [[[[[-2]]]]]]], [[[[[[[1]]]]]], [[[[[[0]]]]]]]], [[[[[[[[-3]]]]], [[[[[4]]]]]], [[[[[[-1]]]]], [[[[[-2]]]]]]], [[[[[[[-2]]]]]], [[[[[[-3]]]]]]]]]], [[[[[[[[[[1]]]]], [[[[[2]]]]]], [[[[[[-2]]]]], [[[[[3]]]]]]], [[[[[[[1]]]]], [[[[[2]]]]]], [[[[[[1]]]]], [[[[[2]]]]]]], [[[[[[[2]]]]], [[[[[2]]]]]], [[[[[[5]]]]], [[[[[2]]]]]]]], [[[[[[[[3]]]]], [[[[[4]]]]]], [[[[[[3]]]]], [[[[[3]]]]]]], [[[[[[[-2]]]]], [[[[[2]]]]]], [[[[[[-2]]]]], [[[[[2]]]]]]], [[[[[[[-1]]]]], [[[[[2]]]]]], [[[[[[-1]]]]], [[[[[0]]]]]]]], [[[[[[[[0]]]]], [[[[[1]]]]]], [[[[[[4]]]]], [[[[[5]]]]]]], [[[[[[[1]]]]], [[[[[2]]]]]], [[[[[[1]]]]], [[[[[2]]]]]]], [[[[[[[2]]]]], [[[[[2]]]]]], [[[[[[0]]]]], [[[[[0]]]]]]]]], [[[[[[[[[-1]]]]], [[[[[0]]]]]], [[[[[[-1]]]]], [[[[[0]]]]]]], [[[[[[[-1]]]]]], [[[[[[-1]]]]]]]], [[[[[[[[1]]]]], [[[[[2]]]]]], [[[[[[1]]]]], [[[[[2]]]]]]], [[[[[[[-1]]]]]], [[[[[[-1]]]]]]]], [[[[[[[[-4]]]]], [[[[[-3]]]]]], [[[[[[-4]]]]], [[[[[-3]]]]]]], [[[[[[[-4]]]]]], [[[[[[-4]]]]]]]]], [[[[[[[[[-3]]]]], [[[[[-2]]]]]], [[[[[[0]]]]], [[[[[-2]]]]]]], [[[[[[[-3]]]]]], [[[[[[-2]]]]]]]], [[[[[[[[-4]]]]], [[[[[0]]]]]], [[[[[[-2]]]]], [[[[[0]]]]]]], [[[[[[[-1]]]]]], [[[[[[-2]]]]]]]], [[[[[[[[-1]]]]], [[[[[0]]]]]], [[[[[[-1]]]]], [[[[[0]]]]]]], [[[[[[[0]]]]]], [[[[[[-1]]]]]]]]]], [[[[[[[[[[-3]]]]],
[[[[[0]]]]]], [[[[[[-6]]]]], [[[[[-3]]]]]]], [[[[[[[-6]]]]], [[[[[0]]]]]], [[[[[[-6]]]]], [[[[[-4]]]]]]], [[[[[[[-7]]]]], [[[[[-4]]]]]], [[[[[[1]]]]], [[[[[-2]]]]]]]], [[[[[[[[-1]]]]], [[[[[0]]]]]], [[[[[[-4]]]]], [[[[[-3]]]]]]], [[[[[[[-6]]]]], [[[[[-2]]]]]], [[[[[[-6]]]]], [[[[[-2]]]]]]], [[[[[[[1]]]]], [[[[[4]]]]]], [[[[[[1]]]]], [[[[[2]]]]]]]], [[[[[[[[-2]]]]], [[[[[-3]]]]]], [[[[[[2]]]]], [[[[[5]]]]]]], [[[[[[[-1]]]]], [[[[[-2]]]]]], [[[[[[-1]]]]], [[[[[2]]]]]]], [[[[[[[0]]]]], [[[[[0]]]]]], [[[[[[-4]]]]], [[[[[-4]]]]]]]]], [[[[[[[[[-3]]]]], [[[[[4]]]]]], [[[[[[-1]]]]], [[[[[-2]]]]]]], [[[[[[[1]]]]]], [[[[[[-3]]]]]]]], [[[[[[[[3]]]]], [[[[[4]]]]]], [[[[[[3]]]]], [[[[[4]]]]]]], [[[[[[[-1]]]]]], [[[[[[-1]]]]]]]], [[[[[[[[-4]]]]], [[[[[-5]]]]]], [[[[[[-2]]]]], [[[[[5]]]]]]], [[[[[[[-5]]]]]], [[[[[[-1]]]]]]]]], [[[[[[[[[-6]]]]], [[[[[4]]]]]], [[[[[[2]]]]], [[[[[-2]]]]]]], [[[[[[[1]]]]]], [[[[[[-2]]]]]]]], [[[[[[[[-2]]]]], [[[[[2]]]]]], [[[[[[0]]]]], [[[[[2]]]]]]], [[[[[[[-1]]]]]], [[[[[[-4]]]]]]]], [[[[[[[[-1]]]]], [[[[[-2]]]]]], [[[[[[-3]]]]], [[[[[4]]]]]]], [[[[[[[-2]]]]]], [[[[[[-2]]]]]]]]]], [[[[[[[[[[-1]]]]], [[[[[2]]]]]], [[[[[[-1]]]]], [[[[[-2]]]]]]], [[[[[[[-1]]]]], [[[[[2]]]]]], [[[[[[-1]]]]], [[[[[-2]]]]]]], [[[[[[[-2]]]]], [[[[[-2]]]]]], [[[[[[0]]]]], [[[[[0]]]]]]]], [[[[[[[[-3]]]]], [[[[[-2]]]]]], [[[[[[-3]]]]], [[[[[-2]]]]]]], [[[[[[[-3]]]]], [[[[[-2]]]]]], [[[[[[-3]]]]], [[[[[-2]]]]]]], [[[[[[[4]]]]], [[[[[4]]]]]], [[[[[[2]]]]], [[[[[2]]]]]]]], [[[[[[[[-1]]]]], [[[[[-2]]]]]], [[[[[[-1]]]]], [[[[[2]]]]]]], [[[[[[[-1]]]]], [[[[[-2]]]]]], [[[[[[-1]]]]], [[[[[2]]]]]]], [[[[[[[0]]]]], [[[[[0]]]]]], [[[[[[-4]]]]], [[[[[-4]]]]]]]]], [[[[[[[[[-2]]]]], [[[[[5]]]]]], [[[[[[0]]]]], [[[[[-1]]]]]]], [[[[[[[-2]]]]]], [[[[[[-6]]]]]]]], [[[[[[[[-2]]]]], [[[[[-1]]]]]], [[[[[[-2]]]]], [[[[[-1]]]]]]], [[[[[[[-4]]]]]], [[[[[[-4]]]]]]]], [[[[[[[[-3]]]]], [[[[[-4]]]]]], [[[[[[-5]]]]], [[[[[2]]]]]]], [[[[[[[-5]]]]]], [[[[[[-1]]]]]]]]], [[[[[[[[[-3]]]]], [[[[[4]]]]]], [[[[[[-1]]]]], [[[[[-2]]]]]]], [[[[[[[1]]]]]], [[[[[[-3]]]]]]]], [[[[[[[[1]]]]], [[[[[2]]]]]], [[[[[[1]]]]], [[[[[2]]]]]]], [[[[[[[-1]]]]]], [[[[[[-1]]]]]]]], [[[[[[[[-1]]]]], [[[[[-2]]]]]], [[[[[[-3]]]]], [[[[[4]]]]]]], [[[[[[[-3]]]]]], [[[[[[1]]]]]]]]]]]"
```
---
### Author:  [@gbaraldi](https://discourse.julialang.org/u/gbaraldi)
#### Post date: [February 11, 2022, 1:28am UTC](https://discourse.julialang.org/t/string-compression-on-windows/69585/14 "2022-02-11T01:28:30Z")
That’s a string among strings, I will take a look when able to see where the problem might be.
---
### Author:  [@Palli](https://discourse.julialang.org/u/Palli)
#### Post date: [February 11, 2022, 10:00am UTC](https://discourse.julialang.org/t/string-compression-on-windows/69585/15 "2022-02-11T10:00:22Z")
I see Unishox (i.e. Unishox2) has been updated to 1.0.2, and Julia’s wrapper seems to use 1.0.0. In the meantime some bugs were fixed on Sep 25, 2021.
The github link to Unishox now redirects to Unishox2 but I see the former in Yggrasil. Does that mean it was built for Unishox, not Unishox2, that may or may not (but I find likely) to be incompatible?
---
### Author:  [@Palli](https://discourse.julialang.org/u/Palli)
#### Post date: [February 11, 2022, 10:18am UTC](https://discourse.julialang.org/t/string-compression-on-windows/69585/16 "2022-02-11T10:18:33Z")
> [@gbaraldi](#):
>
> looks like Unishox is getting more development than Shoco. I have no idea about differences in performance.
Self-admitted in the docs (section 9.5):
> Unishox was found to be the slowest of all since employs several [methods?] to achieve the
> best compression. However this should not be too much of an issue in most
> cases when a single string or few strings are handled at a time.
but it compresses best, compared to the (non-Unicode) Shoco (and Smaz, there in all cases except 1, with Smaz down to 58 bytes vs 60), it’s also much better than (Unicode-handling) SCSU and BOCU (one of, or both patented).
My concern (and I’ve sent a question on it to the author), is that it seems to me it requires validated UTF-8, i.e. I’m not sure it handled invalid UTF-8/arbitrary bytestrings.
---
### Author:  [@gbaraldi](https://discourse.julialang.org/u/gbaraldi)
#### Post date: [February 11, 2022, 10:26am UTC](https://discourse.julialang.org/t/string-compression-on-windows/69585/17 "2022-02-11T10:26:35Z")
I haven’t looked at it for a little while, the package is very simple, but if there was no API breakage it should just be a case of updating the Yggdrasil build script
---
### Author:  [@gbaraldi](https://discourse.julialang.org/u/gbaraldi)
#### Post date: [February 11, 2022, 11:03am UTC](https://discourse.julialang.org/t/string-compression-on-windows/69585/18 "2022-02-11T11:03:38Z")
Some quick tests show that it was the lib that was returning the wrong number of bytes of the uncompressed string even though the part it returns is correctly decompressed. I am updating the jll to see if that fixes it, otherwise I will open an issue upstream 🙂 .
edit:
I found the issue, the string returned by Unishox after compression had a NUL character in the middle of, which made julia cut it off at the middle, I’m not sure if that was expected or not, but it’s easy to fix on my part. Will tag a release with a fix .