# String pool / indirect array analogue to make a struct isbits?

**URL:** <https://discourse.julialang.org/t/string-pool-indirect-array-analogue-to-make-a-struct-isbits/43503>\
**Category:** Performance\
**Created:** [July 22, 2020, 4:31pm UTC](https://discourse.julialang.org/t/string-pool-indirect-array-analogue-to-make-a-struct-isbits/43503 "2020-07-22T16:31:39Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![evanfields](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/evanfields/32/1744_2.png) [@evanfields](https://discourse.julialang.org/u/evanfields)\
**Post date:** [July 22, 2020, 4:31pm UTC](https://discourse.julialang.org/t/string-pool-indirect-array-analogue-to-make-a-struct-isbits/43503/1 "2020-07-22T16:31:39Z")

</div>

I have a program which ingests JSON describe a list of objects. For the sake of example, let’s say each object is a Cartesian point with a name:

```julia
struct NamedPoint
    x::Float64
    y::Float64
    name::String
end

```

The actual use case is more complicated and not relevant here; the relevant feature is the pattern of bits fields like `Float64` + a `String`. In practice, for data with millions of input points, there are only ~10 distinct names. So it seems a bit of a waste to have every `NamedPoint` contain a `String` - and thus not be `isbits`. Instead, each `NamedPoint` should be able to just contain a small integer indicating which name it has.

This seems intuitively similar to the idea behind `PooledArrays` and `IndirectArrays`. But I don’t want an indirect or pooled array of just names; the code uses (and is way more readable with) `NamedPoint` objects.

My thought now is to just collect the unique names seen when parsing a JSON and build a dict `UInt8 => name` at parse time. Then each `NamedPoint` can have a `UInt8` instead of a `String` name, and will be `isbits`. Does a package implementing this better than “roll your own” already exist? Is there a different recommended solution? I don’t love this mapping dictionary approach, since the mapping has to be passed around to helper functions and a `NamedPoint` is no longer meaningful without the associated name mapping dictionary.

I welcome any advice on how to maximize performance here while keeping the code well readable and maintainable. Thanks!

---

<div class="post-metadata">

**Author:** ![CameronBieganek](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cameronbieganek/32/6915_2.png) [@CameronBieganek](https://discourse.julialang.org/u/CameronBieganek)\
**Post date:** [July 22, 2020, 5:06pm UTC](https://discourse.julialang.org/t/string-pool-indirect-array-analogue-to-make-a-struct-isbits/43503/2 "2020-07-22T17:06:13Z")

</div>

Do you know the set of possible names beforehand? If so, you could use an `Enum`, since they’re `isbits`. Under the hood, the `@enum` macro actually defines a primitive type.

If you don’t know the set of possible names beforehand, perhaps you could repurpose the `Base.@enum` code so that it can accept a vector of strings, so you could do something like the following:

```julia
names = ["A", "B"]
@myenum names

```

---

<div class="post-metadata">

**Author:** ![evanfields](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/evanfields/32/1744_2.png) [@evanfields](https://discourse.julialang.org/u/evanfields)\
**Post date:** [July 22, 2020, 5:12pm UTC](https://discourse.julialang.org/t/string-pool-indirect-array-analogue-to-make-a-struct-isbits/43503/3 "2020-07-22T17:12:35Z")

</div>

Good question; I should have put this in the top post:

- Each input JSON to be processed has ~10-20 names
- The set of possible names is not known in advance, i.e. any JSON _could_ contain a never-before-seen name.
- However, the set of possible names is fairly small, maybe ~100 elements.

I’m a bit confused about the `@myenum` definition for unknown names. That’d dynamically define a new type each time a new JSON is processed, right? (I should clarify that for this use case, a long-running (hopefully) Julia server accepts many JSON requests.)

---

<div class="post-metadata">

**Author:** ![CameronBieganek](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cameronbieganek/32/6915_2.png) [@CameronBieganek](https://discourse.julialang.org/u/CameronBieganek)\
**Post date:** [July 22, 2020, 5:27pm UTC](https://discourse.julialang.org/t/string-pool-indirect-array-analogue-to-make-a-struct-isbits/43503/4 "2020-07-22T17:27:09Z")

</div>

Ah, ok. I was thinking about processing just one big JSON. Yes, that would create a new type each time you call `@myenum`. Although I suppose I got the syntax above wrong. It would also need to include a name for the type:

```julia
@myenum typename names

```

Interestingly, it turns out that when you define an `Enum`, it adds a `namemap` method inside the `Base.Enums` module:

```julia
julia> @enum Fruit apple orange kiwi

julia> methods(Base.Enums.namemap)
# 31 methods for generic function "namemap":
[1] namemap(::Type{Base.MPFR.MPFRRoundingMode}) in Base.MPFR at Enums.jl:189
[2] namemap(::Type{LibGit2.Consts.OBJECT}) in LibGit2.Consts at Enums.jl:189
# Entries 3 - 30 omitted for brevity.
[31] namemap(::Type{Fruit}) in Main at Enums.jl:189

julia> Base.Enums.namemap(Fruit)
Dict{Int32,Symbol} with 3 entries:
  0 => :apple
  2 => :kiwi
  1 => :orange

```

So there’s still a dictionary involved, it’s just hidden away in a module.

---

<div class="post-metadata">

**Author:** ![CameronBieganek](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cameronbieganek/32/6915_2.png) [@CameronBieganek](https://discourse.julialang.org/u/CameronBieganek)\
**Post date:** [July 22, 2020, 6:30pm UTC](https://discourse.julialang.org/t/string-pool-indirect-array-analogue-to-make-a-struct-isbits/43503/5 "2020-07-22T18:30:42Z")

</div>

Upon further reflection, my `@enum` suggestion probably wouldn’t be very useful anyways, since `@enum` has to be used at top-level:

```julia
julia> function bar()
           @enum Fruit apple orange kiwi
       end
ERROR: syntax: "toplevel" expression not at top level

```

And the same would probably be true for any macro that is creating new types.

---

<div class="post-metadata">

**Author:** ![tomerarnon](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tomerarnon/32/3170_2.png) [@tomerarnon](https://discourse.julialang.org/u/tomerarnon)\
**Post date:** [July 23, 2020, 10:09pm UTC](https://discourse.julialang.org/t/string-pool-indirect-array-analogue-to-make-a-struct-isbits/43503/6 "2020-07-23T22:09:30Z")

</div>

Perhaps this is exactly what you don’t want, but here’s a mapping approach anyway. While it’s true that a NamedPoint name isn’t meaningful without the mapping, overwriting `getproperty` makes this distinction completely invisible I would say. The only thing I would be careful of is if saving these to a file, to make sure any code uses `getproperty` rather than explicitly `getfield`.

```julia
const NAME_TO_INT = Dict{String, Int}()
const INT_TO_NAME = Vector{String}()

# wherever you read from the json:
x, y, s = #one line of the json
if !haskey(NAME_TO_INT, s)
    NAME_TO_INT[s] = length(NAME_TO_INT)+1
    push!(INT_TO_NAME, s)
end

p = NamedPoint(x, y, NAME_TO_INT[s])    

# to map back, either define an interface
name(p::NamedPoint) = INT_TO_NAME[p.s] 

# or just overwrite getproperty
Base.getproperty(p::NamedPoint, s::Symbol) = s === :name ? INT_TO_NAME[p.s] : getfield(p, s)

```