# CSV.read: why do String columns show up as PooledArrays?

**URL:** <https://discourse.julialang.org/t/csv-read-why-do-string-columns-show-up-as-pooledarrays/30501>\
**Category:** New to Julia\
**Tags:** question\
**Created:** [October 30, 2019, 3:39pm UTC](https://discourse.julialang.org/t/csv-read-why-do-string-columns-show-up-as-pooledarrays/30501 "2019-10-30T15:39:04Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Sam\_Johnson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sam_johnson/32/10955_2.png) [@Sam\_Johnson](https://discourse.julialang.org/u/Sam_Johnson)\
**Post date:** [October 30, 2019, 3:39pm UTC](https://discourse.julialang.org/t/csv-read-why-do-string-columns-show-up-as-pooledarrays/30501/1 "2019-10-30T15:39:04Z")

</div>

I’m opening a simple .CSV file with CSV.read with no parameter modifiers. The returned DataFrame is fine and describe(df) gives me the correct element types, which are all Int64 or Strings.

When I try to onehot-encode the String attributes the function fails because the String attributes are not Strings inside the function. When I examine the attribute with typeof(df.prev\_campaign\_outcome) I get:

PooledArrays.PooledArray{String,UInt32,1,Array{UInt32,1}}

How has Julia made this decision? Why does describe() return the eltype as String but internally has converted the attribute to PooledArray? How can I force Julia to leave well enough alone?

Why does it seem Julia is trying too hard?

---

<div class="post-metadata">

**Author:** ![ExpandingMan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/expandingman/32/866_2.png) [@ExpandingMan](https://discourse.julialang.org/u/ExpandingMan)\
**Post date:** [October 30, 2019, 3:56pm UTC](https://discourse.julialang.org/t/csv-read-why-do-string-columns-show-up-as-pooledarrays/30501/2 "2019-10-30T15:56:28Z")

</div>

First of all, it’s not Julia, it’s CSV.jl.

Second, I’m not sure what you mean by

> [@Sam\_Johnson](#):
>
> When I try to onehot-encode the String attributes the function fails because the String attributes are not Strings inside the function.

Indeed, `PooledArrays.PooledArray{String,UInt32,1,Array{UInt32,1}}` should have an `eltype` of `String` which you should be able to confirm. It ought to work just like any other `AbstractVector{String}`. Could you be more specific about the problem it’s causing?

As to why CSV.jl constructs columns like this, it is common for tables to have string-valued columns with a number of possible values much smaller than the number of rows. In such cases, it’s much more efficient to use a `PooledArray`, which is sort of like a simple dict.

---

<div class="post-metadata">

**Author:** ![Sam\_Johnson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sam_johnson/32/10955_2.png) [@Sam\_Johnson](https://discourse.julialang.org/u/Sam_Johnson)\
**Post date:** [October 30, 2019, 4:22pm UTC](https://discourse.julialang.org/t/csv-read-why-do-string-columns-show-up-as-pooledarrays/30501/3 "2019-10-30T16:22:19Z")

</div>

Here’s the onehot-encoder function I wrote. It’s now failing by saying the return value “onehot” is not defined.

```julia
function onehotenc(df::DataFrame)

    # Loop to develop onehot-encoding
    for col in eachcol(df)
        
        # Can we determine if the column is a String and only encode it if it is?
        if col isa String
            
            # How long is the current column we're going to onehot-encode?
            len = length(col)

            # Save the unique values (or Set) and how many there are for our initial zeros matrix
            vals = unique(col)

            # Set up the Dict key-value pairs to save the Boolean operation results
            # For each val in vals make that val a key (:key) with 
            # and fill that column with Boolean falses to begin with
            dict = Dict(Symbol(val) => falses(len) for val in vals)

            # Once the Dict is populated convert it to a DataFrame
            onehot = DataFrame(dict)

            # Change the value from 0 to 1 in the zero DataFrame based on whether the unique value matches the original DF
            for (i, v) in enumerate(col)
                # At the row and col of the current enumerated col value, set the value to true
                # EX: In the onehot DF at row 1, column :A set equal to true
                # EX: In the onehot DF at row 2, column :B set equal to true
                onehot[i, Symbol(v)] = true
            end
        end
    end
    
    return onehot
    
end
```

---

<div class="post-metadata">

**Author:** ![davidanthoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidanthoff/32/223493_2.png) [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)\
**Post date:** [October 30, 2019, 4:35pm UTC](https://discourse.julialang.org/t/csv-read-why-do-string-columns-show-up-as-pooledarrays/30501/4 "2019-10-30T16:35:44Z")

</div>

You’ll want to change `if col isa String` to `if eltype(col) isa String`, or even better, `if eltype(col) isa AbstractString`.

The type of `col` will never be `String` because it is a vector of `String`s, so you need to test the element type of that vector.

---

<div class="post-metadata">

**Author:** ![Sam\_Johnson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sam_johnson/32/10955_2.png) [@Sam\_Johnson](https://discourse.julialang.org/u/Sam_Johnson)\
**Post date:** [October 30, 2019, 4:48pm UTC](https://discourse.julialang.org/t/csv-read-why-do-string-columns-show-up-as-pooledarrays/30501/5 "2019-10-30T16:48:09Z")

</div>

Thanks, David. I blundered that comparison.

Any thoughts on why the function returns “UndefVarError: onehot not defined”?

---

<div class="post-metadata">

**Author:** ![davidanthoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidanthoff/32/223493_2.png) [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)\
**Post date:** [October 30, 2019, 4:58pm UTC](https://discourse.julialang.org/t/csv-read-why-do-string-columns-show-up-as-pooledarrays/30501/6 "2019-10-30T16:58:47Z")

</div>

Take a look at the [scoping](https://docs.julialang.org/en/v1.2/manual/variables-and-scoping/) part of the documentation. You are defining the `onehot` variable inside a loop, and then it won’t be visible outside of the loop.

---

<div class="post-metadata">

**Author:** ![mbauman](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mbauman/32/31082_2.png) [@mbauman](https://discourse.julialang.org/u/mbauman)\
**Post date:** [October 30, 2019, 5:15pm UTC](https://discourse.julialang.org/t/csv-read-why-do-string-columns-show-up-as-pooledarrays/30501/7 "2019-10-30T17:15:04Z")

</div>

Even if that worked with our scope rules, it’d still be a bug. You’re generating a new dictionary for each and every column of strings, clobbering the last one you created. It’d probably make sense to just specify which column you want to one-hot encode as a second argument to the function.

You may also be interested in the fact that there are one hot encoders already written:

[https://fluxml.ai/Flux.jl/stable/data/onehot/](https://fluxml.ai/Flux.jl/stable/data/onehot/)