# Disable SentinelArrays for CSV.read

**URL:** <https://discourse.julialang.org/t/disable-sentinelarrays-for-csv-read/54843>\
**Category:** General Usage\
**Tags:** csv\
**Created:** [February 8, 2021, 2:14pm UTC](https://discourse.julialang.org/t/disable-sentinelarrays-for-csv-read/54843 "2021-02-08T14:14:39Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![robsmith11](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/robsmith11/32/29641_2.png) [@robsmith11](https://discourse.julialang.org/u/robsmith11)\
**Post date:** [February 8, 2021, 2:14pm UTC](https://discourse.julialang.org/t/disable-sentinelarrays-for-csv-read/54843/1 "2021-02-08T14:14:39Z")

</div>

I’m reading a large CSV file of many Float64 values and then applying many possible transformations before doing a least squares fit.

I noticed that the type is not plain Float64s despite there being no missing values:

```julia
julia> d = CSV.read("/data/m4.csv", NamedTuple)
julia> typeof(d)
NamedTuple{(:hu, :hc, :hf, :hs, :bu, :bs, :cs), NTuple{7, SentinelArrays.ChainedVector{Float64, Vector{Float64}}}}

```

If I manually convert to plain Float64s (btw is there an easier way to do this?), I get a 28% speed up in my transformations code:

```julia
julia> typeof((bs=d.bs[:], bu=d.bu[:], cs=d.cs[:], hc=d.hc[:], hf=d.hf[:], hs=d.hs[:], hu=d.hu[:]))
NamedTuple{(:bs, :bu, :cs, :hc, :hf, :hs, :hu), NTuple{7, Vector{Float64}}}

```

Is there any way to tell `CSV.read` to not use SentinelArrays so I don’t have to do the conversion myself?

EDIT:  
As a workaround, I’m using this:

```julia
d = JuliaDB.loadtable("/data/m4.csv").columns.columns

```

---

<div class="post-metadata">

**Author:** ![dereksz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dereksz/32/26307_2.png) [@dereksz](https://discourse.julialang.org/u/dereksz)\
**Post date:** [June 23, 2021, 8:45am UTC](https://discourse.julialang.org/t/disable-sentinelarrays-for-csv-read/54843/2 "2021-06-23T08:45:14Z")

</div>

I hit the same issue, and have developed a very efficient - but also highly fragile - solution.

If you look to the layout of SentinelArray, there is a `data` attribute that holds the underlying vector, so I simply do `fd[!,col] = fd[!,col].data`. However, if your CSV is running multithreaded, you actually end up with a set of SentinelArray packaged within a ChainedArray. To address both situations it’s easiest to use functions multiple-dispatch to make sure the “de\_sentinelization” happens correctly:

```julia
# _de_sentinelise digs into Sentinel & Chained Vectors and "pulls up" the Vector under the Sentinel
function _de_sentinelise(v::AbstractVector)
    # @info "In AbstractVector with type $(typeof(v))"
    return v
end

function _de_sentinelise(v::SentinelVector{T}) where {T<:Base.AbstractFloat}
    # @info "In SentinelArray with type $(typeof(v))"
    return v.data
end

function _de_sentinelise(v::ChainedVector)
    # @info "In ChainedVector with type $(typeof(v))"
    a = v.arrays
    (eltype(a) <: SentinelVector) || return v
    n = Vector{typeof(a[1].data)}(undef, length(a));
    for i in 1:length(n)
        #= @inbounds =# n[i] = _de_sentinelise(a[i]) # premature optimisation - take the safety instead!
    end;
    return ChainedVector(n)
end

function de_sentinelise!(df::DataFrame)
    for (i,col) in enumerate(eachcol(df))
        df[!,i] = _de_sentinelise(col)
    end
    return nothing
end

function de_sentinelise!(df::DataFrame, cols)
    for col in cols
        df[!,col] = _de_sentinelise(df[!,col])
    end
    return nothing
end

de_sentinelise!(df)

```

---

<div class="post-metadata">

**Author:** ![jondea](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jondea/32/39086_2.png) [@jondea](https://discourse.julialang.org/u/jondea)\
**Post date:** [January 21, 2022, 10:30am UTC](https://discourse.julialang.org/t/disable-sentinelarrays-for-csv-read/54843/3 "2022-01-21T10:30:15Z")

</div>

I recently came across the same problem, thank you for the solution @dereksz

However, I managed to simplify it by using `collect` instead

```julia
function de_sentinelise!(df::DataFrame)
    for (i,col) in enumerate(eachcol(df))
        df[!,i] = collect(col)
    end
    return df
end

```

---

<div class="post-metadata">

**Author:** ![sijo](https://avatars.discourse-cdn.com/v4/letter/s/da6949/32.png) [@sijo](https://discourse.julialang.org/u/sijo)\
**Post date:** [January 21, 2022, 1:29pm UTC](https://discourse.julialang.org/t/disable-sentinelarrays-for-csv-read/54843/4 "2022-01-21T13:29:43Z")

</div>

I think this is equivalent to `mapcols!(collect, df)` ?

But I couldn’t find a simple way to make one of these data frames with SentinalArrays columns. It would be nice to have a MWE to produce one.

---

<div class="post-metadata">

**Author:** ![jondea](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jondea/32/39086_2.png) [@jondea](https://discourse.julialang.org/u/jondea)\
**Post date:** [January 21, 2022, 4:50pm UTC](https://discourse.julialang.org/t/disable-sentinelarrays-for-csv-read/54843/5 "2022-01-21T16:50:45Z")

</div>

I’m not at my computer to check, but that sounds right, and very elegant!

Unfortunately I can’t share the dataset sorry, but it happened for me when the CSV got suitably large (1000s of rows). The columns became chained sentinel arrays, and GLM complained.

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [January 21, 2022, 5:40pm UTC](https://discourse.julialang.org/t/disable-sentinelarrays-for-csv-read/54843/6 "2022-01-21T17:40:56Z")

</div>

> [@jondea](#):
>
> The columns became chained sentinel arrays, and GLM complained.

This will be fixed soon.

In general if you want to avoid `SentinelVector` (which often makes sense from my experience) set `CSV.read` to use a single thread when reading the data in (it will then be a bit slower but usually acceptably fast).

---

<div class="post-metadata">

**Author:** ![jondea](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jondea/32/39086_2.png) [@jondea](https://discourse.julialang.org/u/jondea)\
**Post date:** [January 21, 2022, 7:06pm UTC](https://discourse.julialang.org/t/disable-sentinelarrays-for-csv-read/54843/7 "2022-01-21T19:06:31Z")

</div>

Oh brilliant, thank you 😄

---

<div class="post-metadata">

**Author:** ![andreasnoack](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/andreasnoack/32/27_2.png) [@andreasnoack](https://discourse.julialang.org/u/andreasnoack)\
**Post date:** [June 1, 2023, 12:52pm UTC](https://discourse.julialang.org/t/disable-sentinelarrays-for-csv-read/54843/8 "2023-06-01T12:52:08Z")

</div>

> [@bkamins](#):
>
> This will be fixed soon.

Was this fixed? If so, I’m curious how it was fixed.

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [June 1, 2023, 1:53pm UTC](https://discourse.julialang.org/t/disable-sentinelarrays-for-csv-read/54843/9 "2023-06-01T13:53:21Z")

</div>

AFAICT It was fixed by @andreasnoack in [https://github.com/JuliaStats/GLM.jl/pull/446](https://github.com/JuliaStats/GLM.jl/pull/446) (and related PRs) 😃, but I have not tested this now.