# Plots borking on SentinelArrays produced by CSV.read

**URL:** <https://discourse.julialang.org/t/plots-borking-on-sentinelarrays-produced-by-csv-read/89505>\
**Category:** General Usage\
**Tags:** csv, plots\
**Created:** [October 30, 2022, 12:11am UTC](https://discourse.julialang.org/t/plots-borking-on-sentinelarrays-produced-by-csv-read/89505 "2022-10-30T00:11:46Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![dlakelan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dlakelan/32/8491_2.png) [@dlakelan](https://discourse.julialang.org/u/dlakelan)\
**Post date:** [October 30, 2022, 12:11am UTC](https://discourse.julialang.org/t/plots-borking-on-sentinelarrays-produced-by-csv-read/89505/1 "2022-10-30T00:11:46Z")

</div>

```julia
using CSV, DataFrames, DataFramesMeta, Downloads, StatsPlots

Downloads.download("https://data.cdc.gov/api/views/g653-rqe2/rows.csv?accessType=DOWNLOAD","NWSSSarsCov2WastewaterConc.csv")
Downloads.download("https://data.cdc.gov/api/views/2ew6-ywp6/rows.csv?accessType=DOWNLOAD","NWSSSarsCov2WastewaterMetr.csv")

wwdat = CSV.read("NWSSSarsCov2WastewaterConc.csv",DataFrame)

@subset!(wwdat,.!ismissing.(:pcr_conc_smoothed) .&& .!ismissing.(:date))

wwdat.pcr_conc_smoothed = Float64.(wwdat.pcr_conc_smoothed)
scatter(wwdat.date,wwdat.pcr_conc_smoothed) ## fails.

```

This gives a weird error (when run in VSCode)

```julia
ERROR: MethodError: no method matching _cycle(::SentinelArrays.ChainedVector{Float64, Vector{Float64}}, ::SentinelArrays.ChainedVectorIndex{Vector{Float64}})
Closest candidates are:
  _cycle(::AbstractVector, ::Int64) at ~/.julia/packages/Plots/4UTBj/src/utils.jl:179
  _cycle(::AbstractVector, ::AbstractVector{Int64}) at ~/.julia/packages/Plots/4UTBj/src/utils.jl:183
  _cycle(::Any, ::AbstractVector{Int64}) at ~/.julia/packages/Plots/4UTBj/src/utils.jl:185
  ...
Stacktrace:
  [1] gr_draw_markers(series::Plots.Series, x::SentinelArrays.ChainedVector{Float64, Vector{Float64}}, y::SentinelArrays.ChainedVector{Float64, Vector{Float64}}, z::Nothing, clims::Tuple{Float64, Float64}, msize::Int64, strokewidth::Int64)
    @ Plots ~/.julia/packages/Plots/4UTBj/src/backends/gr.jl:1983
  [2] gr_draw_markers(series::Plots.Series, x::SentinelArrays.ChainedVector{Float64, Vector{Float64}}, y::SentinelArrays.ChainedVector{Float64, Vector{Float64}}, z::Nothing, clims::Tuple{Float64, Float64})
    @ Plots ~/.julia/packages/Plots/4UTBj/src/backends/gr.jl:1971
  [3] gr_add_series(sp::Plots.Subplot{Plots.GRBackend}, series::Plots.Series)
    @ Plots ~/.julia/packages/Plots/4UTBj/src/backends/gr.jl:1874
  [4] gr_display(sp::Plots.Subplot{Plots.GRBackend}, w::Measures.AbsoluteLength, h::Measures.AbsoluteLength, viewport_canvas::Vector{Float64})
    @ Plots ~/.julia/packages/Plots/4UTBj/src/backends/gr.jl:1033
  [5] (::Plots.var"#465#466"{Int64, Int64, Vector{Float64}})(sp::Plots.Subplot{Plots.GRBackend})
    @ Plots ~/.julia/packages/Plots/4UTBj/src/backends/gr.jl:699
  [6] foreach(f::Plots.var"#465#466"{Int64, Int64, Vector{Float64}}, itr::Vector{Plots.Subplot})
    @ Base ./abstractarray.jl:2774
  [7] gr_display(plt::Plots.Plot{Plots.GRBackend}, dpi_factor::Int64)
    @ Plots ~/.julia/packages/Plots/4UTBj/src/backends/gr.jl:699
  [8] #502
    @ ~/.julia/packages/Plots/4UTBj/src/backends/gr.jl:2177 [inlined]
  [9] withenv(::Plots.var"#502#503"{Plots.Plot{Plots.GRBackend}, Int64}, ::Pair{String, String}, ::Vararg{Pair{String, String}})
    @ Base ./env.jl:172
 [10] _show(io::IOBuffer, #unused#::MIME{Symbol("image/svg+xml")}, plt::Plots.Plot{Plots.GRBackend})
    @ Plots ~/.julia/packages/Plots/4UTBj/src/backends/gr.jl:2172
 [11] #invokelatest#2
    @ ./essentials.jl:729 [inlined]
 [12] invokelatest
    @ ./essentials.jl:726 [inlined]
 [13] show
    @ ~/.julia/packages/Plots/4UTBj/src/output.jl:237 [inlined]
 [14] __binrepr(m::MIME{Symbol("image/svg+xml")}, x::Plots.Plot{Plots.GRBackend}, context::Nothing)
    @ Base.Multimedia ./multimedia.jl:159
 [15] display(d::VSCodeServer.InlineDisplay, m::MIME{Symbol("image/svg+xml")}, x::Plots.Plot{Plots.GRBackend})
    @ VSCodeServer ./strings/string.jl:0
 [16] display(d::VSCodeServer.InlineDisplay, mime::String, x::Any)
    @ Base.Multimedia ./multimedia.jl:216
 [17] display(d::VSCodeServer.InlineDisplay, x::Plots.Plot{Plots.GRBackend})
    @ VSCodeServer ~/.vscode-oss/extensions/julialang.language-julia-1.38.2-universal/scripts/packages/VSCodeServer/src/display.jl:165
 [18] display(x::Any)
    @ Base.Multimedia ./multimedia.jl:328
 [19] #invokelatest#2
    @ ./essentials.jl:729 [inlined]
 [20] invokelatest
    @ ./essentials.jl:726 [inlined]
 [21] (::VSCodeServer.var"#66#70"{Bool, Bool, Bool, Module, String, Int64, Int64, String, VSCodeServer.ReplRunCodeRequestParams})()
    @ VSCodeServer ~/.vscode-oss/extensions/julialang.language-julia-1.38.2-universal/scripts/packages/VSCodeServer/src/eval.jl:199
 [22] withpath(f::VSCodeServer.var"#66#70"{Bool, Bool, Bool, Module, String, Int64, Int64, String, VSCodeServer.ReplRunCodeRequestParams}, path::String)
    @ VSCodeServer ~/.vscode-oss/extensions/julialang.language-julia-1.38.2-universal/scripts/packages/VSCodeServer/src/repl.jl:249
 [23] (::VSCodeServer.var"#65#69"{Bool, Bool, Bool, Module, String, Int64, Int64, String, VSCodeServer.ReplRunCodeRequestParams})()
    @ VSCodeServer ~/.vscode-oss/extensions/julialang.language-julia-1.38.2-universal/scripts/packages/VSCodeServer/src/eval.jl:155
 [24] hideprompt(f::VSCodeServer.var"#65#69"{Bool, Bool, Bool, Module, String, Int64, Int64, String, VSCodeServer.ReplRunCodeRequestParams})
    @ VSCodeServer ~/.vscode-oss/extensions/julialang.language-julia-1.38.2-universal/scripts/packages/VSCodeServer/src/repl.jl:38
 [25] (::VSCodeServer.var"#64#68"{Bool, Bool, Bool, Module, String, Int64, Int64, String, VSCodeServer.ReplRunCodeRequestParams})()
    @ VSCodeServer ~/.vscode-oss/extensions/julialang.language-julia-1.38.2-universal/scripts/packages/VSCodeServer/src/eval.jl:126
 [26] with_logstate(f::Function, logstate::Any)
    @ Base.CoreLogging ./logging.jl:511
 [27] with_logger
    @ ./logging.jl:623 [inlined]
 [28] (::VSCodeServer.var"#63#67"{VSCodeServer.ReplRunCodeRequestParams})()
    @ VSCodeServer ~/.vscode-oss/extensions/julialang.language-julia-1.38.2-universal/scripts/packages/VSCodeServer/src/eval.jl:225
 [29] #invokelatest#2
    @ ./essentials.jl:729 [inlined]
 [30] invokelatest(::Any)
    @ Base ./essentials.jl:726
 [31] macro expansion
    @ ~/.vscode-oss/extensions/julialang.language-julia-1.38.2-universal/scripts/packages/VSCodeServer/src/eval.jl:34 [inlined]
 [32] (::VSCodeServer.var"#61#62")()
    @ VSCodeServer ./task.jl:484

```

---

<div class="post-metadata">

**Author:** ![digital\_carver](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/digital_carver/32/33818_2.png) [@digital\_carver](https://discourse.julialang.org/u/digital_carver)\
**Post date:** [October 30, 2022, 6:10am UTC](https://discourse.julialang.org/t/plots-borking-on-sentinelarrays-produced-by-csv-read/89505/2 "2022-10-30T06:10:10Z")

</div>

I came across this post

> [@Disable SentinelArrays for CSV.read](https://discourse.julialang.org/t/disable-sentinelarrays-for-csv-read/54843/6):
>
> This will be fixed soon. In general if you want to avoid SentinelVector (which often makes sense from my experience) set CSV.read to use a single thread when reading the data in (it will then be a bit slower but usually acceptably fast).

which recommends reading the CSV single-threaded. That works:

```julia
julia> wwdat1 = CSV.read("NWSSSarsCov2WastewaterConc.csv",DataFrame; ntasks = 1)
335417×4 DataFrame
...

julia> wwdat1.key_plot_id |> typeof
       # now it's a normal Vector, not a SentinelArray type
Vector{String} (alias for Array{String, 1})

julia> scatter(wwdat1.date,wwdat1.pcr_conc_smoothed) ## works.

```

![Screenshot_20221030_113317](https://global.discourse-cdn.com/julialang/original/3X/e/f/ef8132ac93aac5cbab8f4eab84d041d107e8bde9.png)

The same thread also suggests `collect`ing the column before plotting, as another way of managing this, and that works too:

```julia
julia> scatter(collect(wwdat.date), collect(wwdat.pcr_conc_smoothed)) # works

```

But reading the CSV with `ntasks = 1` is a fixed, one-time (per session) cost, so that’s probably better than `collect`ing for every plot call.

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [October 30, 2022, 7:39am UTC](https://discourse.julialang.org/t/plots-borking-on-sentinelarrays-produced-by-csv-read/89505/3 "2022-10-30T07:39:12Z")

</div>

This is a current fix. However, the problem reported here is a problem on Plots.jl side.

x-ref [eachindex call incorrectly assumes that the returned index is IndexLinear by bkamins · Pull Request #4479 · JuliaPlots/Plots.jl · GitHub](https://github.com/JuliaPlots/Plots.jl/pull/4479)

---

<div class="post-metadata">

**Author:** ![t-bltg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/t-bltg/32/25526_2.png) [@t-bltg](https://discourse.julialang.org/u/t-bltg)\
**Post date:** [October 30, 2022, 9:00am UTC](https://discourse.julialang.org/t/plots-borking-on-sentinelarrays-produced-by-csv-read/89505/4 "2022-10-30T09:00:11Z")

</div>

Would it be possible to reduce the failing example to an offline one - with as few dependencies as possible - so that we can introduce it in `Plots` test suite for non-regression (in @bkamins 's PR) ?

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [October 30, 2022, 9:13am UTC](https://discourse.julialang.org/t/plots-borking-on-sentinelarrays-produced-by-csv-read/89505/5 "2022-10-30T09:13:56Z")

</div>

@t-bltg:  
Probably it is best to use OffsetArrays.jl (for non 1-based indexing case) and use:

```julia
using SentinelArrays
x = ChainedVector([[1, 2], [3, 4]])

```

(for the case mentioned in this thread).

---

<div class="post-metadata">

**Author:** ![dlakelan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dlakelan/32/8491_2.png) [@dlakelan](https://discourse.julialang.org/u/dlakelan)\
**Post date:** [October 30, 2022, 3:56pm UTC](https://discourse.julialang.org/t/plots-borking-on-sentinelarrays-produced-by-csv-read/89505/6 "2022-10-30T15:56:08Z")

</div>

Awesome that you guys figured this out and pushed a fix overnight. For the mean time I’m just dropmissing! The data and then constructing a vector to replace the columns in the data frame.

ie.

```julia
mydata.mycol = Vector{Float64}(mydata.mycol)

```

(Do it once after reading rather than every plot). Then I can still get fast threaded reading

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [October 30, 2022, 4:52pm UTC](https://discourse.julialang.org/t/plots-borking-on-sentinelarrays-produced-by-csv-read/89505/7 "2022-10-30T16:52:27Z")

</div>

You can replace all columns to be `Vector` in a data frame as follows:

```julia
df = copy(df)

```

or e.g.

```julia
mapcols!(copy, df)

```

or

```julia
select!(df, All() => copy, renamecols=false)

```

---

<div class="post-metadata">

**Author:** ![dlakelan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dlakelan/32/8491_2.png) [@dlakelan](https://discourse.julialang.org/u/dlakelan)\
**Post date:** [October 30, 2022, 5:05pm UTC](https://discourse.julialang.org/t/plots-borking-on-sentinelarrays-produced-by-csv-read/89505/8 "2022-10-30T17:05:30Z")

</div>

> [@bkamins](#):
>
> `df = copy(df)`

that works well, I tried the `select!(df,All() => copy, renamecols=false)` and it borks on no methods for sentinelarrays:

```julia
ERROR: MethodError: no method matching copy(::SentinelArrays.ChainedVector{String, Vector{String}}, ::SentinelArrays.ChainedVector{Date, Vector{Date}}, ::SentinelArrays.ChainedVector{Float64, Vector{Float64}}, ::PooledArrays.PooledVector{String, UInt32, Vector{UInt32}})
Closest candidates are:
  copy(::SentinelArrays.ChainedVector{T, A} where A<:AbstractVector{T}) where T at ~/.julia/packages/SentinelArrays/XvFr7/src/chainedvector.jl:464
  copy(::AbstractArray) at abstractarray.jl:1093
Stacktrace:
 [1] _transformation_helper(df::DataFrame, col_idx::UnitRange{Int64}, ::Base.RefValue{Any})
   @ DataFrames ~/.julia/packages/DataFrames/Lrd7K/src/abstractdataframe/selection.jl:606
 [2] select_transform!(::Base.RefValue{Any}, df::DataFrame, newdf::DataFrame, transformed_cols::Set{Symbol}, copycols::Bool, allow_resizing_newdf
...

```

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [October 30, 2022, 6:42pm UTC](https://discourse.julialang.org/t/plots-borking-on-sentinelarrays-produced-by-csv-read/89505/9 "2022-10-30T18:42:48Z")

</div>

Sorry, I made a typo because I was writing from my head.  
`All() .=> copy` should be written (with a `.` in front of `=>`)