Dropmissing!() on a dataframe causes segfault

I have code that was working just a week ago, and now when I get to the line with dropmissing!(df), I get this:

MethodError: Cannot `convert` an object of type Missing to an object of type Float64
Closest candidates are:
  convert(::Type{T}, ::T) where T<:Number at number.jl:6
  convert(::Type{T}, ::Number) where T<:Number at number.jl:7
  convert(::Type{T}, ::Base.TwicePrecision) where T<:Number at twiceprecision.jl:250
  ...
Stacktrace:
 [1] setindex!(::Array{Float64,1}, ::Missing, ::Int64) at ./array.jl:826
 [2] copyto!(::Array{Float64,1}, ::SentinelArrays.SentinelArray{Float64,1,Float64,Missing,SentinelArrays.ChainedVector{Float64,Array{Float64,1}}}) at ./multidimensional.jl:962
 [3] AbstractArray at ./array.jl:542 [inlined]
 [4] AbstractArray at ./boot.jl:432 [inlined]
 [5] convert at ./abstractarray.jl:15 [inlined]
 [6] disallowmissing(::SentinelArrays.SentinelArray{Float64,1,Float64,Missing,SentinelArrays.ChainedVector{Float64,Array{Float64,1}}}) at /home/efurtak/.julia/packages/Missings/notdc/src/Missings.jl:50
 [7] disallowmissing!(::DataFrame, ::Int64; error::Bool) at /home/efurtak/.julia/packages/DataFrames/4ov0U/src/dataframe/dataframe.jl:1001
 [8] #disallowmissing!#177 at /home/efurtak/.julia/packages/DataFrames/4ov0U/src/dataframe/dataframe.jl:1009 [inlined]
 [9] #disallowmissing!#180 at /home/efurtak/.julia/packages/DataFrames/4ov0U/src/dataframe/dataframe.jl:1025 [inlined]
 [10] disallowmissing! at /home/efurtak/.julia/packages/DataFrames/4ov0U/src/dataframe/dataframe.jl:1025 [inlined]
 [11] dropmissing!(::DataFrame, ::Colon; disallowmissing::Bool) at /home/efurtak/.julia/packages/DataFrames/4ov0U/src/abstractdataframe/abstractdataframe.jl:877
 [12] dropmissing! at /home/efurtak/.julia/packages/DataFrames/4ov0U/src/abstractdataframe/abstractdataframe.jl:876 [inlined] (repeats 2 times)
 [13] top-level scope at /run/media/efurtak/spindisk/oceano/oceano_met/code_explore_CDF_data/read_scrub_data.jl:24

julia> 
signal (11): Segmentation fault
in expression starting at none:0
jl_is_concrete_type at /buildworker/worker/package_linux64/build/src/julia.h:1230 [inlined]
jl_apply_tuple_type_v_ at /buildworker/worker/package_linux64/build/src/jltypes.c:1351 [inlined]
jl_apply_tuple_type_v at /buildworker/worker/package_linux64/build/src/jltypes.c:1364
arg_type_tuple at /buildworker/worker/package_linux64/build/src/gf.c:1772
jl_lookup_generic_ at /buildworker/worker/package_linux64/build/src/gf.c:2288 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2319
getmaxwidths at /home/efurtak/.julia/packages/DataFrames/4ov0U/src/abstractdataframe/show.jl:175
unknown function (ip: 0x7f38731fbddd)
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2145 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2323
#_show#444 at /home/efurtak/.julia/packages/DataFrames/4ov0U/src/abstractdataframe/show.jl:588
_show##kw at /home/efurtak/.julia/packages/DataFrames/4ov0U/src/abstractdataframe/show.jl:564 [inlined]
#show#445 at /home/efurtak/.julia/packages/DataFrames/4ov0U/src/abstractdataframe/show.jl:665 [inlined]
show##kw at /home/efurtak/.julia/packages/DataFrames/4ov0U/src/abstractdataframe/show.jl:665 [inlined]
#show#460 at /home/efurtak/.julia/packages/DataFrames/4ov0U/src/abstractdataframe/io.jl:56
unknown function (ip: 0x7f38731f8ca1)
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2145 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2323
show at /home/efurtak/.julia/packages/DataFrames/4ov0U/src/abstractdataframe/io.jl:56
#85 at /home/efurtak/.julia/packages/Atom/ipSjf/src/display/base.jl:56
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2145 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2323
#sprint#338 at ./strings/io.jl:105
sprint at ./strings/io.jl:101 [inlined]
macro expansion at /home/efurtak/.julia/packages/Atom/ipSjf/src/display/base.jl:56 [inlined]
render at /home/efurtak/.julia/packages/Media/ItEPc/src/system.jl:173
render′ at /home/efurtak/.julia/packages/Atom/ipSjf/src/display/errors.jl:119
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2145 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2323
wsitem at /home/efurtak/.julia/packages/Atom/ipSjf/src/workspace.jl:22
wsitem at /home/efurtak/.julia/packages/Atom/ipSjf/src/workspace.jl:19
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2145 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2323
#251 at /home/efurtak/.julia/packages/Atom/ipSjf/src/workspace.jl:8
iterate at ./generator.jl:47 [inlined]
collect_to! at ./array.jl:711 [inlined]
collect_to_with_first! at ./array.jl:689
_collect at ./array.jl:683
collect_similar at ./array.jl:607 [inlined]
map at ./abstractarray.jl:2072 [inlined]
workspace at /home/efurtak/.julia/packages/Atom/ipSjf/src/workspace.jl:8
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2145 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2323
handlemsg at /home/efurtak/.julia/packages/Atom/ipSjf/src/comm.jl:169
unknown function (ip: 0x7f388e5983c7)
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2145 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2323
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1700 [inlined]
do_apply at /buildworker/worker/package_linux64/build/src/builtins.c:643
#31 at ./task.jl:358
unknown function (ip: 0x7f388e566add)
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2145 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2323
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1700 [inlined]
start_task at /buildworker/worker/package_linux64/build/src/task.c:687
unknown function (ip: (nil))
Allocations: 218692717 (Pool: 218637202; Big: 55515); GC: 167

I didn’t change any code or data, does anyone have any ideas what is causing this issue?

Thanks

Please file an issue for this with a minimum working example (including the CSV you got the data from, if possible.

What version of DataFrames are you on? Can you try updating all your packages and trying again?

1 Like

In particular, the version of SentinelArrays you’re using is the most important information.

Thanks for the replies. I thought it was up to date yesterday, but now I am not sure. I just updated again, and am still getting the same error without the segfault. The version is v0.21.7. I’ll try to put a small example together at some point today. Where/ how do I file an issue?

Also, sorry I am not sure what SentinelArrays is, or how to check the version? It doesn’t seem to be an available package or one that I have installed.

can you show the results of ] st in your REPL?

File an issue at github here!

Here it is:

(@v1.4) pkg> st
Status `~/.julia/environments/v1.4/Project.toml`
  [c52e3926] Atom v0.12.21
  [336ed68f] CSV v0.7.7
  [8f4d0f93] Conda v1.4.1
  [a93c6f00] DataFrames v0.21.7
  [31c24e10] Distributions v0.23.12
  [5789e2e9] FileIO v1.4.3
  [5752ebe1] GMT v0.24.0
  [28b8d3ca] GR v0.48.0
  [0ef565a4] Geodesy v0.5.0
  [6218d12a] ImageMagick v1.1.6
  [4138dd39] JLD v0.10.0
  [494afd89] JavaCall v0.7.5
  [e5e0dc1b] Juno v0.8.4
  [47be7bcc] ORCA v0.5.0
  [3b7a836e] PGFPlots v3.3.2
  [8314cec4] PGFPlotsX v1.2.10
  [f0f68f2c] PlotlyJS v0.14.0
  [91a5bcdd] Plots v0.29.9
  [438e738f] PyCall v1.91.4
  [d330b81b] PyPlot v2.9.0
  [f2b01f46] Roots v1.0.5
  [60ddc479] StatPlots v0.9.2
  [f3b207a7] StatsPlots v0.14.13
  [61d0e4fa] Taro v0.8.2
  [fdbf4ff8] XLSX v0.7.3

Update:

The segfault seems to end on its own, but I figured out that for some reason

dropmissing!(df)

throws and error, but
df = dropmissing!(copy(df))

works! That seems really redundant. The strange thing is that I can execute the simple examples in the help, but when I have a dataframe with a larger number of missing values it acts up. Does this look like something that should be opened as an issue, or was there some syntax change maybe I am not aware of?

Yes please open an issue. The issue definitely has to do with SentinalArrays, when those kinds of arrays are copied they become normal arrays. This is why copy fixes things.

1 Like