reproducible crash related to DataFrames, parametric types, method dispatching?

Christoph_Staub · October 24, 2018, 2:52pm

The code listed below reproducibly crashes Julia 1.0 and 0.7 (but not 0.6.2)

The crash does not occur in Julia 1.0 and 0.7 if one of the following modifications is applied to the code

the DataFrame is replaced by an Array and fdummy! is called with a view of the Array
the parameter of fdummy! is defined explicitly as an Array{Union{Missing, Float64}}
a second method for fdummy! is defined (which however is never called!)

function fdummy!(x::AbstractArray{T} where T<:Number)
    print("fdummy! Array{T}: ")
    println(x)
    return 0
end

the code reproducing the crash (print statements and assignment to x[1] just used for debugging) is:

using DataFrames

function fdummy!(x::AbstractArray{Union{Missing, T}} where T <: Number )
    print("fdummy! Array{U(M,T}}: ")
    x[1] = 777
    println(x)
    return 0
end

function f_df!(dfData::DataFrame)
    fdummy!(dfData[:B])
    return 0
end

df = DataFrame(A = collect(1.0:10.0), B=[collect(1.0:5.0)..., repeat([missing],outer=5)...])

f_df!(df)       # crashes

and produces the following output

fdummy! Array{U(M,T}}: Union{Missing, Float64}[777.0, 2.0, 3.0, 4.0, 5.0, missing, missing, missing,
 missing, missing]

Unreachable reached at 00000000174595C5

Please submit a bug report with steps to reproduce this fault, and any error messages that follow (i
n their entirety). Thanks.
Exception: EXCEPTION_ILLEGAL_INSTRUCTION at 0x174595c5 -- f_df! at C:\Users\christau\JuliaWorkspace\
crash.jl:20
in expression starting at C:\Users\christau\JuliaWorkspace\crash.jl:26
f_df! at C:\Users\christau\JuliaWorkspace\crash.jl:20
jl_fptr_trampoline at /home/Administrator/buildbot/worker/package_win64/build/src\gf.c:1829
jl_apply_generic at /home/Administrator/buildbot/worker/package_win64/build/src\gf.c:2182
do_call at /home/Administrator/buildbot/worker/package_win64/build/src\interpreter.c:324
eval_value at /home/Administrator/buildbot/worker/package_win64/build/src\interpreter.c:428
eval_stmt_value at /home/Administrator/buildbot/worker/package_win64/build/src\interpreter.c:363 [in
lined]
eval_body at /home/Administrator/buildbot/worker/package_win64/build/src\interpreter.c:682
jl_interpret_toplevel_thunk_callback at /home/Administrator/buildbot/worker/package_win64/build/src\
interpreter.c:799
unknown function (ip: FFFFFFFFFFFFFFFE)
unknown function (ip: 000000001394BD4F)
unknown function (ip: FFFFFFFFFFFFFFFF)
jl_toplevel_eval_flex at /home/Administrator/buildbot/worker/package_win64/build/src\toplevel.c:787
jl_parse_eval_all at /home/Administrator/buildbot/worker/package_win64/build/src\ast.c:838
include_string at .\loading.jl:1002
include_string at C:\Users\christau\.julia\packages\CodeTools\hB4Hy\src\eval.jl:30
unknown function (ip: 0000000017435ADA)
jl_fptr_trampoline at /home/Administrator/buildbot/worker/package_win64/build/src\gf.c:1829
jl_apply_generic at /home/Administrator/buildbot/worker/package_win64/build/src\gf.c:2182
#117 at C:\Users\christau\.julia\packages\Atom\7rQ1O\src\eval.jl:94
withpath at C:\Users\christau\.julia\packages\CodeTools\hB4Hy\src\utils.jl:30
jl_fptr_trampoline at /home/Administrator/buildbot/worker/package_win64/build/src\gf.c:1829
jl_apply_generic at /home/Administrator/buildbot/worker/package_win64/build/src\gf.c:2182 [inlined]
jl_apply at /home/Administrator/buildbot/worker/package_win64/build/src\julia.h:1536 [inlined]
jl_invoke at /home/Administrator/buildbot/worker/package_win64/build/src\gf.c:56
withpath at C:\Users\christau\.julia\packages\Atom\7rQ1O\src\eval.jl:46 [inlined]
#116 at C:\Users\christau\.julia\packages\Atom\7rQ1O\src\eval.jl:93 [inlined]
with_logstate at .\logging.jl:397
with_logger at .\logging.jl:493 [inlined]
#115 at C:\Users\christau\.julia\packages\Atom\7rQ1O\src\eval.jl:92 [inlined]
hideprompt at C:\Users\christau\.julia\packages\Atom\7rQ1O\src\repl.jl:85
jl_fptr_trampoline at /home/Administrator/buildbot/worker/package_win64/build/src\gf.c:1829
jl_apply_generic at /home/Administrator/buildbot/worker/package_win64/build/src\gf.c:2182
macro expansion at C:\Users\christau\.julia\packages\Atom\7rQ1O\src\eval.jl:91 [inlined]
#114 at .\task.jl:85
jl_fptr_trampoline at /home/Administrator/buildbot/worker/package_win64/build/src\gf.c:1829
jl_apply_generic at /home/Administrator/buildbot/worker/package_win64/build/src\gf.c:2182
jl_apply at /home/Administrator/buildbot/worker/package_win64/build/src\julia.h:1536 [inlined]
start_task at /home/Administrator/buildbot/worker/package_win64/build/src\task.c:268
Allocations: 33322083 (Pool: 33316307; Big: 5776); GC: 72

Julia has exited. Press Enter to start a new session.

Julia> versioninfo()
Julia Version 1.0.0
Commit 5d4eaca0c9 (2018-08-08 20:58 UTC)
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: Intel(R) Xeon(R) CPU E3-1220 v6 @ 3.00GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.0 (ORCJIT, skylake)
Environment:
  JULIA_NUM_THREADS = 2

kristoffer.carlsson · October 24, 2018, 2:59pm

fix #29269, type intersection bug in union parameters with typevars by JeffBezanson · Pull Request #29406 · JuliaLang/julia · GitHub fixes quite a few of these crashes so, if possible, it might be worth trying to build julia with that PR and see if the issue is still there

ExpandingMan · October 24, 2018, 3:12pm

Thanks for tracking down the MWE for this bug. Note that I’m unable to reproduce the crash on Julia 1.0.1 and DataFrames 0.14.1.

A couple of potentially helpful style tips that are probably not relevant to the bug:

It’s common prectice to put the where clause outside of the enclosing parentheses, it just makes it more clear that the type parameters apply to the entire function (admittedly the precedence of where is confusing to me).
DataFrames was designed very deliberately so that one doesn’t necessarily need Vector objects as columns, but can take any AbstractVector, so you could just do A = 1.0:10.0 and you’d be fine (granted I realize you probably wanted to replicate the original context of the bug as closely as possible).
Likewise, you could just do B = [1.0:5.0; fill(missing, 5)] (creates a Vector{Union{Missing,Float64}}).

Christoph_Staub · October 26, 2018, 11:10am

Thank you for the style tips, as expected they are not relevant to the bug.
Upgrading the version didn’t help in my case, I can reproduce the bug on Windows and Mac OS X with Julia 1.0.1 and DataFrame 0.14.1

Julia Version 1.0.1
Commit 0d713926f8 (2018-09-29 19:05 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin14.5.0)
  CPU: Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.0 (ORCJIT, skylake)
Environment:
  JULIA_EDITOR = atom -a
  JULIA_NUM_THREADS = 4

Liso · October 28, 2018, 12:59pm

You could use triple apostrophe ``` in line before and after text to format it as code.

julia> module Test
       using DataFrames 
       function fdummy!(x::AbstractArray{Union{Missing, T}} where T <: Number ) 
         print("fdummy! Array{U(M,T}}: ") 
         x[1] = 777 
         println(x) 
         return 0 
       end 
       function f_df!(dfData::DataFrame) 
         fdummy!(dfData[:B]) 
         return 0 
       end 
       df = DataFrame(A = collect(1.0:10.0), B=[collect(1.0:5)..., repeat([missing],outer=5)...]) 
       f_df!(df) # crashes
       end

signal (4): Illegal instruction
in expression starting at no file:14
f_df! at ./REPL[14]:10
…