Segfault in precompilation (Julia v1.0)

General Usage
1

Badly blocked by this one - any clue as to what to do next? Works fine in v0.7. Also using HTTP works. But this doesn’t:

julia> using Genie
[ Info: Precompiling Genie [c43c736e-a2d1-11e8-161f-af95117fbd1e]
WARNING: could not import Base.reload into Util

signal (11): Segmentation fault: 11
in expression starting at no file:0
jl_typemap_entry_assoc_exact at /Users/osx/buildbot/slave/package_osx64/build/src/typemap.c:774
jl_typemap_assoc_exact at /Users/osx/buildbot/slave/package_osx64/build/src/./julia_internal.h:883 [inlined]
jl_lookup_generic_ at /Users/osx/buildbot/slave/package_osx64/build/src/gf.c:2133
jl_apply_generic at /Users/osx/buildbot/slave/package_osx64/build/src/gf.c:2179
#print_to_string#330 at ./strings/io.jl:124
print_to_string at ./strings/io.jl:112 [inlined]
string at ./strings/io.jl:143 [inlined]
__init__ at ./sysinfo.jl:114
jl_apply at /Users/osx/buildbot/slave/package_osx64/build/src/./julia.h:1536 [inlined]
jl_module_run_initializer at /Users/osx/buildbot/slave/package_osx64/build/src/toplevel.c:90
_julia_init at /Users/osx/buildbot/slave/package_osx64/build/src/init.c:811
julia_init__threading at /Users/osx/buildbot/slave/package_osx64/build/src/task.c:302
main at /Applications/Julia-1.0.app/Contents/Resources/julia/bin/julia (unknown line)
Allocations: 2520 (Pool: 2512; Big: 8); GC: 0
ERROR: LoadError: LoadError: Failed to precompile HTTP [cd3eb016-35fb-5094-929b-558a96fad6f3] to /Users/adrian/.julia/compiled/v1.0/HTTP/zXWya.ji.
Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] macro expansion at ./logging.jl:313 [inlined]
 [3] compilecache(::Base.PkgId, ::String) at ./loading.jl:1184
 [4] _require(::Base.PkgId) at ./logging.jl:311
 [5] require(::Base.PkgId) at ./loading.jl:852
 [6] macro expansion at ./logging.jl:311 [inlined]
 [7] require(::Module, ::Symbol) at ./loading.jl:834
 [8] include at ./boot.jl:317 [inlined]
 [9] include_relative(::Module, ::String) at ./loading.jl:1038
 [10] include at ./sysimg.jl:29 [inlined]
 [11] include(::String) at /Users/adrian/.julia/packages/Genie/LPtpY/src/Genie.jl:4
 [12] top-level scope at none:0
 [13] include at ./boot.jl:317 [inlined]
 [14] include_relative(::Module, ::String) at ./loading.jl:1038
 [15] include(::Module, ::String) at ./sysimg.jl:29
 [16] top-level scope at none:2
 [17] eval at ./boot.jl:319 [inlined]
 [18] eval(::Expr) at ./client.jl:389
 [19] top-level scope at ./none:3
in expression starting at /Users/adrian/.julia/packages/Genie/LPtpY/src/Cookies.jl:6
in expression starting at /Users/adrian/.julia/packages/Genie/LPtpY/src/Genie.jl:31
ERROR: Failed to precompile Genie [c43c736e-a2d1-11e8-161f-af95117fbd1e] to /Users/adrian/.julia/compiled/v1.0/Genie/8eazC.ji.
Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] macro expansion at ./logging.jl:313 [inlined]
 [3] compilecache(::Base.PkgId, ::String) at ./loading.jl:1184
 [4] macro expansion at ./logging.jl:311 [inlined]
 [5] _require(::Base.PkgId) at ./loading.jl:941
 [6] require(::Base.PkgId) at ./loading.jl:852
 [7] macro expansion at ./logging.jl:311 [inlined]
 [8] require(::Module, ::Symbol) at ./loading.jl:834
2

After adding and removing HTTP the segfault is now in Gumbo:

(v1.0) pkg> precompile
Precompiling project...
Precompiling Genie
[ Info: Precompiling Genie [c43c736e-a2d1-11e8-161f-af95117fbd1e]

signal (11): Segmentation fault: 11
in expression starting at no file:0
jl_typemap_entry_assoc_exact at /Users/osx/buildbot/slave/package_osx64/build/src/typemap.c:774
jl_typemap_assoc_exact at /Users/osx/buildbot/slave/package_osx64/build/src/./julia_internal.h:883 [inlined]
jl_lookup_generic_ at /Users/osx/buildbot/slave/package_osx64/build/src/gf.c:2133
jl_apply_generic at /Users/osx/buildbot/slave/package_osx64/build/src/gf.c:2179
#print_to_string#330 at ./strings/io.jl:124
print_to_string at ./strings/io.jl:112 [inlined]
string at ./strings/io.jl:143 [inlined]
__init__ at ./sysinfo.jl:114
jl_apply at /Users/osx/buildbot/slave/package_osx64/build/src/./julia.h:1536 [inlined]
jl_module_run_initializer at /Users/osx/buildbot/slave/package_osx64/build/src/toplevel.c:90
_julia_init at /Users/osx/buildbot/slave/package_osx64/build/src/init.c:811
julia_init__threading at /Users/osx/buildbot/slave/package_osx64/build/src/task.c:302
main at /Applications/Julia-1.0.app/Contents/Resources/julia/bin/julia (unknown line)
Allocations: 2519 (Pool: 2512; Big: 7); GC: 0
ERROR: LoadError: LoadError: Failed to precompile Gumbo [708ec375-b3d6-5a57-a7ce-8257bf98657a] to /Users/adrian/.julia/compiled/v1.0/Gumbo/mllB2.ji.
Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] macro expansion at ./logging.jl:313 [inlined]
 [3] compilecache(::Base.PkgId, ::String) at ./loading.jl:1184
 [4] _require(::Base.PkgId) at ./logging.jl:311
 [5] require(::Base.PkgId) at ./loading.jl:852
 [6] macro expansion at ./logging.jl:311 [inlined]
 [7] require(::Module, ::Symbol) at ./loading.jl:834
 [8] include at ./boot.jl:317 [inlined]
 [9] include_relative(::Module, ::String) at ./loading.jl:1038
 [10] include at ./sysimg.jl:29 [inlined]
 [11] include(::String) at /Users/adrian/.julia/packages/Genie/LPtpY/src/Genie.jl:4
 [12] top-level scope at none:0
 [13] include at ./boot.jl:317 [inlined]
 [14] include_relative(::Module, ::String) at ./loading.jl:1038
 [15] include(::Module, ::String) at ./sysimg.jl:29
 [16] top-level scope at none:2
 [17] eval at ./boot.jl:319 [inlined]
 [18] eval(::Expr) at ./client.jl:389
 [19] top-level scope at ./none:3
in expression starting at /Users/adrian/.julia/packages/Genie/LPtpY/src/Flax.jl:6
in expression starting at /Users/adrian/.julia/packages/Genie/LPtpY/src/Genie.jl:34
ERROR: Failed to precompile Genie [c43c736e-a2d1-11e8-161f-af95117fbd1e] to /Users/adrian/.julia/compiled/v1.0/Genie/8eazC.ji.
Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] macro expansion at ./logging.jl:313 [inlined]
 [3] compilecache(::Base.PkgId, ::String) at ./loading.jl:1184
 [4] precompile(::Pkg.Types.Context) at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.0/Pkg/src/API.jl:489
 [5] do_precompile!(::Dict{Symbol,Any}, ::Array{String,1}, ::Dict{Symbol,Any}) at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.0/Pkg/src/REPLMode.jl:586
 [6] #invokelatest#1(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::Any, ::Any, ::Vararg{Any,N} where N) at ./essentials.jl:686
 [7] invokelatest(::Any, ::Any, ::Vararg{Any,N} where N) at ./essentials.jl:685
 [8] do_cmd!(::Pkg.REPLMode.PkgCommand, ::REPL.LineEditREPL) at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.0/Pkg/src/REPLMode.jl:542
 [9] #do_cmd#30(::Bool, ::Function, ::REPL.LineEditREPL, ::String) at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.0/Pkg/src/REPLMode.jl:507
 [10] do_cmd at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.0/Pkg/src/REPLMode.jl:503 [inlined]
 [11] (::getfield(Pkg.REPLMode, Symbol("##41#44")){REPL.LineEditREPL,REPL.LineEdit.Prompt})(::REPL.LineEdit.MIState, ::Base.GenericIOBuffer{Array{UInt8,1}}, ::Bool) at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.0/Pkg/src/REPLMode.jl:842
 [12] #invokelatest#1 at ./essentials.jl:686 [inlined]
 [13] invokelatest at ./essentials.jl:685 [inlined]
 [14] run_interface(::REPL.Terminals.TextTerminal, ::REPL.LineEdit.ModalInterface, ::REPL.LineEdit.MIState) at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.0/REPL/src/LineEdit.jl:2261
 [15] run_frontend(::REPL.LineEditREPL, ::REPL.REPLBackendRef) at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.0/REPL/src/REPL.jl:1029
 [16] run_repl(::REPL.AbstractREPL, ::Any) at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.0/REPL/src/REPL.jl:191
 [17] (::getfield(Base, Symbol("##720#722")){Bool,Bool,Bool,Bool})(::Module) at ./logging.jl:311
 [18] #invokelatest#1 at ./essentials.jl:686 [inlined]
 [19] invokelatest at ./essentials.jl:685 [inlined]
 [20] macro expansion at ./logging.jl:308 [inlined]
 [21] run_main_repl(::Bool, ::Bool, ::Bool, ::Bool, ::Bool) at ./client.jl:330
 [22] exec_options(::Base.JLOptions) at ./client.jl:242
 [23] _start() at ./client.jl:421
3

Nevermind - removing the Julia package folder and doing a clean install fixed the problem.

4

In that case it still seems very much like a bug worthy of reporting (unless you did something crazy). No idea how you’d go about doing that however…

5

Unfortunately, it keeps popping up.

It seems to be related to using precompiled packages. For instance, I had an environment which worked (1.0). I’ve made a few changes to a dev package and pushed to master. Then did an update in a different env which was using this changed master branch. This triggered a precompilation of the package and again segfaults (reproducible on every run):

[ Info: Recompiling stale cache file /Users/adrian/.julia/compiled/v1.0/SearchLight/GqMHX.ji for SearchLight [340e8cb6-72eb-11e8-37ce-c97ebeb32050]
WARNING: using SearchLight.Loggers in module REPL conflicts with an existing identifier.
WARNING: redefining constant config

signal (11): Segmentation fault: 11
in expression starting at no file:0
strlen at /usr/lib/system/libsystem_c.dylib (unknown line)
Allocations: 69994057 (Pool: 69978818; Big: 15239); GC: 158
[1]    19397 segmentation fault  julia

Then I wiped off the .julia/compiled folder and upon recompilation, everything worked again:

[ Info: Precompiling SearchLight [340e8cb6-72eb-11e8-37ce-c97ebeb32050]
WARNING: using SearchLight.Loggers in module REPL conflicts with an existing identifier.
WARNING: redefining constant config
[ Info: Listening on: Sockets.InetAddr{Sockets.IPv4}(ip"127.0.0.1", 0x1f40)

This is consistent with my experience yesterday when wiping off the .julia/ folder “fixed” the segfaults.

6

For what it’s worth, I often do things like what you are describing, and I’ve never seen a segfault during precompilation.

I suggest once you start seeing the error again you work your way down your dependency tree and try to determine the minmal set of packages with which the error occurs. It seems highly likely that a specific package is the culprit.

7

OK :slight_smile: I kept digging – it seems to be related to a variable Genie.config (an object of type Settings) which does not look right.

When I attempt to access it in a try ... catch block, I catch a: ErrorException("fatal error in type inference (type bound)")

Outside the try ... catch block this line breaks Julia:

@show Genie.config

Genie.config = Core.Compiler.DomTreeNode[LineInfoNode(Base, :Type, Symbol("show.jl"), 192, 0), LineInfoNode(Base, :Type, Symbol("show.jl"), 193, 0), LineInfoNode(Base, :unwrapcontext, Symbol("show.jl"), 180, 1), LineInfoNode(Base, :getproperty, Symbol("sysimg.jl"), 18, 3), LineInfoNode(Base, :getproperty, Symbol("sysimg.jl"), 18, 3), LineInfoNode(Base, :indexed_iterate, Symbol("tuple.jl"), 60, 1), LineInfoNode(Base, :indexed_iterate, Symbol("tuple.jl"), 60, 1), LineInfoNode(Base, :indexed_iterate, Symbol("tuple.jl"), 60, 7), LineInfoNode(Base, :indexed_iterate, Symbol("tuple.jl"), 60, 7), LineInfoNode(Base, :getindex, Symbol("tuple.jl"), 24, 9), LineInfoNode(Base, :indexed_iterate, Symbol("tuple.jl"), 60, 1), LineInfoNode(Base, :indexed_iterate, Symbol("tuple.jl"), 60, 1), LineInfoNode(Base, :getindex, Symbol("tuple.jl"), 24, 12), LineInfoNode(Base, :getindex, Symbol("pair.jl"), 59, 2), LineInfoNode(Base, :getindex, Symbol("pair.jl"), 59, 2), LineInfoNode(Base, :Type, Symbol("dict.jl"), 696, 2), LineInfoNode(Base, :convert, Symbol("abstractdict.jl"), 487, 16), LineInfoNode(Base, :convert, Symbol("essentials.jl"), 154, 16), LineInfoNode(Base, :convert, Symbol("essentials.jl"), 153, 16), LineInfoNode(Base, :Type, Symbol("show.jl"), 183, 2), LineInfoNode(Base, :Type, Symbol("show.jl"), 184, 2), LineInfoNode(Base, :unwrapcontext, Symbol("show.jl"), 179, 20), LineInfoNode(Base, :Type, Symbol("dict.jl"), 694, 22), LineInfoNode(Base, :getindex, Symbol("tuple.jl"), 24, 20), LineInfoNode(Base, :Type, Symbol("show.jl"), 173, 21), LineInfoNode(Base, :Type, Symbol("show.jl"), 174, 21), LineInfoNode(Base, :convert, Symbol("essentials.jl"), 154, 26), LineInfoNode(Base, :convert, Symbol("abstractdict.jl"), 487, 26)]
Invalid instruction at 0x107b3fa2f: 0x48, 0x83, 0x7d, 0x00, 0x00, 0x7e, 0x76, 0x0f, 0xb6, 0x4d, 0x08, 0x81, 0xf9, 0xf8, 0x00

signal (4): Illegal instruction: 4
in expression starting at no file:0
ncodeunits at ./strings/string.jl:82 [inlined]
iterate at ./strings/string.jl:176 [inlined]
iterate at ./strings/string.jl:176 [inlined]
isempty at ./essentials.jl:710 [inlined]
startswith at ./strings/util.jl:25
isabspath at ./path.jl:84 [inlined]
joinpath at ./path.jl:217
macro expansion at ./show.jl:555 [inlined]
log_path at /Users/adrian/.julia/dev/Genie/src/Loggers.jl:82
log_path at /Users/adrian/.julia/dev/Genie/src/Loggers.jl:81 [inlined]
log at /Users/adrian/.julia/dev/Genie/src/Loggers.jl:44
log at /Users/adrian/.julia/dev/Genie/src/Loggers.jl:27 [inlined]
#load_app#4 at /Users/adrian/.julia/dev/Genie/src/REPL.jl:77
unknown function (ip: 0x1217f215e)
jl_fptr_trampoline at /Users/osx/buildbot/slave/package_osx64/build/src/gf.c:1829
#load_app at ./none:0 [inlined]
#new_app#1 at /Users/adrian/.julia/dev/Genie/src/REPL.jl:47
unknown function (ip: 0x1217a4e29)
jl_fptr_trampoline at /Users/osx/buildbot/slave/package_osx64/build/src/gf.c:1829
new_app at /Users/adrian/.julia/dev/Genie/src/REPL.jl:25
jl_fptr_trampoline at /Users/osx/buildbot/slave/package_osx64/build/src/gf.c:1829
do_call at /Users/osx/buildbot/slave/package_osx64/build/src/interpreter.c:324
eval_stmt_value at /Users/osx/buildbot/slave/package_osx64/build/src/interpreter.c:363 [inlined]
eval_body at /Users/osx/buildbot/slave/package_osx64/build/src/interpreter.c:686
jl_interpret_toplevel_thunk_callback at /Users/osx/buildbot/slave/package_osx64/build/src/interpreter.c:799
unknown function (ip: 0xfffffffffffffffe)
unknown function (ip: 0x111adf75f)
unknown function (ip: 0x1)
jl_interpret_toplevel_thunk at /Users/osx/buildbot/slave/package_osx64/build/src/interpreter.c:808
jl_toplevel_eval_flex at /Users/osx/buildbot/slave/package_osx64/build/src/toplevel.c:787
jl_toplevel_eval_in at /Users/osx/buildbot/slave/package_osx64/build/src/builtins.c:622
eval at ./boot.jl:319
eval_user_input at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.0/REPL/src/REPL.jl:85
run_backend at /Users/adrian/.julia/packages/Revise/51oQc/src/Revise.jl:766
#58 at ./task.jl:259
jl_fptr_trampoline at /Users/osx/buildbot/slave/package_osx64/build/src/gf.c:1829
jl_apply at /Users/osx/buildbot/slave/package_osx64/build/src/./julia.h:1536 [inlined]
start_task at /Users/osx/buildbot/slave/package_osx64/build/src/task.c:268
Allocations: 49257860 (Pool: 49246344; Big: 11516); GC: 110
[1]    23803 illegal hardware instruction  julia

Genie.config should be an instance of a Settings type which normally looks like this:

Genie.config = Genie.Configuration.Settings(8000, 1, "public", true, "Genie/0.7.0/Julia/1.0.0", "dev", Dict("Access-Control-Allow-Headers"=>"","Access-Control-Allow-Methods"=>"","Access-Control-Allow-Credentials"=>"","Access-Control-Allow-Origin"=>"","Access-Control-Expose-Headers"=>"","Access-Control-Max-Age"=>"86400"), String[], false, 10000, "task", "test", "log", "cache", :FileCacheAdapter, 0, "storage_caches", false, :debug, :verbose, true, true, true, "/", true, false, true, true, "__GENIESID", :File, "session", "storage_sessions", Tuple{String,String}[], :Flax, :Flax, false, true, false, false, :app)
8

Sounds like you are probably at a point where you should report it as a julia issue.

1 Like
9

Will do, thanks!

10

Might be Revise related though – that’s where it seems to originate. And it does follow a file include which might redefine some variables, including this one. Not sure if it makes any sense, but the object might be in a state of being recompiled and updated? Don’t know much about the Revise internals but I think @tim.holy mentioned something about diff-ing files and partial recompilation.

And before the crash there’s also a warning

WARNING: redefining constant config
11

@essenciary, you figured this out, right? After wiping everything and starting over?

12

@quinnj I can’t say that I fixed it… I think I managed to go around the problem. I traced this to the following:

1 - a const pointing to an object of type Genie.Configuration.Settings, which
2 - upon a (big) file include
3 - gets overwritten
4 - when trying to access a property of the overwritten object, right after the include, Julia crashes

Per the above, the object is no longer a Genie.Configuration.Settings but a Core.Compiler.DomTreeNode. My suspicion is that this is Revise.jl related and that I’m trying to access the variable right after the include, while the code changes are being “ingested” by Revise and the app is being recompiled (I’m not sure if that makes any sense, it’s just a gut feeling).

The segfault was present in maybe 50% of my tests, on multiple platforms – and also reported by other users.

I used a workaround and no longer access the object in that function.

13

Your stacktraces might link back to Revise because Revise “takes over” the REPL’s call to eval_user_input (i.e., the function that executes what the user types on the command prompt); as a consequence, all commands ultimately are being issued from Revise. However, if you weren’t running Revise, in theory you’d get exactly the same behavior. If you use julia --startup-file=no do you still get the segfaults? If not that would more definitively place the blame in Revise’s court. If you are doing things before the segfault that might trigger revisions, perhaps try capturing and submitting a log file right before the segfault?

14

@tim.holy Thanks

It wasn’t only the stacktrace that got me thinking about Revise – but also the fact that what is initially an instance of Genie.Configuration.Settings becomes a Core.Compiler.DomTreeNode[...] right before the crash. I thought this would be consistent with a recompilation.

But then again, I have no understanding of this level of the stack so that’s why I pinged you :slight_smile:

15

The most reliable test is to start your Julia session without Revise and see if you still get the segfaults. If the answer is yes, then it’s very unlikely to be Revise; if not, then we have some debugging to do.

1 Like
16

Good point.

I’ll give it a try, although it’s not that straightforward. I have since refactored the problematic code – and I have using Revise in a few places in the app. But I can set up a test branch to mess around a bit :slight_smile: I’ll let you know. Thanks again!