How to generate a minidump and backtrace for a hanging Julia program on Appveyor/Windows?

General Usage
, , ,
1

Hello all,

I have what appears to be a deadlock(?) bug in my Julia package that is exposed only on Appveyor, and only when I test through the CI – if I enter the Appveyor VM through RDP and manually run tests, they pass…weird. Also, FWIW, this only happens on Julia 1.3. It all works for Julia 1.2. Please see this github issue for context.

Essentially, because the bug is only exposed through the CI, I need a way to generate a minidump or other debugging info via commands in the appveyor.yml, or I can RDP into the VM while my program is hanging and run commands manually from another terminal session (that might be the best option?).

I’m pretty new to debugging at this level, so any help (especially suggesting specific commands to run to generate the info I need) would be very much appreciated.

Thanks!
Vincent

2

Since you have RDP access it is probably easiest to do this manually:

  • download and install msys2 (https://www.msys2.org/)
  • install gdb (probably gdb or mingw-w64-gdb)
  • run gdb
  • at the (gdb) prompt, type attach <pid> where <pid> is the process ID of the deadlocked process (visible in the task manager)
  • at the (gdb) prompt, type bt, which should (hopefully) give you a backtrace hinting toward the problem

See also

1 Like
3

Thanks so much @ihnorton! I will look into all of this.

4

Hmm, when trying to run the above, I’m getting some errors (I think)

# gdb
GNU gdb (GDB) 8.3
Copyright (C) 2019 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-w64-mingw32".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word".
(gdb) attach 1316
Attaching to process 1316
[New Thread 1316.0x2e4]
[New Thread 1316.0x60c]
[New Thread 1316.0x500]
[New Thread 1316.0x34c]
warning: "C:\julia\bin\julia.exe": not in executable format: no symbols
warning: `C:\julia\bin\julia.exe': can't read symbols: no symbols.
(gdb) bt
#0  0x00000000 in ?? ()

I get a similar result for both of the PIDs named julia.exe

5

I don’t know why this has been happening, since I thought older versions used to work, but type sharedlibrary and (nearly) everything may start working again (of things that worked before).

1 Like
6

Hmm, still getting the same error. I ran sharedlibrary at the gdb prompt. Is that what you meant?

7

Tried again and ran sharedlibrary after attach and got a lot of output, but the backtrace was still empty upon running bt

This appeared quite a lot in the output of sharedlibrary:

Shared library architecture i386:x86-64 is not compatible with target architecture i386.
8

Just bumping this. Anyone have any advice on how to get a backtrace or minidump from a hanging Julia process? I think this will help resolve a bug in Julia 1.3 as well (as this issue does not occur in 1.2) if that provides any additional motivation :slight_smile:

Thanks again!

9

Ah, you’re also using the wrong build of gdb. I recommend getting it from mingw64 directly.

1 Like
10

Ah, thanks!! I’ll try that.

11

I’m using the 64 bit version of MinGW provided on the appveyor VM, and running gdb.exe directly. Appveyor has several versions of MinGW (64 and 32 bit) installed, and I’ve tried all of them with no luck (I’m getting the same error as above about incompatible architecture).

12

In gdb do:
gdb > file C:\julia\bin\libjulia.dll

That will load the symbols and the backtrace will work fine. Do
gdb > thread 1
to switch to the main thread before bt

1 Like
13

Thank you so much!!! It worked!!

Posting the entire process here to have it in one post, which I will mark as the solution.

Step 1: open the 64 bit version of gdb (navigated to C:\mingw-w64\x86_64-8.1.0-posix-seh-rt_v6-rev0\mingw64\bin\gdb.exe on appveyor)

Step 2: gdb > sharedlibrary

Step 3: gdb > file C:\julia\bin\libjulia.dll

Step 4: gdb > attach <pid> The PID be found in the resource monitor through task manager

Step 4: gdb > thread 1 to switch to the main thread

Step 5: gdb > bt to print the back trace.

Thank you @ihnorton, @adrpo, and @jameson for taking the time to help me figure this out.

EDIT
And since this post was originally about creating a mindump, here’s how to do that:
Step 1: Open an internet browser
Step 2: go to Process Explorer - Windows Sysinternals | Microsoft Docs
Step 3: click run now from Sysinternals Live
Step 4: Right click on the Julia process and select create dump > create minidump