Intermittent non-Julia error

General Usage
,
1

I have a deved copy of XLSX.jl which I’m working on. It currently passes all tests. When I try to use it in my own code, it works some of the time and fails others. This is not a run time failure that produces a normal Julia stacktrace. It doesn’t behave like a coding error. If I run it twice in succession, it will work one time and fail the next. There is no difference (from a user perspective) in the circumstances of the run. I’m just trying the same thing again. The failure rate is at least 50%.

I have no idea what is causing this, so do not know how to produce a MWE. I don’t think it is in XLSX (or my edits), but wonder if it may be something in my environment. Perhaps the error below will give someone else a clue.

  Activating project at `c:\Users\TGebbels\...\Good Cause Summaries Team - General\Good Cause Summaries`

Please submit a bug report with steps to reproduce this fault, and any error messages that follow (in their entirety). Thanks.
Exception: EXCEPTION_ACCESS_VIOLATION at 0x7ffa8901d78c -- jl_svecref at C:/workdir/src\julia.h:1118 [inlined]
jl_field_type_concrete at C:/workdir/src\julia.h:1325 [inlined]
immut_id_ at C:/workdir/src\builtins.c:416
in expression starting at c:\Users\TGebbels\...\Good Cause Summaries Team - General\Good Cause Summaries\Code\ConsentChooseBatches.jl:42
jl_svecref at C:/workdir/src\julia.h:1118 [inlined]
jl_field_type_concrete at C:/workdir/src\julia.h:1325 [inlined]
immut_id_ at C:/workdir/src\builtins.c:416
ijl_object_id_ at C:/workdir/src\builtins.c:481 [inlined]
ijl_object_id at C:/workdir/src\builtins.c:487 [inlined]
jl_idset_peek_bp at C:/workdir/src\idset.c:34 [inlined]
jl_idset_get at C:/workdir/src\idset.c:40
jl_as_global_root at C:/workdir/src\staticdata.c:2521 [inlined]
jl_as_global_root at C:/workdir/src\staticdata.c:2501
inst_datatype_inner at C:/workdir/src\jltypes.c:2115
jl_inst_arg_tuple_type at C:/workdir/src\jltypes.c:2338
arg_type_tuple at C:/workdir/src\gf.c:2278 [inlined]
jl_lookup_generic_ at C:/workdir/src\gf.c:3074
ijl_apply_generic at C:/workdir/src\gf.c:3121
unknown function (ip: 00000241525f7c04)
xmlParserInputBufferGrow at /workspace/srcdir/libxml2-2.13.5\xmlIO.c:2373
xmlTextReaderPushData at /workspace/srcdir/libxml2-2.13.5\xmlreader.c:790
xmlTextReaderRead at /workspace/srcdir/libxml2-2.13.5\xmlreader.c:1286
macro expansion at C:\Users\TGebbels\.julia\packages\EzXML\DL8na\src\error.jl:50 [inlined]
read_node at C:\Users\TGebbels\.julia\packages\EzXML\DL8na\src\streamreader.jl:199 [inlined]
iterate at C:\Users\TGebbels\.julia\packages\EzXML\DL8na\src\streamreader.jl:190 [inlined]
iterate at C:\Users\TGebbels\.julia\packages\EzXML\DL8na\src\streamreader.jl:190 [inlined]
iterate at C:\Users\TGebbels\...\Documents\Julia\Julia Experimenting\NewXLSX\XLSXmaster\src\stream.jl:129
iterate at C:\Users\TGebbels\...\Documents\Julia\Julia Experimenting\NewXLSX\XLSXmaster\src\stream.jl:239
unknown function (ip: 00000241525f9783)
open_or_read_xlsx at C:\Users\TGebbels\...\Documents\Julia\Julia Experimenting\NewXLSX\XLSXmaster\src\read.jl:239
open_xlsx_template at C:\Users\TGebbels\...\Documents\Julia\Julia Experimenting\NewXLSX\XLSXmaster\src\write.jl:9 [inlined]
#32 at .\none:0 [inlined]
iterate at .\generator.jl:48 [inlined]
collect_to! at .\array.jl:849
collect_to_with_first! at .\array.jl:827 [inlined]
collect at .\array.jl:801
getnewblanktemplates at C:\Users\TGebbels\...\Good Cause Summaries Team - General\Good Cause Summaries\Code\Template-jl-Library\lib-chelpers.jl:322
unknown function (ip: 00000241525f7594)
jl_apply at C:/workdir/src\julia.h:2157 [inlined]
do_call at C:/workdir/src\interpreter.c:126
eval_value at C:/workdir/src\interpreter.c:223
eval_stmt_value at C:/workdir/src\interpreter.c:174 [inlined]
eval_body at C:/workdir/src\interpreter.c:681
jl_interpret_toplevel_thunk at C:/workdir/src\interpreter.c:821
jl_toplevel_eval_flex at C:/workdir/src\toplevel.c:943
jl_toplevel_eval_flex at C:/workdir/src\toplevel.c:886
ijl_toplevel_eval at C:/workdir/src\toplevel.c:952 [inlined]
ijl_toplevel_eval_in at C:/workdir/src\toplevel.c:994
eval at .\boot.jl:430 [inlined]
include_string at .\loading.jl:2734
_include at .\loading.jl:2794
include at .\sysimg.jl:38
unknown function (ip: 00000241525c224b)
run at c:\Users\TGebbels\.vscode\extensions\julialang.language-julia-1.127.2\scripts\packages\DebugAdapter\src\packagedef.jl:122
startdebugger at c:\Users\TGebbels\.vscode\extensions\julialang.language-julia-1.127.2\scripts\packages\VSCodeDebugger\src\VSCodeDebugger.jl:45
unknown function (ip: 000002415259acd4)
jl_apply at C:/workdir/src\julia.h:2157 [inlined]
do_call at C:/workdir/src\interpreter.c:126
eval_value at C:/workdir/src\interpreter.c:223
eval_stmt_value at C:/workdir/src\interpreter.c:174 [inlined]
eval_body at C:/workdir/src\interpreter.c:681
jl_interpret_toplevel_thunk at C:/workdir/src\interpreter.c:821
jl_toplevel_eval_flex at C:/workdir/src\toplevel.c:943
jl_toplevel_eval_flex at C:/workdir/src\toplevel.c:886
ijl_toplevel_eval at C:/workdir/src\toplevel.c:952 [inlined]
ijl_toplevel_eval_in at C:/workdir/src\toplevel.c:994
eval at .\boot.jl:430 [inlined]
include_string at .\loading.jl:2734
_include at .\loading.jl:2794
include at .\Base.jl:557
jfptr_include_48090.1 at C:\Users\TGebbels\.julia\juliaup\julia-1.11.2+0.x64.w64.mingw32\lib\julia\sys.dll (unknown line)
exec_options at .\client.jl:323
_start at .\client.jl:531
jfptr__start_75162.1 at C:\Users\TGebbels\.julia\juliaup\julia-1.11.2+0.x64.w64.mingw32\lib\julia\sys.dll (unknown line)
jl_apply at C:/workdir/src\julia.h:2157 [inlined]
true_main at C:/workdir/src\jlapi.c:900
jl_repl_entrypoint at C:/workdir/src\jlapi.c:1059
mainCRTStartup at C:/workdir/cli\loader_exe.c:58
BaseThreadInitThunk at C:\WINDOWS\System32\KERNEL32.DLL (unknown line)
RtlUserThreadStart at C:\WINDOWS\SYSTEM32\ntdll.dll (unknown line)
Allocations: 16797303 (Pool: 16796565; Big: 738); GC: 13

 *  The terminal process "C:\Users\TGebbels\.julia\juliaup\julia-1.11.2+0.x64.w64.mingw32\bin\julia.exe '--color=yes', '--startup-file=no', '--history-file=no', 'c:\Users\TGebbels\.vscode\extensions\julialang.language-julia-1.127.2\scripts\debugger\run_debugger.jl', '\\.\pipe\vsc-jl-dbg-f87be04d-56c1-48b0-982e-8e7d8477c54d', '\\.\pipe\vsc-jl-dbg-45240bee-e524-4813-8452-5cb8f2bd2325', '\\.\pipe\vsc-jl-cr-8e007ae3-fa2a-4b18-8c49-0927afee36a3'" terminated with exit code: 1. 
 *  Terminal will be reused by tasks, press any key to close it. 

(Good Cause Summaries) pkg> st
Status `C:\Users\TGebbels\...\Good Cause Summaries Team - General\Good Cause Summaries\Project.toml`
  [69666777] Arrow v2.8.0
  [336ed68f] CSV v0.10.15
  [13f3f980] CairoMakie v0.12.18
  [a93c6f00] DataFrames v1.7.0
  [864edb3b] DataStructures v0.18.20
  [f5a160d5] GeoIO v1.19.8
  [dcc97b0b] GeoStats v0.72.0
  [cd3eb016] HTTP v1.10.15
  [0f8b85d8] JSON3 v1.14.1
  [c94c279d] Proj v1.8.0
  [2913bbd2] StatsBase v0.34.4
  [856f2bd8] StructTypes v1.11.0
  [fdbf4ff8] XLSX v0.10.5-dev `C:\Users\TGebbels\...\Documents\Julia\Julia Experimenting\NewXLSX\XLSXmaster`
  [49080126] ZipArchives v2.4.0
  [ade2ca70] Dates v1.11.0
  [de0858da] Printf v1.11.0
  [4ec0a83e] Unicode v1.11.0

julia> versioninfo()
Julia Version 1.11.2
Commit 5e9a32e7af (2024-12-01 20:02 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: 8 × 11th Gen Intel(R) Core(TM) i7-1185G7 @ 3.00GHz
  WORD_SIZE: 64
  LLVM: libLLVM-16.0.6 (ORCJIT, tigerlake)
Threads: 4 default, 0 interactive, 2 GC (on 8 virtual cores)
Environment:
  JULIA_EDITOR = code
  JULIA_NUM_THREADS = 4
2

I seem to remember that your dev changes to XLSX were related to file opening and closing and buffer, so maybe you’re somehow corrupting memory somewhere?

2 Likes
3

Yeah, this behavior tracks with a buffer overflow or other out-of-bounds memory corruption. Exactly where your out-of-bounds write lands might be a consistent offset from the end of the array, but what it writes over — and corrupts — will by arbitrary. And so sometimes it might crash, other times it might not.

Try using --check-bounds=yes or — if you can run on a Linux machine — using rr to allow you to “go back” to see what corrupted that location.

1 Like
4

I can reproducibly get a similar error with the following MWE from (Using `StreamReader` with TranscodingStreams leads to random Segmentation faults · Issue #200 · JuliaIO/EzXML.jl · GitHub).

There is a bug in how EzXML.jl or libxml2 uses pointers to Julia objects inside C callbacks.

using TranscodingStreams
using EzXML

undirected = """
<?xml version="1.0" encoding="UTF-8"?>
<graphml>
    <graph edgedefault="undirected">
        <node id="n0"/>
        <node id="n1"/>
        <edge source="n0" target="n1"/>
    </graph>
</graphml>
"""

reader = EzXML.StreamReader(NoopStream(IOBuffer(undirected)))

GC.gc()
GC.gc()
GC.gc()

while (item = iterate(reader)) != nothing
    @show reader.type, reader.name
end
1 Like
5

At present, XLSX.jl has dependencies on both ZipFile.jl and ZipArchive.jl. I’ve been making very small changes to remove the ZipFile.jl dependency. My changes are small and seemingly straightforward, but they are related to opening and closing files as you say @nilshg.

@nhz2 previously suggested I try to migrate the dependency on EzXML.jl to XML.jl. I was hoping to do this as a second stage - because it looks much harder to me and is probably beyond my skill level. If I cannot remove the ZipFile.jl dependency without also migrating to XML.jl, I’ll probable put the effort on the back-burner again.

6

The bugs in EzXML.jl might be fixable with significant work and knowledge of C programming. It involves going through EzXML.jl/src/streamreader.jl at master · JuliaIO/EzXML.jl · GitHub and ensuring everything matches with the documentation in xmlreader: the XMLReader implementation and how calling c functions with Julia callbacks should be done:
Calling C and Fortran Code · The Julia Language
Passing Julia Callback Functions to C
If libxml2 is not behaving as documented or is under-documented you may need to open an issue, for example, xmlTextReaderReadString() No longer returns NULL when node type is XML_READER_TYPE_END_ELEMENT (#817) · Issues · GNOME / libxml2 · GitLab
Fortunately, libxml2 is currently actively being maintained.

It might be simpler to create a streaming XML parser in pure Julia though, especially if you know a lot about XML already but don’t know much about C. (I don’t know anything about XML format, so I’m not sure how hard this would be)

This also might be a nice way to try out the proposed new Julia IO interface Ref: @jakobnissen’s GitHub - jakobnissen/IO-playground: Testing ground for possible new Julia interface