Broadcast assignment for a custom array type without linear indices fails

General Usage
1

I just did some housekeeping for one of my packages that defines a custom array type. In doing so, I ran my test suite and it fails for julia 1.5 where it worked for 1.3. I haven’t done a full detailed analysis of the problem but here are my findings/noteworthy info so far:

  • My array type (NpyArray, ~quf/Npy.jl: / - sourcehut git) doesn’t allow linear indices (I overload LinearIndices and throw an error to make sure of this), because the arrays aren’t necessarily linearised in Fortran-contiguous order and not being aware of this may lead to hard-to-diagnose bugs.

  • Here’s the failed build log: build #285247 - failed The stacktrace for this bug is:

  Stacktrace:
   [1] LinearIndices(::NpyArray{Bool,1}) at /home/build/Npy.jl/src/Npy.jl:277
   [2] copyto_unaliased! at ./abstractarray.jl:851 [inlined]
   [3] copyto! at ./abstractarray.jl:840 [inlined]
   [4] copyto! at ./broadcast.jl:927 [inlined]
   [5] copyto! at ./broadcast.jl:886 [inlined]
   [6] materialize! at ./broadcast.jl:848 [inlined]
   [7] materialize! at ./broadcast.jl:845 [inlined]
   [8] NpyArray(::IOStream, ::Array{Bool,1}) at /home/build/Npy.jl/src/Npy.jl:166
   [9] #8 at /home/build/Npy.jl/src/Npy.jl:182 [inlined]
   [10] open(::Npy.var"#8#9"{Tuple{Array{Bool,1}}}, ::String; kwargs::Base.Iterators.Pairs{Symbol,Bool,Tuple{Symbol,Symbol,Symbol},NamedTuple{(:read, :write, :create),Tuple{Bool,Bool,Bool}}}) at ./io.jl:325
   [11] NpyArray(::String, ::Array{Bool,1}) at /home/build/Npy.jl/src/Npy.jl:181
   [12] NpyArray(::var"#2#5", ::String, ::Vararg{Any,N} where N; kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /home/build/Npy.jl/src/Npy.jl:189
   [13] NpyArray at /home/build/Npy.jl/src/Npy.jl:189 [inlined]
   [14] (::var"#1#4"{DataType,Tuple{Int64},Array{Bool,1}})(::String) at /home/build/Npy.jl/test/runtests.jl:102
   [15] mktempdir(::var"#1#4"{DataType,Tuple{Int64},Array{Bool,1}}, ::String; prefix::String) at ./file.jl:682
   [16] mktempdir(::Function, ::String) at ./file.jl:680 (repeats 2 times)
   [17] top-level scope at /home/build/Npy.jl/test/runtests.jl:101
   [18] top-level scope at /build/julia/src/julia-1.5.0/usr/share/julia/stdlib/v1.5/Test/src/Test.jl:1115
   [19] top-level scope at /home/build/Npy.jl/test/runtests.jl:98
   [20] include(::String) at ./client.jl:457
   [21] top-level scope at none:6

The problem starts at [8], where I set npy_arr .= arr (~quf/Npy.jl: src/Npy.jl - sourcehut git), i.e. a simple assignment broadcast. This broadcast delegates to a couple of other functions, until copyto_unaliased! calls LinearIndices for my array type and fails.

  • I tried to identify a likely commit in Julia that caused/surfaced this bug (I hesitate to call this a regression in Julia since my package relies partly on undocumented behaviour - that an array with IndexStyle() = IndexCartesian() is never indexed with linear indices), but was unsuccessful (it doesn’t help that for some reason the line numbers in the stacktrace don’t line up with the line numbers in the v.1.5.0 tree).

  • I can fix this particular problem easily by replacing this broadcast with an explicit loop, but that doesn’t fix the problem anywhere else, and I want it to “just work”.

On to my questions: What is the best way to proceed in order to fix the problem now and make sure it stays fixed: Am I looking at a Julia regression that needs to be fixed upstream (linear indexing an IndexCartesian array isn’t expressly forbidden, but seems to go against the spirit of the index types)? Should I overload copyto! for my type, and if so which function signature do I choose? Do I have to worry about general broadcasting breaking in the future as well (it seems to work at the moment)?

2

Sorry to hear about the breakage. However, I do think you’re slightly mis-using this trait: the intention behind it is to signal the most efficient indexing style, not the only one that is supported. In particular the documentation for IndexStyle says

This allows users to access elements of your array using any indexing style, even when explicit methods have not been provided.

and being able to access with a linear index requires (I believe) that LinearIndices(A) works.

That said, if you identify the change that turned this into an error and engage in a dialog in a GitHub issue, there’s always a chance that we’d make changes to avoid the error. (No promises, though.)

2 Likes
3

Thanks for the reply. I’ve missed that line as I was only focused on the AbstractArray interface documentation.

Unfortunately, I am unable to effectively identify the change that surfaced this issue at the moment. As a workaround, I have decided to emulate column major linearisation for row major arrays for now.

1 Like