Is Julia REPL supposed to choke on this?

General Usage
1

Julia 1.4.2 REPL completely chokes on this expression:

let
  n = 10_000

  struct Sensors
    depth::NTuple{n, Float32}
    bottom::Float32
    top::Float32
  end
  
  s = Sensors(ntuple(x->0, n), 0, 0)
end

What is wrong with it?

2

Reproduced. Looks like a stack overflow.

julia> let
         n = 10_000

         struct Sensors
             depth::NTuple{n, Float32}
                 bottom::Float32
                     top::Float32
                       end

         s = Sensors(ntuple(x->0, n), 0, 0)
       end

Internal error: encountered unexpected error in runtime:
StackOverflowError()
jl_typemap_insert_generic at /buildworker/worker/package_linux64/build/src/typemap.c:944
jl_typemap_array_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:969 [inlined]
jl_typemap_level_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:1010
jl_typemap_insert_generic at /buildworker/worker/package_linux64/build/src/typemap.c:944
jl_typemap_array_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:969 [inlined]
jl_typemap_level_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:1010
jl_typemap_insert_generic at /buildworker/worker/package_linux64/build/src/typemap.c:944
jl_typemap_array_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:969 [inlined]
jl_typemap_level_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:1010
jl_typemap_insert_generic at /buildworker/worker/package_linux64/build/src/typemap.c:944
jl_typemap_array_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:969 [inlined]
jl_typemap_level_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:1010
jl_typemap_insert_generic at /buildworker/worker/package_linux64/build/src/typemap.c:944
jl_typemap_array_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:969 [inlined]
jl_typemap_level_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:1010
jl_typemap_insert_generic at /buildworker/worker/package_linux64/build/src/typemap.c:944
jl_typemap_array_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:969 [inlined]
jl_typemap_level_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:1010
jl_typemap_insert_generic at /buildworker/worker/package_linux64/build/src/typemap.c:944
jl_typemap_array_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:969 [inlined]
jl_typemap_level_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:1010
jl_typemap_insert_generic at /buildworker/worker/package_linux64/build/src/typemap.c:944
jl_typemap_array_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:969 [inlined]
jl_typemap_level_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:1010
jl_typemap_insert_generic at /buildworker/worker/package_linux64/build/src/typemap.c:944
jl_typemap_array_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:969 [inlined]
jl_typemap_level_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:1010
jl_typemap_insert_generic at /buildworker/worker/package_linux64/build/src/typemap.c:944
jl_typemap_array_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:969 [inlined]
jl_typemap_level_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:1010
jl_typemap_insert_generic at /buildworker/worker/package_linux64/build/src/typemap.c:944
jl_typemap_array_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:969 [inlined]
jl_typemap_level_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:1010
jl_typemap_insert_generic at /buildworker/worker/package_linux64/build/src/typemap.c:944
jl_typemap_array_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:969 [inlined]
jl_typemap_level_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:1010
jl_typemap_insert_generic at /buildworker/worker/package_linux64/build/src/typemap.c:944
jl_typemap_array_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:969 [inlined]
jl_typemap_level_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:1010
jl_typemap_insert_generic at /buildworker/worker/package_linux64/build/src/typemap.c:944
jl_typemap_array_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:969 [inlined]
jl_typemap_level_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:1010
jl_typemap_insert_generic at /buildworker/worker/package_linux64/build/src/typemap.c:944
jl_typemap_array_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:969 [inlined]
jl_typemap_level_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:1010
jl_typemap_insert_generic at /buildworker/worker/package_linux64/build/src/typemap.c:944
jl_typemap_array_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:969 [inlined]
jl_typemap_level_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:1010
jl_typemap_insert_generic at /buildworker/worker/package_linux64/build/src/typemap.c:944
jl_typemap_array_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:969 [inlined]
jl_typemap_level_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:1010
jl_typemap_insert_generic at /buildworker/worker/package_linux64/build/src/typemap.c:944
jl_typemap_array_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:969 [inlined]
jl_typemap_level_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:1010
jl_typemap_insert_generic at /buildworker/worker/package_linux64/build/src/typemap.c:944
jl_typemap_array_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:969 [inlined]
jl_typemap_level_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:1010
jl_typemap_insert_generic at /buildworker/worker/package_linux64/build/src/typemap.c:944
jl_typemap_array_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:969 [inlined]
jl_typemap_level_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:1010
jl_typemap_insert_generic at /buildworker/worker/package_linux64/build/src/typemap.c:944
jl_typemap_array_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:969 [inlined]
jl_typemap_level_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:1010
jl_typemap_insert_generic at /buildworker/worker/package_linux64/build/src/typemap.c:944
jl_typemap_array_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:969 [inlined]
jl_typemap_level_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:1010
jl_typemap_insert_generic at /buildworker/worker/package_linux64/build/src/typemap.c:944
jl_typemap_array_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:969 [inlined]
jl_typemap_level_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:1010
jl_typemap_insert_generic at /buildworker/worker/package_linux64/build/src/typemap.c:944
jl_typemap_array_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:969 [inlined]
jl_typemap_level_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:1010
jl_typemap_insert_generic at /buildworker/worker/package_linux64/build/src/typemap.c:944
jl_typemap_array_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:969 [inlined]
jl_typemap_level_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:1010
jl_typemap_insert_generic at /buildworker/worker/package_linux64/build/src/typemap.c:944
jl_typemap_array_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:969 [inlined]
jl_typemap_level_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:1010
jl_typemap_insert_generic at /buildworker/worker/package_linux64/build/src/typemap.c:944
jl_typemap_array_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:969 [inlined]
jl_typemap_level_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:1010
jl_typemap_insert_generic at /buildworker/worker/package_linux64/build/src/typemap.c:944
jl_typemap_array_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:969 [inlined]
jl_typemap_level_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:1010
jl_typemap_insert_generic at /buildworker/worker/package_linux64/build/src/typemap.c:944
jl_typemap_array_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:969 [inlined]
jl_typemap_level_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:1010
jl_typemap_insert_generic at /buildworker/worker/package_linux64/build/src/typemap.c:944
jl_typemap_array_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:969 [inlined]
jl_typemap_level_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:1010
jl_typemap_insert_generic at /buildworker/worker/package_linux64/build/src/typemap.c:944
jl_typemap_array_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:969 [inlined]
jl_typemap_level_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:1010
jl_typemap_insert_generic at /buildworker/worker/package_linux64/build/src/typemap.c:944
jl_typemap_array_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:969 [inlined]
jl_typemap_level_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:1010
jl_typemap_insert_generic at /buildworker/worker/package_linux64/build/src/typemap.c:944
jl_typemap_array_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:969 [inlined]
jl_typemap_level_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:1010
jl_typemap_insert_generic at /buildworker/worker/package_linux64/build/src/typemap.c:944
jl_typemap_array_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:969 [inlined]
jl_typemap_level_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:1010
jl_typemap_insert_generic at /buildworker/worker/package_linux64/build/src/typemap.c:944
jl_typemap_array_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:969 [inlined]
jl_typemap_level_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:1010
jl_typemap_insert_generic at /buildworker/worker/package_linux64/build/src/typemap.c:944
jl_typemap_array_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:969 [inlined]
jl_typemap_level_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:1010
jl_typemap_insert_generic at /buildworker/worker/package_linux64/build/src/typemap.c:944
jl_typemap_array_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:969 [inlined]
jl_typemap_level_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:1010
jl_typemap_insert_generic at /buildworker/worker/package_linux64/build/src/typemap.c:944
jl_typemap_array_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:969 [inlined]
jl_typemap_level_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:1010
jl_typemap_insert_generic at /buildworker/worker/package_linux64/build/src/typemap.c:944
jl_typemap_array_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:969 [inlined]
jl_typemap_level_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:1010
jl_typemap_insert_generic at /buildworker/worker/package_linux64/build/src/typemap.c:944
jl_typemap_array_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:969 [inlined]
jl_typemap_level_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:1010
jl_typemap_insert_generic at /buildworker/worker/package_linux64/build/src/typemap.c:944
jl_typemap_array_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:969 [inlined]
jl_typemap_level_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:1010
jl_typemap_insert_generic at /buildworker/worker/package_linux64/build/src/typemap.c:944
jl_typemap_array_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:969 [inlined]
jl_typemap_level_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:1010
jl_typemap_insert_generic at /buildworker/worker/package_linux64/build/src/typemap.c:944
jl_typemap_array_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:969 [inlined]
jl_typemap_level_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:1010
jl_typemap_insert_generic at /buildworker/worker/package_linux64/build/src/typemap.c:944
jl_typemap_array_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:969 [inlined]
jl_typemap_level_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:1010
jl_typemap_insert_generic at /buildworker/worker/package_linux64/build/src/typemap.c:944
jl_typemap_array_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:969 [inlined]
jl_typemap_level_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:1010
jl_typemap_insert_generic at /buildworker/worker/package_linux64/build/src/typemap.c:944
jl_typemap_array_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:969 [inlined]
jl_typemap_level_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:1010
jl_typemap_insert_generic at /buildworker/worker/package_linux64/build/src/typemap.c:944
jl_typemap_array_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:969 [inlined]
jl_typemap_level_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:1010
jl_typemap_insert_generic at /buildworker/worker/package_linux64/build/src/typemap.c:944
jl_typemap_array_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:969 [inlined]
jl_typemap_level_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:1010
jl_typemap_insert_generic at /buildworker/worker/package_linux64/build/src/typemap.c:944
jl_typemap_array_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:969 [inlined]
jl_typemap_level_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:1010
jl_typemap_insert_generic at /buildworker/worker/package_linux64/build/src/typemap.c:944
jl_typemap_array_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:969 [inlined]
jl_typemap_level_insert_ at /buildworker/worker/package_linux64/build/src/typemap.c:1010
jl_typemap_insert_generic at /buildworker/worker/package_linux64/build/src/typemap.c:944
3

Tuples with a large number of elements tend to result in longer compile times and worse code being generated. For anything over 32? elements I would suggest using an array instead.

In this case it appears that trying to specialize the code for a 10,000 element Tuple caused a stack overflow in the compiler.

5 Likes
4

I was trying to use a large tuple, in fact I need much more than 10,000 elements, because the examples I googled, suggested NTuple{n, Float32} as a type compatible with C array float[n]. I will try Array{Float32, 1} in that role.

5

I see, if your trying to be make a type with the same layout as a c struct then Array{Float32,1} won’t work. What about using NTuple{100, NTuple{100, Float32} } I believe that should have the same layout in terms of bytes.

6

Are you sure? I’m doing more reading on this subject and I found this example in Julia manual:

julia> A = [1.3, -2.7, 4.4, 3.1]
4-element Array{Float64,1}:
  1.3
 -2.7
  4.4
  3.1

julia> ccall(:qsort, Cvoid, (Ptr{Cdouble}, Csize_t, Csize_t, Ptr{Cvoid}),
             A, length(A), sizeof(eltype(A)), mycompare_c)

It indicates that I should be able to read C array float[n] via Ptr{Float32}.

7

I am talking about the layout within the struct.

struct Sensors
    depth::NTuple{n, Float32}
    bottom::Float32
    top::Float32
  end

and

struct Sensors
    depth::Array{Float32, 1}
    bottom::Float32
    top::Float32
  end

Have different layouts. The first has the same layout as the following c struct.

struct Sensor {
  float depth[10000];
  float bottom;
  float top;
};

While the second one would have a pointer to the Julia array rather then storing the data within the struct

8

I see. I have a lot of flexibility, since I’m writing both Julia and C ends. I don’t have to pass a C struct, I can pass each field individually, if it’s easier. I know how to pass simple types, but I’m not sure how to deal with float[]. I just need to read from C array. What is the easiest way? NTuple{100, NTuple{100, Float32} } changes the dimensionality, which I would like to avoid.

9

I think Array{Float32,1} is easier.
Note that this should work:

n = 10_000

struct Sensors{n}
    depth::NTuple{n, Float32}
    bottom::Float32
    top::Float32
end

sr = Ref{Sensors{n}}();

But then reading from sr will be much more annoying than reading from an array.

10

How would I do it with an array? I don’t need struct Sensors, I can use individual array: float depth_c[10000] on C side. How do I read from depth_c into depth_julia::Array{Float32, 1}? Is there an Array constructor from a memory blob?

11

There is unsafe_wrap if you’re creating the array in C, but as you’re writing the C I’d recommend allocating the array in Julia and passing it to C to fill it.

12

In C, I can assure a fixed address of float depth_c[10000] over the lifetime of my program. Can I create Julia array depth_julia::Array{Float32, 1}, pass its pointer to C (pointer(depth_julia) I presume) and assume that it will be fixed and valid over the lifetime of my program? depth_julia will be fixed size and read only on Julia side.

14

Thanks. I will follow your advice and pass Julia objects to C. So far it seems to be working. I found lifetime information at: Calling C and Fortran Code · The Julia Language

1 Like
15

I’m just repeating some of what your link says, but in case not everyone clicks through:

To pass as a pointer to C, all you need is to pass depth_julia, declared as a Ptr{Float32} argument.
Then “under the hood”, it will call Base.unsafe_convert:

julia> small_depth_julia = rand(Float32, 4)
4-element Array{Float32,1}:
 0.6620523
 0.134426
 0.8403262
 0.5449269

julia> Base.unsafe_convert(Ptr{Float32}, small_depth_julia)
Ptr{Float32} @0x00007f3b9eaccbf0

Doing it this was has the advantage of protecting it from Julia’s GC.

2 Likes