ERROR: UndefVarError: Compiler not defined

General Usage
1

Hi all,

I’m getting the following error:

ERROR: UndefVarError: Compiler not defined
Stacktrace:
 [1] collect_to_with_first! at ./array.jl:689 [inlined]
 [2] collect(::Base.Generator{UnitRange{Int64},IGGen.var"#126#127"{MyOwnType1,MyOwnType2}}) at ./array.jl:670
 [3] transform(::MyOwnType1, ::MyOwnType2) at /home/colin/Cloud/Codebase/Julia/Modules/IGGen/src/types.jl:522
 [4] top-level scope at REPL[45]:53

Here’s the referenced line (in my file types.jl line 522) that is triggering the error:

x = [ transform(so.st, get_x(rd, so.tt, n), rd.dxtsvec[so.tsx], rd.dxtsvec[so.tsxbi]) for n = 1:length(rd) ]

Now for the really bad news. I can’t get an MWE on this one. This is part of a long routine (runs for 10 hours or so) where this line will get called many thousands of times. I’ve run this line for the specific values of MyOwnType1 and MyOwnType2 that cause the error separately from the long routine and I do not get the error. I can also start the long routine partway through and when I do this I get much further through it before the error is triggered (and for different types than MyOwnType1 and MyOwnType2).

All this combined suggests some sort of buffer is overflowing. I’ve tried doing the run while monitoring system RAM and it never gets above 50%.

EDIT: I can actually get the entire routine to run fine if I break it up into 4 equal parts and run them separately. But if I try to do it all in one go I get the error.

Does anyone else have any experience with anything like this?

Cheers and thanks in advance,

Colin

2

I’m only speculating, but here’s what I think is happening:

  1. Your transform function uses a generator to create the vector x.
  2. In order to infer the element type of x, the generator calls Core.Compiler.return_type
  3. Core.Compiler.return_type looks at the function in the generator. This function calls transform.
  4. The recursion sometimes confuses the compiler somehow? It tries to call itself? (I have no idea.)

A work-around would be to initialize x as a Vector{Any} and then fill it using a for-loop.

Which Julia version are you using?

3

Speculation is most welcome, thank you. I should have included version info. I’m still on v1.4.2 for now.

It is actually certain that x will always be Vector{Float64}. However, that may not be obvious to the compiler. I’ll try initializing it with that and filling it in with an explicit loop. I’ll need to leave it to run overnight, so I won’t be able to report back here until tomorrow (Australia time).

4

I switched the line as suggested to a pre-allocated explicit loop. I’m still getting an error although it is a different one in a different place this time. Interestingly, I can actually get the whole routine to run just by breaking it up into 4 separate pieces. Ultimately the routine is just a loop over a vector that prints some output as it goes. If I split the vector up into 4 pieces and run it on each piece it works fine. If I try to do the whole vector in one hit I get an error. The error I’m getting now is this:

ERROR: TypeError: in new, expected Int64, got SigObj{MyType1,MyType2,MyType3,MyType4}
Stacktrace:
 [1] indexed_iterate(::Tuple{Int64,SigObj{MyType1,MyType2,MyType3,MyType4}}, ::Int64, ::Int64) at ./tuple.jl:81
 [2] top-level scope at REPL[45]:52

Very confusing. I’m not too concerned now, since the routine was a one-time thing, so I’ve got the results I need. However, I’m happy to pursue it further if we think it might be a potential bug in Base. It certainly is odd that I can get an error-free run just by splitting it up into 4 pieces…

5

It sounds a bit like the compiler has been convinced into generating bad code. It’s hard to guess why (compiler bug? abuse of Core.Compiler.return_types or @pure? incorrectly written @generated function? misuse of unsafe_pointer_from_objref? missing GC.@preserve? etc.)

Is your code public, and would you be willing to make a reproducible example with --bug-report=rr (Julia 1.5 Feature Preview: Time Traveling (Linux) Bug Reporting)?

6

Thanks for responding. While much of my code is either already public, or can be made public, unfortunately this particular routine cannot be made public (it contains proprietary trading knowledge/algorithms - I even needed to change the type names in the code I posted above).

None of these are triggering anything in my brain alas.

The only feature of the code that is perhaps a little unusual compared to my usual code is that it compiles a lot of methods via multiple dispatch. For example, in SigObj{MyType1,MyType2,MyType3,MyType4}}, each of those four types are subtypes of a different abstract type, and each of those four abstract types have on the order of 20 to 30 subtypes, and code may need to compile methods for every cross-combination of those subtypes - possibly on the order of 200,000 methods.

Perhaps I’m encountering some hard limit on the method table that is manifesting in an unusual way? It would certainly explain why everything works fine when I split it up into four bits, since each of those four bits would probably only be defining on the order of 50,000 methods instead of 200,000.

7

Ah, yeah, the internals probably get very little testing at that size and might be overflowing some internal counter. You might want to try a julia-debug build and submit specific issues (or pull requests) if it asserts when something goes wrong.

8

I’ll keep that in mind thanks.

This has also prompted a wider question in my mind about how to structure the code when there are a potentially large number of type intersections. The more I think about it, I suspect I’ve structured my code badly such that 200,000 methods are being generated, even though many of those methods will be functionally identical. I’ll do a bit of research on this myself and might even be able to come up with an MWE for this issue based on the assumption that this is indeed the source of the problem.

Thanks again.