Anyone can help explain about these statements?

Specific Domains Data
1

I am trying to find out an issue that causes julia to freeze when my large script uses with new releases of CSV and Tables. I tried to cook up simple demo code but none could reproduce the issue. So I tried to dig into the packages to see where it freezes. It happens in the following code neighborhood:

struct Schema{names, types} end
Schema(names::Tuple{Vararg{Symbol}}, types::Type{T}) where {T <: Tuple} = Schema{names, T}()
Schema(::Type{NamedTuple{names, types}}) where {names, types} = Schema{names, types}()
Schema(names, ::Nothing) = Schema{Tuple(map(Symbol, names)), nothing}()
# Schema(names, types) = Schema{Tuple(map(Symbol, names)), Tuple{types...}}()
function Schema(names, types)
    println("Tables Schema 1: ", names)
    println("Tables Schema 2: ", types)
    println("Tables Schema 3: ", Tuple(map(Symbol, names)))
    println("Tables Schema 4: ", Tuple{types...})
    Schema{Tuple(map(Symbol, names)), Tuple{types...}}()
end

It freezes at this statement:
Schema{Tuple(map(Symbol, names)), Tuple{types...}}()

I don’t quite understand about this and where further the execution goes. Can anyone please explain it for me?

2

This works fine here. A MWE that doesn’t work could help.

julia> struct Schema{names, types} end

julia> Schema(names::Tuple{Vararg{Symbol}}, types::Type{T}) where {T <: Tuple} = Schema{names, T}()
Schema

julia> Schema(::Type{NamedTuple{names, types}}) where {names, types} = Schema{names, types}()
Schema

julia> Schema(names, ::Nothing) = Schema{Tuple(map(Symbol, names)), nothing}()
Schema

julia> # Schema(names, types) = Schema{Tuple(map(Symbol, names)), Tuple{types...}}()
       function Schema(names, types)
           println("Tables Schema 1: ", names)
           println("Tables Schema 2: ", types)
           println("Tables Schema 3: ", Tuple(map(Symbol, names)))
           println("Tables Schema 4: ", Tuple{types...})
           Schema{Tuple(map(Symbol, names)), Tuple{types...}}()
       end
Schema

julia> Schema((:a, :b, :c), (Int, Float64, String))
Tables Schema 1: (:a, :b, :c)
Tables Schema 2: (Int64, Float64, String)
Tables Schema 3: (:a, :b, :c)
Tables Schema 4: Tuple{Int64,Float64,String}
Schema{(:a, :b, :c),Tuple{Int64,Float64,String}}()
3

What does this do?

4

It is a constructor. It makes an instance of:

struct Schema{names, types} end

where names is Tuple(map(Symbol, names)) and types is Tuple{types…}.

5

That’s what I thought, but in what way can it not finish? In your simple case, it definitely has no problem, but it stuck in my larger script. I can’t understand.

6

Does it always freeze (when you retry or re start the program)? Is it always in the same place?
I have seen issues where julia would freeze (on Windows) until a command (such as enter) is given to the REPL.

7

Yes, in my large script the freeze happen 100% of the time at the same location with the same types of files (not the same file), although as I said I can’t reproduce it in demo script. I have more descriptions in this filed issue here

8

My guess is that names and types are very long in your case, and that this becomes a problem when creating the Schema type.

9

In my test, it actually did occur that when I reduce the columns to 20 from like 35 (about which I reported in the issue), then it works through. How come can there be this kind of problem? What’s limiting Schema in this? And, it is also strange that my simple demo code has no problem reading in the files, the problem only occurs with my larger script.

10

I’m not sure why it would be having a hard time constructing the Schema object here; I’ve done tons of tests where I’m working on datasets w/ up to thousands of columns and they do fine (I mean, they’re slower than small datasets, but it doesn’t hang). Can you try running your script that hangs on this branch and see if that makes a difference? It seems to be like some kind of compiler bug or something, so I wonder if avoiding splatting helps at all. You can pull that code by doing pkg> add Tables#jq/schema.

11

Tried that branch. It made no difference for me.

12

We’re going to have to figure out a way for me to reproduce the issue then; can you share the script/data w/ me privately somehow? As I’ve mentioned on the issue and elsewhere, I’ve run hundreds of tests on much wider datasets and never seen things hang like you’re describing.

13

I am very glad to have reproduced the issue with a test code. It has to do with possible conflicts with Plots/GR. Just uploaded the code and data to the issue page. Hopefully this can be fixed easily.