# DataFrame join error

**URL:** <https://discourse.julialang.org/t/dataframe-join-error/32141>\
**Category:** Data\
**Created:** [December 10, 2019, 12:32pm UTC](https://discourse.julialang.org/t/dataframe-join-error/32141 "2019-12-10T12:32:38Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [December 10, 2019, 12:32pm UTC](https://discourse.julialang.org/t/dataframe-join-error/32141/1 "2019-12-10T12:32:38Z")

</div>

I have

```julia
julia> timeuse = join(people[:, [:idhh, :idpers, :gender]], timeuse;
       on = [:idhh, :idpers], kind = :inner)
ERROR: DimensionMismatch("destination must have length equal to sums of concatenated vectors")
Stacktrace:
 [1] vcat_copyto!(::Array{Int64,1}, ::SubArray{Int64,1,Array{Int64,1},Tuple{Array{Int64,1}},false}, ::Vararg{SubArray{Int64,1,Array{Int64,1},Tuple{Array{Int64,1}},false},N} where N) at /home/tamas/.julia/packages/LazyArrays/14GOk/src/lazyconcat.jl:158
 [2] copyto!(::Array{Int64,1}, ::SubArray{Int64,1,LazyArrays.ApplyArray{Int64,1,typeof(vcat),NTuple{8,Array{Int64,1}}},Tuple{Array{Int64,1}},false}) at /home/tamas/.julia/packages/LazyArrays/14GOk/src/lazyconcat.jl:579
 [3] #compose_joined_table#286(::Bool, ::typeof(DataFrames.compose_joined_table), ::DataFrames.DataFrameJoiner{DataFrame,DataFrame}, ::Symbol, ::DataFrames.RowIndexMap, ::DataFrames.RowIndexMap, ::DataFrames.RowIndexMap, ::DataFrames.RowIndexMap) at /home/tamas/.julia/packages/DataFrames/uPgZV/src/abstractdataframe/join.jl:106
 [4] #compose_joined_table at ./none:0 [inlined]
 [5] #join#294(::Array{Symbol,1}, ::Symbol, ::Bool, ::Nothing, ::Tuple{Bool,Bool}, ::typeof(join), ::DataFrame, ::DataFrame) at /home/tamas/.julia/packages/DataFrames/uPgZV/src/abstractdataframe/join.jl:364
 [6] (::Base.var"#kw##join")(::NamedTuple{(:on, :kind),Tuple{Array{Symbol,1},Symbol}}, ::typeof(join), ::DataFrame, ::DataFrame) at ./none:0
 [7] top-level scope at REPL[136]:1

```

while

```julia
timeuse = join(people[1:size(people, 1), [:idhh, :idpers, :gender]], timeuse;
               on = [:idhh, :idpers], kind = :inner)

```

works fine (the only difference is `:` vs `1:size(people, 1)`). I have not been able to produce an MWE, and I am sorry but I am not able to share the data (it is confidential). The `:` version worked fine in 0.19.

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [December 10, 2019, 1:25pm UTC](https://discourse.julialang.org/t/dataframe-join-error/32141/2 "2019-12-10T13:25:51Z")

</div>

I did not think we have changed this between 0.19 and 0.20. But the reason is that `df[:, cols]` uses `copy` to copy columns, while `df[rows, cols]` uses vector subsetting, i.e. for each column `c` it does `c[rows]`.

For columns being a `Vector` this should be equivalent. For custom column types this might not be equivalent. For instance a reason might be that you use [https://github.com/JuliaArrays/LazyArrays.jl](https://github.com/JuliaArrays/LazyArrays.jl) from CSV.jl (this is a known bug that should be fixed in CSV.jl, see [https://github.com/JuliaData/CSV.jl/issues/539](https://github.com/JuliaData/CSV.jl/issues/539)). Is this the reason?

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [December 10, 2019, 1:45pm UTC](https://discourse.julialang.org/t/dataframe-join-error/32141/3 "2019-12-10T13:45:31Z")

</div>

Thanks. I was using `CSV.File` from the beginning. Indeed

```julia
 DataFrame(CSV.File("some_path.csv"); threaded = false); copycols = true)

```

fixes the issue.

What is more worrying is that unless I apply both options above, `join` results in **data corruption** (for some colums, values are exchanged for the same join key in random order). I tried making an MWE, but for some reason it only kicks in if read the data from CSV, and it is large enough.

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [December 10, 2019, 4:01pm UTC](https://discourse.julialang.org/t/dataframe-join-error/32141/4 "2019-12-10T16:01:25Z")

</div>

It seems that the problem is with how arrays from LazyArrays.jl work, probably within `compose_joined_table`. The code in this part has not been touched for over 2 years, so something might get off sync. @quinnj will probably fix CSV.jl soon, so this should not surface in common usage, but it would be great to have some MWE as probably either LazyArrays.jl or DataFrames.jl need fixing in general. Thank you for reporting this.

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [December 10, 2019, 8:34pm UTC](https://discourse.julialang.org/t/dataframe-join-error/32141/5 "2019-12-10T20:34:51Z")

</div>

@Tamas_Papp - if you are unable to share the file (or some other file that reproduces the error; I have tried to create one and was unable). Then can you please put some value tracing code before line [https://github.com/JuliaData/DataFrames.jl/blob/master/src/abstractdataframe/join.jl#L106](https://github.com/JuliaData/DataFrames.jl/blob/master/src/abstractdataframe/join.jl#L106).  
To make sure what is the type and size of `cols[i]` and what is the type, size and contents of `view(col, all_orig_left_ixs)` and if they are the same as should be expected. Thank you!

---

<div class="post-metadata">

**Author:** ![ExpandingMan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/expandingman/32/866_2.png) [@ExpandingMan](https://discourse.julialang.org/u/ExpandingMan)\
**Post date:** [December 10, 2019, 9:25pm UTC](https://discourse.julialang.org/t/dataframe-join-error/32141/6 "2019-12-10T21:25:31Z")

</div>

I also can’t reproduce @Tamas_Papp’s issue. My guess would be that it only occurs for a very specific array type. Could you give what version of `LazyArray`s you are using?

It would be nice if we could split off the conversation about this error to another thread.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [December 11, 2019, 9:47am UTC](https://discourse.julialang.org/t/dataframe-join-error/32141/7 "2019-12-11T09:47:31Z")

</div>

Thanks for moving the topic, I will dig into this.