# Problem using DataFrames master

**URL:** <https://discourse.julialang.org/t/problem-using-dataframes-master/17972>\
**Category:** Tooling\
**Tags:** question\
**Created:** [November 25, 2018, 10:42am UTC](https://discourse.julialang.org/t/problem-using-dataframes-master/17972 "2018-11-25T10:42:25Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Ajaychat3](https://avatars.discourse-cdn.com/v4/letter/a/ecd19e/32.png) [@Ajaychat3](https://discourse.julialang.org/u/Ajaychat3)\
**Post date:** [November 25, 2018, 10:42am UTC](https://discourse.julialang.org/t/problem-using-dataframes-master/17972/1 "2018-11-25T10:42:25Z")

</div>

Taking a cue from the following posts, I upgraded to DataFrames latest master branch hoping to see some speed improvements in reading the csv data but got an error. The excerpts of the code where the issue occurs is given below which works fine with the current stable release. Not a showstopper for me as by reveritng back to the stable version, I am able to get the desired result.

> [@Reading Data Is Still Too Slow](https://discourse.julialang.org/t/reading-data-is-still-too-slow/17919/2):
>
> Could you try [CSVFiles.jl](https://github.com/queryverse/CSVFiles.jl) on your dataset? using DataFrames, CSVFiles, FileIO crsp = load(File(format"CSV", "crspdaily-clean.csv.gz"), nastrings=["NA", "NaN", ""]) |\> DataFrame # or for already uncompressed file crsp = load("crspdaily-clean.csv", nastrings=["NA", "NaN", ""]) |\> DataFrame [This](https://discourse.julialang.org/t/textparse-jl-is-fast-again/16664) indicates that [TextParse.jl](https://github.com/JuliaComputing/TextParse.jl) is often faster (or en par) than [CSV.jl](https://github.com/JuliaData/CSV.jl). [CSVFiles.jl](https://github.com/queryverse/CSVFiles.jl) uses [TextParse.jl](https://github.com/JuliaComputing/TextParse.jl) under the hood. The benchmarks I posted there showed an additional overhead when using [CSVFiles.jl](https://github.com/queryverse/CSVFiles.jl) …

> [@Reading Data Is Still Too Slow](https://discourse.julialang.org/t/reading-data-is-still-too-slow/17919/26):
>
> I tried to load a feather file now that seems roughly similar to the CRSP file described [here](http://julia.cookbook.tips/doku.php?id=fileformats): 90 million rows, a mix of Int and Float64 columns. The file is about 8.5 GB on disc. With the current released version of [DataFrames.jl](https://github.com/JuliaData/DataFrames.jl), it takes about 80 seconds to load on my system (I do have a very fast system slight_smile ). When I use the master branch of [DataFrames.j](https://github.com/JuliaData/DataFrames.jl), it takes somewhere between 7-15 seconds to load. All of these numbers are for [FeatherFiles.jl](https://github.com/queryverse/FeatherFiles.jl). So I suspect (or hope wink )…

```julia-auto
using MLDataUtils #MLDataPattern,
@views X,y=undersample((X_train,y_train),shuffle=false);
# X_resamp,y_resamp=oversample((X_train,y_train),fraction=0.2);

@views X = convert(Array{Float64}, X);
@views y = convert(Array{Int64}, y);
@views X_tst= convert(Array{Float64}, X_test);
@views y_tst = convert(Array{Int64}, y_test);

@views X_train0 = convert(Array{Float64}, X_train);
@views y_train0 = convert(Array{Int64}, y_train);

println(size(X),size(y))

print(size(X_test),size(y_test))

┌ Info: Recompiling stale cache file C:\Users\chatura\.julia\compiled\v1.0\MLDataUtils\CQWB9.ji for MLDataUtils [cc2ba9b6-d476-5e6d-8eaf-a92d5412d41d]
└ @ Base loading.jl:1190
MethodError: getindex(::DataFrames.DataFrameColumns{SubDataFrame{Array{Int64,1}},AbstractArray{T,1} where T}, ::Int64) is ambiguous. Candidates:
  getindex(itr::DataFrames.DataFrameColumns{#s383,AbstractArray{T,1} where T} where #s383<:SubDataFrame, j) in DataFrames at C:\Users\chatura\.julia\packages\DataFrames\VGojy\src\deprecated.jl:1428
  getindex(itr::DataFrames.DataFrameColumns{#s46,AbstractArray{T,1} where T} where #s46<:AbstractDataFrame, j::Int64) in DataFrames at C:\Users\chatura\.julia\packages\DataFrames\VGojy\src\abstractdataframe\iteration.jl:135
Possible fix, define
  getindex(::DataFrames.DataFrameColumns{#s383,AbstractArray{T,1} where T} where #s383<:SubDataFrame, ::Int64)

Stacktrace:
 [1] iterate(::DataFrames.DataFrameColumns{SubDataFrame{Array{Int64,1}},AbstractArray{T,1} where T}, ::Tuple{Base.OneTo{Int64}}) at .\abstractarray.jl:838
 [2] iterate at .\abstractarray.jl:836 [inlined]
 [3] zip_iterate at .\iterators.jl:304 [inlined]
 [4] iterate at .\iterators.jl:320 [inlined]
 [5] convert at C:\Users\chatura\.julia\packages\DataFrames\VGojy\src\abstractdataframe\abstractdataframe.jl:802 [inlined]
 [6] convert(::Type{Array{Float64,N} where N}, ::SubDataFrame{Array{Int64,1}}) at C:\Users\chatura\.julia\packages\DataFrames\VGojy\src\abstractdataframe\abstractdataframe.jl:796
 [7] top-level scope at In[7]:3

```

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [November 25, 2018, 2:39pm UTC](https://discourse.julialang.org/t/problem-using-dataframes-master/17972/2 "2018-11-25T14:39:38Z")

</div>

I think you should file an issue. This looks like a missing method after the recent `getindex` restructuring. Can you put together a copy-and-pasteable MWE?

---

<div class="post-metadata">

**Author:** ![Ajaychat3](https://avatars.discourse-cdn.com/v4/letter/a/ecd19e/32.png) [@Ajaychat3](https://discourse.julialang.org/u/Ajaychat3)\
**Post date:** [November 25, 2018, 4:03pm UTC](https://discourse.julialang.org/t/problem-using-dataframes-master/17972/3 "2018-11-25T16:03:33Z")

</div>

Sure I would do that.

---

<div class="post-metadata">

**Author:** ![nalimilan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nalimilan/32/147_2.png) [@nalimilan](https://discourse.julialang.org/u/nalimilan)\
**Post date:** [November 25, 2018, 8:44pm UTC](https://discourse.julialang.org/t/problem-using-dataframes-master/17972/4 "2018-11-25T20:44:23Z")

</div>

I think that’s fixed by [https://github.com/JuliaData/DataFrames.jl/pull/1601/files#r236036652](https://github.com/JuliaData/DataFrames.jl/pull/1601/files#r236036652).

---

<div class="post-metadata">

**Author:** ![Ajaychat3](https://avatars.discourse-cdn.com/v4/letter/a/ecd19e/32.png) [@Ajaychat3](https://discourse.julialang.org/u/Ajaychat3)\
**Post date:** [November 26, 2018, 5:12am UTC](https://discourse.julialang.org/t/problem-using-dataframes-master/17972/5 "2018-11-26T05:12:34Z")

</div>

I don’t find so as I updated the DataFrames master package today and face the same issue. Let me know if should file an issue.

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [November 26, 2018, 5:14am UTC](https://discourse.julialang.org/t/problem-using-dataframes-master/17972/6 "2018-11-26T05:14:44Z")

</div>

That PR hasn’t been merged yet. You can check out the branch if you want and test it out. However it should be merged soon. Sorry for this! Welcome to the life of being on `master`