# Using JuliaDB to read in all the CSVs fails, but not when reading them singly

**URL:** <https://discourse.julialang.org/t/using-juliadb-to-read-in-all-the-csvs-fails-but-not-when-reading-them-singly/21967>\
**Category:** General Usage\
**Tags:** data\
**Created:** [March 17, 2019, 4:29pm UTC](https://discourse.julialang.org/t/using-juliadb-to-read-in-all-the-csvs-fails-but-not-when-reading-them-singly/21967 "2019-03-17T16:29:09Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![natemcintosh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/natemcintosh/32/9021_2.png) [@natemcintosh](https://discourse.julialang.org/u/natemcintosh)\
**Post date:** [March 17, 2019, 4:29pm UTC](https://discourse.julialang.org/t/using-juliadb-to-read-in-all-the-csvs-fails-but-not-when-reading-them-singly/21967/1 "2019-03-17T16:29:09Z")

</div>

I have a set of [CSVs](https://rredc.nrel.gov/solar/old_data/nsrdb/1991-2005/tmy3/), about 1.7GB total size. (I’m attempting to compete in the [r/dataisbeautiful visualization competition for this month](https://www.reddit.com/r/dataisbeautiful/comments/axknia/battle_dataviz_battle_for_the_month_of_march_2019/)).

When trying to load all the files with `loadtable()`, I run into some problems (running with multiple cores and with one core).

```julia
table = loadtable("/Users/mcintna1/Documents/DataSets/dataisbeautiful_march/",
                output = "data",
                indexcols = [1], 
                skiplines_begin = 1, 
                distributed = true)

```

This gives me the following error

```julia
Error parsing /Users/mcintna1/Documents/DataSets/dataisbeautiful_march/julia_analysis.ipynb

Couldn't split line, error at char 11:
"cells": [
_______ ^
error(::String) at ./error.jl:33
quotedsplit(::TextParse.VectorBackedUTF8String, ::TextParse.LocalOpts{UInt8,UInt8,UInt8}, ::Bool, ::Int64, ::Int64) at /Users/mcintna1/.julia/packages/TextParse/o3nmV/src/csv.jl:668
readcolnames(::TextParse.VectorBackedUTF8String, ::TextParse.LocalOpts{UInt8,UInt8,UInt8}, ::Int64, ::Array{String,1}) at /Users/mcintna1/.julia/packages/TextParse/o3nmV/src/csv.jl:462
#_csvread_internal#26(::Bool, ::Char, ::Char, ::Type, ::Type, ::Bool, ::Int64, ::Array{String,1}, ::Nothing, ::Int64, ::Array{String,1}, ::Bool, ::Array{String,1}, ::Array{String,1}, ::OrderedCollections.OrderedDict{Union{Int64, String},AbstractArray{T,1} where T}, ::Int64, ::Dict{Any,Any}, ::Array{Any,1}, ::String, ::Int64, ::typeof(TextParse._csvread_internal), ::TextParse.VectorBackedUTF8String, ::Char) at /Users/mcintna1/.julia/packages/TextParse/o3nmV/src/csv.jl:209
(::getfield(TextParse, Symbol("#kw##_csvread_internal")))(::NamedTuple{(:filename, :rowno, :colspool, :prevheaders, :noresize, :prev_parsers, :samecols, :skiplines_begin),Tuple{String,Int64,OrderedCollections.OrderedDict{Union{Int64, String},AbstractArray{T,1} where T},Array{String,1},Bool,Dict{Any,Any},Array{String,1},Int64}}, ::typeof(TextParse._csvread_internal), ::TextParse.VectorBackedUTF8String, ::Char) at ./none:0
(::getfield(TextParse, Symbol("##22#24")){Base.Iterators.Pairs{Symbol,Any,NTuple{7,Symbol},NamedTuple{(:rowno, :colspool, :prevheaders, :noresize, :prev_parsers, :samecols, :skiplines_begin),Tuple{Int64,OrderedCollections.OrderedDict{Union{Int64, String},AbstractArray{T,1} where T},Array{String,1},Bool,Dict{Any,Any},Array{String,1},Int64}}},String,Char})(::IOStream) at /Users/mcintna1/.julia/packages/TextParse/o3nmV/src/csv.jl:108
#open#310(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::getfield(TextParse, Symbol("##22#24")){Base.Iterators.Pairs{Symbol,Any,NTuple{7,Symbol},NamedTuple{(:rowno, :colspool, :prevheaders, :noresize, :prev_parsers, :samecols, :skiplines_begin),Tuple{Int64,OrderedCollections.OrderedDict{Union{Int64, String},AbstractArray{T,1} where T},Array{String,1},Bool,Dict{Any,Any},Array{String,1},Int64}}},String,Char}, ::String, ::Vararg{String,N} where N) at ./iostream.jl:369
open at ./iostream.jl:367 [inlined]
#_csvread_f#20 at /Users/mcintna1/.julia/packages/TextParse/o3nmV/src/csv.jl:105 [inlined]
(::getfield(TextParse, Symbol("#kw##_csvread_f")))(::NamedTuple{(:rowno, :colspool, :prevheaders, :noresize, :prev_parsers, :samecols, :skiplines_begin),Tuple{Int64,OrderedCollections.OrderedDict{Union{Int64, String},AbstractArray{T,1} where T},Array{String,1},Bool,Dict{Any,Any},Array{String,1},Int64}}, ::typeof(TextParse._csvread_f), ::String, ::Char) at ./none:0
#csvread#25(::Base.Iterators.Pairs{Symbol,Any,Tuple{Symbol,Symbol},NamedTuple{(:samecols, :skiplines_begin),Tuple{Array{String,1},Int64}}}, ::Function, ::Array{String,1}, ::Char) at /Users/mcintna1/.julia/packages/TextParse/o3nmV/src/csv.jl:140
#csvread at ./none:0 [inlined]
#_loadtable_serial#3(::Char, ::Array{Int64,1}, ::Nothing, ::Nothing, ::Nothing, ::Bool, ::Bool, ::typeof(csvread), ::Base.Iterators.Pairs{Symbol,Int64,Tuple{Symbol},NamedTuple{(:skiplines_begin,),Tuple{Int64}}}, ::typeof(JuliaDB._loadtable_serial), ::UnionAll, ::Array{String,1}) at /Users/mcintna1/.julia/packages/JuliaDB/ZXPIx/src/util.jl:83
(::getfield(JuliaDB, Symbol("##190#193")){Array{Int64,1},Base.Iterators.Pairs{Symbol,Int64,Tuple{Symbol},NamedTuple{(:skiplines_begin,),Tuple{Int64}}},UnionAll})(::Array{String,1}) at ./none:0
do_task(::Dagger.Context, ::Dagger.OSProc, ::Int64, ::Function, ::Tuple{Array{String,1}}, ::Bool, ::Bool, ::Bool) at /Users/mcintna1/.julia/packages/Dagger/sdZXi/src/scheduler.jl:259
#143 at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.1/Distributed/src/remotecall.jl:339 [inlined]
run_work_thunk(::getfield(Distributed, Symbol("##143#144")){typeof(Dagger.Sch.do_task),Tuple{Dagger.Context,Dagger.OSProc,Int64,getfield(JuliaDB, Symbol("##190#193")){Array{Int64,1},Base.Iterators.Pairs{Symbol,Int64,Tuple{Symbol},NamedTuple{(:skiplines_begin,),Tuple{Int64}}},UnionAll},Tuple{Array{String,1}},Bool,Bool,Bool},Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}}, ::Bool) at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.1/Distributed/src/process_messages.jl:56
#remotecall_fetch#148(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::Function, ::Distributed.LocalProcess, ::Dagger.Context, ::Vararg{Any,N} where N) at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.1/Distributed/src/remotecall.jl:364
remotecall_fetch(::Function, ::Distributed.LocalProcess, ::Dagger.Context, ::Vararg{Any,N} where N) at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.1/Distributed/src/remotecall.jl:364
#remotecall_fetch#152(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::Function, ::Int64, ::Dagger.Context, ::Vararg{Any,N} where N) at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.1/Distributed/src/remotecall.jl:406
remotecall_fetch at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.1/Distributed/src/remotecall.jl:406 [inlined]
macro expansion at /Users/mcintna1/.julia/packages/Dagger/sdZXi/src/scheduler.jl:272 [inlined]
(::getfield(Dagger.Sch, Symbol("##13#14")){Dagger.Context,Dagger.OSProc,Int64,getfield(JuliaDB, Symbol("##190#193")){Array{Int64,1},Base.Iterators.Pairs{Symbol,Int64,Tuple{Symbol},NamedTuple{(:skiplines_begin,),Tuple{Int64}}},UnionAll},Tuple{Array{String,1}},Channel{Any},Bool,Bool,Bool})() at ./task.jl:259

Stacktrace:
 [1] compute_dag(::Dagger.Context, ::Dagger.Thunk) at /Users/mcintna1/.julia/packages/Dagger/sdZXi/src/scheduler.jl:62
 [2] compute(::Dagger.Context, ::Dagger.Thunk) at /Users/mcintna1/.julia/packages/Dagger/sdZXi/src/compute.jl:25
 [3] #fromchunks#47(::String, ::Int64, ::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::Array{Dagger.Thunk,1}) at /Users/mcintna1/.julia/packages/JuliaDB/ZXPIx/src/table.jl:141
 [4] (::getfield(JuliaDB, Symbol("#kw##fromchunks")))(::NamedTuple{(:output, :fnoffset),Tuple{String,Int64}}, ::typeof(JuliaDB.fromchunks), ::Array{Dagger.Thunk,1}) at ./none:0
 [5] #_loadtable#188(::Nothing, ::String, ::Bool, ::Array{Int64,1}, ::Bool, ::Bool, ::Base.Iterators.Pairs{Symbol,Int64,Tuple{Symbol},NamedTuple{(:skiplines_begin,),Tuple{Int64}}}, ::Function, ::Type, ::String) at /Users/mcintna1/.julia/packages/JuliaDB/ZXPIx/src/io.jl:140
 [6] #_loadtable at ./none:0 [inlined]
 [7] #loadtable#186 at /Users/mcintna1/.julia/packages/JuliaDB/ZXPIx/src/io.jl:63 [inlined]
 [8] (::getfield(JuliaDB, Symbol("#kw##loadtable")))(::NamedTuple{(:output, :indexcols, :skiplines_begin, :distributed),Tuple{String,Array{Int64,1},Int64,Bool}}, ::typeof(loadtable), ::String) at ./none:0
 [9] top-level scope at In[8]:1

```

However, when loading each table by itself, I run into no errors!

```julia
for file in CSVs
    table = loadtable(file, indexcols = [1], skiplines_begin = 1)
    println("Loaded table: $file")
end

```

Where CSVs is a vector of all the CSV file locations. It cranks through this loop, and finishes with no errors.

I can’t quite figure out where this error is coming from, or how to deal with it. I thought it might be “bad data” in one of the CSVs, but it seems to be able to read each one just fine. Is it an error in the combining part?

---

<div class="post-metadata">

**Author:** ![rdeits](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rdeits/32/286_2.png) [@rdeits](https://discourse.julialang.org/u/rdeits)\
**Post date:** [March 17, 2019, 5:50pm UTC](https://discourse.julialang.org/t/using-juliadb-to-read-in-all-the-csvs-fails-but-not-when-reading-them-singly/21967/2 "2019-03-17T17:50:49Z")

</div>

Take a look at the very first line of the error message:

> [@natemcintosh](#):
>
> > Error parsing /Users/mcintna1/Documents/DataSets/dataisbeautiful\_march/julia\_analysis.ipynb

You’re trying to parse a Jupyter notebook file (.ipynb) as if it were a data file. That’s not going to work. You’ll either need to tell `loadtable` to ignore that file or just store your CSVs in their own folder, rather than storing them with your notebook.

---

<div class="post-metadata">

**Author:** ![natemcintosh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/natemcintosh/32/9021_2.png) [@natemcintosh](https://discourse.julialang.org/u/natemcintosh)\
**Post date:** [March 18, 2019, 6:13pm UTC](https://discourse.julialang.org/t/using-juliadb-to-read-in-all-the-csvs-fails-but-not-when-reading-them-singly/21967/3 "2019-03-18T18:13:53Z")

</div>

Ohhhh, so when I handed in the directory to `loadtable`, it also tried to read in my .ipynb file as well as the CSVs!

Thank you! Glad it was a simple fix