# Ingesting data to JuliaDB without .csv files

**URL:** <https://discourse.julialang.org/t/ingesting-data-to-juliadb-without-csv-files/11347>\
**Category:** Data\
**Tags:** question\
**Created:** [June 1, 2018, 6:50pm UTC](https://discourse.julialang.org/t/ingesting-data-to-juliadb-without-csv-files/11347 "2018-06-01T18:50:34Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![ElOceanografo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/eloceanografo/32/624_2.png) [@ElOceanografo](https://discourse.julialang.org/u/ElOceanografo)\
**Post date:** [June 1, 2018, 6:50pm UTC](https://discourse.julialang.org/t/ingesting-data-to-juliadb-without-csv-files/11347/1 "2018-06-01T18:50:34Z")

</div>

I’ve got a large dataset of several million short audio clips (i.e., waveforms in 1D arrays, plus metadata). Total size is about 500 GB, stored across 140,000 .jld files. I need to go through and calculate a set of summary features from each audio clip, which will then be used for clustering/classification/etc.

I haven’t used `JuliaDB` and `OnlineStats` yet, but they seem like the natural way to handle the big table of features. As I understand it, the simplest/naive way to get these data into JuliaDB would be:

1. Load each .jld file, extract features from each clip, save the resulting table in a .csv file
2. Ingest all .csv files into `JuliaDB`
3. Re-save the dataset in binary format

My questions:  
Is there a way to build the binary-format database without writing and reading all those intermediate .csv files?  
Are there other tools or approaches I should be looking at?

Thanks in advance!

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [June 2, 2018, 12:01pm UTC](https://discourse.julialang.org/t/ingesting-data-to-juliadb-without-csv-files/11347/2 "2018-06-02T12:01:06Z")

</div>

1.4e5 data points should fit easily in memory if you are just after summary statistics, so I would just iterate through the files, extract statistics, `push!`ing them into `Vector`s, which I would then perhaps convert to a `DataFrame`.

---

<div class="post-metadata">

**Author:** ![piever](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/piever/32/1815_2.png) [@piever](https://discourse.julialang.org/u/piever)\
**Post date:** [June 2, 2018, 2:42pm UTC](https://discourse.julialang.org/t/ingesting-data-to-juliadb-without-csv-files/11347/3 "2018-06-02T14:42:28Z")

</div>

The simplest is to use `map`:

```julia
julia> t = table(@NT(filename = ["a.jld2", "b.jld2"]));
julia> f(filename) = @NT(filename=filename, length=1, frequency=2.3);

julia> map(f, t, select=:filename)
Table with 2 rows, 3 columns:
filename length frequency
───────────────────────────
"a.jld2" 1 2.3
"b.jld2" 1 2.3

```

Of course in your case `f` should be the function that loads the file and computes your relevant features. Note that if `t` is stored as a distributed table across several processors, `map` will be done in parallel.

---

<div class="post-metadata">

**Author:** ![ElOceanografo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/eloceanografo/32/624_2.png) [@ElOceanografo](https://discourse.julialang.org/u/ElOceanografo)\
**Post date:** [June 5, 2018, 2:18am UTC](https://discourse.julialang.org/t/ingesting-data-to-juliadb-without-csv-files/11347/4 "2018-06-05T02:18:29Z")

</div>

Thanks for the responses. I guess I didn’t make it clear, but there are 1.4e5 .jld files, each of which contains a variable number of waveforms…most between 10 and 1000. The table of summary features may well fit in RAM, but I’d hate to try processing them all and then get an out-of-memory error 95% of the way through 😅

I’ve ended up splitting the list of files so I can process and save one medium-sized chunk at a time. I’ll be back in a day or two if it doesn’t work!

---

<div class="post-metadata">

**Author:** ![jstrube](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jstrube/32/525_2.png) [@jstrube](https://discourse.julialang.org/u/jstrube)\
**Post date:** [August 30, 2018, 5:52am UTC](https://discourse.julialang.org/t/ingesting-data-to-juliadb-without-csv-files/11347/5 "2018-08-30T05:52:20Z")

</div>

I’m a bit stuck here. I have a pandas data store that looks like this:

```julia
Dict{String,Any} with 6 entries:
  "axis1" => [0, 1, 2, 3, 4, 5, 6, 7, 8, 9 … 29592, 29593, 29594, 295…
  "axis0" => String["run", "event", "moduleID", "pixelID", "time"]
  "block0_values" => [30.4698 31.4345 … 42.8309 30.2701]
  "block1_items" => String["run", "event", "moduleID", "pixelID"]
  "block1_values" => [4118 4118 … 4118 4118; 39248 39248 … 78452 78452; 1 1 … 1…
  "block0_items" => String["time"]

```

What I’ve come up so far is this:

```julia
function readDigitsTableFromHDF5_File(fname)
    d = h5read(fname, "digits")
    names = [(Symbol(i) for i in d["block1_items"])...,collect(Symbol(i) for i in d["block0_items"])...]
    data = [(d["block1_values"][i,:] for i in 1:length(d["block1_items"]))...,(d["block0_values"][i,:] for i in 1:length(d["block0_items"]))...]
    @NT(names)(data)
end

```

But this doesn’t work, a map over filenames returns this shape:

```julia
Table with 1 rows, 1 columns:
Columns:
# colname type
─────────────────────────────────────
1 names Array{Array{Float64,1},1}

```

This isn’t what I was hoping for. What is the right syntax to generate the NamedTuple?