# Reading a non-uniform CSV file

**URL:** <https://discourse.julialang.org/t/reading-a-non-uniform-csv-file/49886>\
**Category:** Data\
**Tags:** csv\
**Created:** [November 10, 2020, 6:15am UTC](https://discourse.julialang.org/t/reading-a-non-uniform-csv-file/49886 "2020-11-10T06:15:36Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![ash](https://avatars.discourse-cdn.com/v4/letter/a/f19dbf/32.png) [@ash](https://discourse.julialang.org/u/ash)\
**Post date:** [November 10, 2020, 6:15am UTC](https://discourse.julialang.org/t/reading-a-non-uniform-csv-file/49886/1 "2020-11-10T06:15:36Z")

</div>

I am using CSV.jl to read a file of this format:

```julia
# 60 lines of completely random unformatted crap here
    1 1 Te 5S 1 0 0.5 -0.5
       1 1 0.0000000000
       1 2 0.0024000000
       1 3 0.0002413400
       2 1 0.0241243140
       2 2 0.1234214000
       2 2 0.0979007240
    2 1 Te 5S 1 0 0.5 0.5
       1 1 0.0000000000
       1 2 0.0024000000
       1 3 0.0002413400
       2 1 0.0241243140
       2 2 0.1234214000
       2 2 0.0979007240
    3 1 Bi 5S 1 0 0.5 -0.5
       1 1 0.0000000000
       1 2 0.0024000000
       1 3 0.0002413400
       2 1 0.0241243140
       2 2 0.1234214000
       2 2 0.0979007240

```

The real file is much larger of course, 1-2 billion lines.

I am only interested in both the “header” line ` 1 1 Te 5S 1 0 0.5 -0.5` and the third column of each data section.

Currently I am reading the file like this:

```julia
function parse(filename)
    skip = 61
    ndata = 6 # kpts * bands + 1
    nsections = 3
    for n in 0:nsection-1
        header = CSV.File(filename, header=false, datarow=skip+ndata*n, limit=1, ignorerepeated=true, delim=" ");
        # Process header
        data = CSV.File(filename, header=false, datarow=skip+(ndata+1)*n, limit=ndata, threaded=false, ignorerepeated=true, delim=" ", select=[3]).Column3;
      # Process and reshape data
    end
end

```

Of course, this is very inefficient. Is there a better way of going about this? Using Pandas in python I was able to read the whole thing directly then pull out the headers and reshape the remaining data into a 3D array quite easily, but I haven’t had much success doing that in Julia.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [November 10, 2020, 8:40am UTC](https://discourse.julialang.org/t/reading-a-non-uniform-csv-file/49886/2 "2020-11-10T08:40:48Z")

</div>

I would consider just using primitives, eg read the lines with `eachline`, `split` on ` `, and then `parse` the fields.