# Parsing strange CSV files

**URL:** <https://discourse.julialang.org/t/parsing-strange-csv-files/35907>\
**Category:** General Usage\
**Tags:** question, csv\
**Created:** [March 12, 2020, 10:56pm UTC](https://discourse.julialang.org/t/parsing-strange-csv-files/35907 "2020-03-12T22:56:06Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![tbeason](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tbeason/32/15898_2.png) [@tbeason](https://discourse.julialang.org/u/tbeason)\
**Post date:** [March 12, 2020, 10:56pm UTC](https://discourse.julialang.org/t/parsing-strange-csv-files/35907/1 "2020-03-12T22:56:06Z")

</div>

I am trying to parse some CSV files that have some extra stuff and actually include multiple different tables. About the only thing that is standardized is that the tables within the file always have the same number of columns. Here is a test file:

```julia
This is an annoying way to make csv files.

Hello, here is a silly place to put a line with a comma.

,col2,col3
20010501,1.0,-0.5
20010502,0.1,0.5

Surpise: there is another table here
,col2,col3
20200101,1.0,-0.5
20200102,0.1,0.5

Made by me.

```

I’m looking for some advice on the best way to parse this programmatically. I do know that if I do something like `CSV.read(file;skipto=5,limit=3,header=5)` then I can correctly get the first table in this case. However, how could I get those line numbers ahead of time?

---

<div class="post-metadata">

**Author:** ![Pbellive](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pbellive/32/3604_2.png) [@Pbellive](https://discourse.julialang.org/u/Pbellive)\
**Post date:** [March 12, 2020, 11:28pm UTC](https://discourse.julialang.org/t/parsing-strange-csv-files/35907/2 "2020-03-12T23:28:22Z")

</div>

I have no idea whether or not there is a nice prepackaged solution for a problem like this. Others may know better. The first idea off the top of my head for dealing with this (assuming that all the tables in the file have the same, known, number of columns:

Write your own code that reads the file line by line, splits each line by commas and identifies the start of a table by the number of items returned after splitting by commas. Then you could add that line and subsequent lines identified as part of the same table to an in memory string. Then use the `CSV` package to parse your table stored as a single long string in memory. Then keep reading the file line by line until you find another table. Then repeat.

Just a thought off the top of my head.

---

<div class="post-metadata">

**Author:** ![tbeason](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tbeason/32/15898_2.png) [@tbeason](https://discourse.julialang.org/u/tbeason)\
**Post date:** [March 12, 2020, 11:39pm UTC](https://discourse.julialang.org/t/parsing-strange-csv-files/35907/3 "2020-03-12T23:39:39Z")

</div>

The files are not extremely large so I can do something like

```julia
stringarray = readlines(file,keep=true)

```

without issue. In this case, `stringarray[5:7]` contains the first table, but I cannot pass this to `CSV` and `string(stringarray[5:7]...)` also does not work.

---

<div class="post-metadata">

**Author:** ![tbeason](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tbeason/32/15898_2.png) [@tbeason](https://discourse.julialang.org/u/tbeason)\
**Post date:** [March 13, 2020, 12:16am UTC](https://discourse.julialang.org/t/parsing-strange-csv-files/35907/4 "2020-03-13T00:16:40Z")

</div>

Wrapping it in an `IOBuffer` does seem to work…

```julia
ios = IOBuffer(string(stringarray[5:7]...))
CSV.read(ios)

```

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [March 13, 2020, 9:19am UTC](https://discourse.julialang.org/t/parsing-strange-csv-files/35907/5 "2020-03-13T09:19:36Z")

</div>

I am not sure what the spec is, but it seems to be `\n\n` separating blocks that may or may not be valid CSV tables.

You could read the whole thing into a `String`, find the block boundaries, then pass on the parts to CSV reader libraries, either wrapped in an `IOBuffer` or similar, depending on how large the files are and whether you want to fiddle with making this fast.

If there is some rule to skip the comments (eg they are always just one line?) you can filter for them.

FWIW, it may be best to just ask the person generating these files to kindly stop doing it, if nesting multiple tables in a single file with metadata is necessary then CSV is not the best format for this (consider HDF5 or similar).

---

<div class="post-metadata">

**Author:** ![tbeason](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tbeason/32/15898_2.png) [@tbeason](https://discourse.julialang.org/u/tbeason)\
**Post date:** [March 13, 2020, 10:58pm UTC](https://discourse.julialang.org/t/parsing-strange-csv-files/35907/6 "2020-03-13T22:58:42Z")

</div>

> [@Tamas\_Papp](#):
>
> FWIW, it may be best to just ask the person generating these files to kindly stop doing it

😄 I concur. It is a pretty established data source though, so doubtful that it will change. I’m just trying to provide easier access to the data.

To provide some closure for this thread, I did figure out how to do it. Your suggestion more or less captures the bulk of the method. I’ll be posting the package announcement shortly.