# Loading a csv-like file with a multi-line header

**URL:** <https://discourse.julialang.org/t/loading-a-csv-like-file-with-a-multi-line-header/113806>\
**Category:** General Usage\
**Tags:** question\
**Created:** [May 3, 2024, 5:09pm UTC](https://discourse.julialang.org/t/loading-a-csv-like-file-with-a-multi-line-header/113806 "2024-05-03T17:09:49Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![Ju\_ska](https://avatars.discourse-cdn.com/v4/letter/j/ecd19e/32.png) [@Ju\_ska](https://discourse.julialang.org/u/Ju_ska)\
**Post date:** [May 3, 2024, 5:09pm UTC](https://discourse.julialang.org/t/loading-a-csv-like-file-with-a-multi-line-header/113806/1 "2024-05-03T17:09:49Z")

</div>

Hi,

I would like to load a IAGA-2002 file like [http://www.bcmt.fr/DATABANK/VARIATION/ppt/sec/2024/03/ppt20240301vsec.sec](http://www.bcmt.fr/DATABANK/VARIATION/ppt/sec/2024/03/ppt20240301vsec.sec)

```julia
 Format IAGA-2002 |
 Source of Data IPGP (France) |
 Station Name Pamatai |
 IAGA Code PPT |
 Geodetic Latitude -17.567 |
 Geodetic Longitude 210.426 |
 Elevation 357.0 |
 Reported XYZF |
 Sensor Orientation HDZF |
 Digital Sampling 0.2 second |
 Data Interval Type Filtered 1-second |
 Data Type variation |
 # |
 # File added to the data base on the 2024-03-02 |
 # Conditions of use: |
 # License: CC BY-NC 4.0 |
 # http://creativecommons.org/licenses/by-nc/4.0/ |
 # DOI citation: http://dx.doi.org/10.18715/BCMT.MAG.VAR |
 # |
 # For any enquiry, please contact: bcmt@ipgp.fr |
 # |
 # Formulas for computing non-reported elements: |
 # H=SquareRoot(X*X+Y*Y), tan(D)=Y/X, tan(I)=Z/SquareRoot(X*X+Y*Y) |
 # 1-second values are computed using a 24 bits digitizer and a |
 # Gaussian filter of 15 coefficients. |
DATE TIME DOY PPTX PPTY PPTZ PPTF |
2024-03-01 00:00:00.000 061 29138.14 5945.07 -18657.18 35103.95
2024-03-01 00:00:01.000 061 29138.14 5945.09 -18657.15 88888.00
2024-03-01 00:00:02.000 061 29138.12 5945.11 -18657.10 88888.00
2024-03-01 00:00:03.000 061 29138.11 5945.13 -18657.09 88888.00
2024-03-01 00:00:04.000 061 29138.13 5945.12 -18657.11 88888.00
2024-03-01 00:00:05.000 061 29138.14 5945.17 -18657.11 35103.98
2024-03-01 00:00:06.000 061 29138.14 5945.21 -18657.08 88888.00
2024-03-01 00:00:07.000 061 29138.12 5945.22 -18657.04 88888.00
etc. 86400 lines of data

```

I don’t need the lines ending with ‘|’, so I did:

```julia
file=readlines("ppt20240301vsec.sec")
filter!(x -> !contains(x, '|'), file)

for stringline in file
    splitted=split(stringline)
    do_something_else
end

```

I’m stuck there, I would like first to end up with a 86400 lines x 7 columns matrix or array.  
Then better have as columns types: Date Date Int Float Float Float Float, or maybe regroup the two Date columns.

I tried stuff, but too complicated for something i feel Julia could do simply.

Any help is welcome.

---

<div class="post-metadata">

**Author:** ![tbeason](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tbeason/32/15898_2.png) [@tbeason](https://discourse.julialang.org/u/tbeason)\
**Post date:** [May 3, 2024, 5:37pm UTC](https://discourse.julialang.org/t/loading-a-csv-like-file-with-a-multi-line-header/113806/2 "2024-05-03T17:37:50Z")

</div>

Have you tried using CSV.jl? It can handle all sorts of delimited files (not just commas) and can skip lines and infer column types.

---

<div class="post-metadata">

**Author:** ![Ju\_ska](https://avatars.discourse-cdn.com/v4/letter/j/ecd19e/32.png) [@Ju\_ska](https://discourse.julialang.org/u/Ju_ska)\
**Post date:** [May 3, 2024, 6:18pm UTC](https://discourse.julialang.org/t/loading-a-csv-like-file-with-a-multi-line-header/113806/3 "2024-05-03T18:18:39Z")

</div>

I tried it, but the problem is that the header can have a variable number of lines.

---

<div class="post-metadata">

**Author:** ![ericphanson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ericphanson/32/215186_2.png) [@ericphanson](https://discourse.julialang.org/u/ericphanson)\
**Post date:** [May 3, 2024, 6:28pm UTC](https://discourse.julialang.org/t/loading-a-csv-like-file-with-a-multi-line-header/113806/4 "2024-05-03T18:28:14Z")

</div>

You could try to detect the size of the header yourself, and then pass that to CSV:

```julia
function count_header_lines(file)
    idx = 1
    for line in eachline(file)
        endswith(line, "|") || return idx
        idx += 1
     end
     error("Found no header lines ending with `|`")
end

CSV.File(file; skipto = count_header_lines(file))

```

```julia

```

---

<div class="post-metadata">

**Author:** ![rocco\_sprmnt21](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rocco_sprmnt21/32/20127_2.png) [@rocco\_sprmnt21](https://discourse.julialang.org/u/rocco_sprmnt21)\
**Post date:** [May 3, 2024, 7:55pm UTC](https://discourse.julialang.org/t/loading-a-csv-like-file-with-a-multi-line-header/113806/5 "2024-05-03T19:55:03Z")

</div>

try this.  
It is based on a specific characteristic of the file you addressed.  
If the others are built “similarly” it should work

```julia
rows=CSV.File("testmh.txt"; header=false,comment=" ")

names=tab[1][1:end-1]

select(DataFrame(Tables.rows(rows[2:end])),1=>ByRow(split)=>names)

```

The title should also be changed: it is misleading. We are not talking about multiline headers in this case (which would still be handled by CSV).

---

<div class="post-metadata">

**Author:** ![joa-quim](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/joa-quim/32/227_2.png) [@joa-quim](https://discourse.julialang.org/u/joa-quim)\
**Post date:** [May 6, 2024, 1:17am UTC](https://discourse.julialang.org/t/loading-a-csv-like-file-with-a-multi-line-header/113806/6 "2024-05-06T01:17:03Z")

</div>

Hmmm, geomagnetic data files. You got me nostalgic so I made some changes in `gmtread` (GMT.jl) and now with master version we can do:

```julia
using GMT

D = gmtread("ppt20240301vsec.sec", table=true, stride=5)
Comment: ["", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "DATE TIME DOY PPTX PPTY PPTZ PPTF |"]
Attribute table
┌─────────┐
│ Timecol │
├─────────┤
│ 1 │
└─────────┘
BoundingBox: [1.7092512e9, 1.7092512e9, 0.0, 23.99861111111111, 61.0, 61.0, 29056.38, 29138.14, 5852.03, 5946.78, -18715.51, -18656.79, 35046.35, 35104.03]

17280×7 GMTdataset{Float64, 2}
 Row │ Time TIME DOY PPTX PPTY PPTZ PPTF
─────┼──────────────────────────────────────────────────────────────────────
   1 │ 2024-03-01T00:00:00 0.0 61.0 29138.1 5945.07 -18657.2 35103.9
   2 │ 2024-03-01T00:00:05 0.0 61.0 29138.1 5945.17 -18657.1 35104.0
   3 │ 2024-03-01T00:00:10 0.0 61.0 29138.1 5945.26 -18657.1 35104.0
...

```

But note that in order to get those column names I had to add a `#` as first char in line 26 to follow the standard that comment lines should start with a hash (people at Chambon should take more care on how they write the file headers).

I also read only one every other 5th row because the others do not have the total field and X,Y,Z components are just repeated. To see it:

```julia
plot(D, xvar=:Time, yvar=:PPTF, show=true)

```

 ![GMTjl_j](https://global.discourse-cdn.com/julialang/original/3X/d/7/d7f921b8648d5ac4bf2ff9db19e0919017767c2d.png)

---

<div class="post-metadata">

**Author:** ![Ju\_ska](https://avatars.discourse-cdn.com/v4/letter/j/ecd19e/32.png) [@Ju\_ska](https://discourse.julialang.org/u/Ju_ska)\
**Post date:** [May 6, 2024, 9:39am UTC](https://discourse.julialang.org/t/loading-a-csv-like-file-with-a-multi-line-header/113806/7 "2024-05-06T09:39:05Z")

</div>

Thanks everyone ! I’ll dig into that.

About the line 26:

> **[IAGA 2002 Data Exchange Format | National Centers for Environmental...](https://www.ncei.noaa.gov/services/world-data-system/v-dat-working-group/iaga-2002-data-exchange-format)**
>
> Additions made June 2015 Additions made August 2011 Additions made July 2003 Revision December 2001 This page provides information on the International Association of Geomagnetism and Aeronomy (IAGA) ASCII Exchange Format, which was adopted August...

Check “The mandatory data header record”.  
And

> **[technical\_manual.pdf](https://intermagnet.org/docs/Technical-Manual/technical_manual.pdf)**
>
> 1148.89 KB

Page 123, labels don’t need a #.

---

<div class="post-metadata">

**Author:** ![joa-quim](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/joa-quim/32/227_2.png) [@joa-quim](https://discourse.julialang.org/u/joa-quim)\
**Post date:** [May 6, 2024, 10:51am UTC](https://discourse.julialang.org/t/loading-a-csv-like-file-with-a-multi-line-header/113806/8 "2024-05-06T10:51:13Z")

</div>

> [@Ju\_ska](#):
>
> Page 123, labels don’t need a #.

They may not need a # when writing a reader for a particular format but they **do** need it when writing generic code that aims to read many different files with variable number of headers. Go to Matlab and try to read text files where comments do not start with a `%` and see what happens. Anyway, you may not add that `#` but then the column names is not found. And, BTW, having two columns for _time_ (one for `date` and another for `time`) is not the best of the ideas too.

---

<div class="post-metadata">

**Author:** ![Ju\_ska](https://avatars.discourse-cdn.com/v4/letter/j/ecd19e/32.png) [@Ju\_ska](https://discourse.julialang.org/u/Ju_ska)\
**Post date:** [May 6, 2024, 11:35am UTC](https://discourse.julialang.org/t/loading-a-csv-like-file-with-a-multi-line-header/113806/9 "2024-05-06T11:35:17Z")

</div>

Well it’s not me or Chambon who decide the International Association of Geomagnetism and Aeronomy standards 🙂

---

<div class="post-metadata">

**Author:** ![joa-quim](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/joa-quim/32/227_2.png) [@joa-quim](https://discourse.julialang.org/u/joa-quim)\
**Post date:** [May 6, 2024, 4:28pm UTC](https://discourse.julialang.org/t/loading-a-csv-like-file-with-a-multi-line-header/113806/10 "2024-05-06T16:28:26Z")

</div>

Sure. 🙂 Just a small complain from one of those who was bitten by those inconsistencies.

---

<div class="post-metadata">

**Author:** ![Ju\_ska](https://avatars.discourse-cdn.com/v4/letter/j/ecd19e/32.png) [@Ju\_ska](https://discourse.julialang.org/u/Ju_ska)\
**Post date:** [May 19, 2024, 11:49pm UTC](https://discourse.julialang.org/t/loading-a-csv-like-file-with-a-multi-line-header/113806/11 "2024-05-19T23:49:06Z")

</div>

By the way, the second column in GMTdataset should be part of the time, as it’s the milliseconds of the ISO 8601 hh:mm:ss.sss standard.

---

<div class="post-metadata">

**Author:** ![joa-quim](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/joa-quim/32/227_2.png) [@joa-quim](https://discourse.julialang.org/u/joa-quim)\
**Post date:** [May 20, 2024, 12:02am UTC](https://discourse.julialang.org/t/loading-a-csv-like-file-with-a-multi-line-header/113806/12 "2024-05-20T00:02:14Z")

</div>

The second column (the time in the original file) was added to the date (first column) as can be seen in the short print. I decided to leave it with zeros to avoid copying the entire table when removing an inner column.