Reading headers of delimited files

[stevengj]

@DanDeepPhase

Is there a way to pass a vector of strings to CSV or DataFrames? Then i’d use readlines to parse the whole file, then pass some portion of the realines vector to CSV or delimitedfiles for parsing.

Better to just pass the f object directly to CSV.read, since that will begin reading from the current position in the stream:

f = open(filename)
header = first(eachline(f), 7) # read 7 header lines
data = CSV.read(f, DataFrame, header=false)

More generally, you can take any string s (or vector of bytes), representing any chunk of the file you want to read from, and call CSV.read on IOBuffer(s).

@DanDeepPhase

is there a way to chunk a file into sections in CSV or delimitedfiles? My experience is that unless i completely isolate the data section ( which i do by dropping lines), the auto-parser is confused by the change of format and will have too few columns and error out.

Sure, just isolate the section you want and wrap an IOBuffer around it.

I would try to avoid reading the file line-by-line if you can. For example, you could mmap the whole file, then wrap an IOBuffer around a @view of the section of the file you want to read from. You can use StringViews.jl to perform regular-expression searches of a memory-mapped file, or just search for bytes like UInt8('\n') (line endings).