There’s an open CSV.jl issue which is generally about the idea of somehow “ingesting” multiple, same-schema delimited data files in a single call. I wanted to open this post as a request for feedback on what people think would be the most helpful/useful here. Here are a few things I think about with…

Would there be any performance benefits? This seems like the biggest advantage if it is the case. Without it, this kinda just seems like dressing up Vector{CSV.File} to be more usable like a single table. Nothing wrong with that, though.

What will happen if they don’t actually have the same schema? e.g. different column names, types, number of columns.

I thought that partitioner was partly (heh heh) about that. In case it’s worth shipping checking schema and colum types, maybe just a keyword settings for Tables.partitioner would be enough (although the implementation might be hard), something like identical_schema = true.

I often use the first solution (dfs = CSV.read.(list_of_files, DataFrame)), but I think the intended use is more to keep files separated (although you can easily do df = vcat(dfs...)), while the opened issue seems more to mean treating several files as if it was one. If so couldn’t this handled as …

I think my initial reaction would be to keep things simple and the surface API as small as possible, which to me seems like a bit of a design principle of the Julia data ecosystem. On the cons of the current approach, I think writing reduce(vcat, CSV.read.(myfiles, DataFrame)) isn’t all that inconv…

This is a good question; the main performance concern would be what to do about automatic threading. If we know there are a number of files, we could automatically adjust settings so each file is read with a single thread, with multiple files being read concurrently (the current default is to use mu…

Two important use cases where an application pulls in a succession of CSV files, each given once in a sequence of CSV file pathnames, are distinguished by whether or not their individual schemas (loosely, the named+typed columns in order of tabulation) are strongly consistent. (quasi-plausable CSV …

@quinnj Just want to say I appreciate you seeking community input on this proposal. There’s only so much a library developer can know about their users’ needs without asking!

[image] quinnj: Cons: …I guess a decent amount of work in CSV.jl. Actually probably not too bad, but it is maintenance going forward. I think that doing this in CSV.jl is the wrong level of abstraction (unless the performance gains are really compelling, but I doubt that). It would be better…

Request for feedback on potential CSV.jl feature

Specific Domains Data

rafael.guerra June 6, 2021, 7:43am 10

@quinnj, fyi there was a related inverse functionality sought here that you may consider.

Topic		Replies	Views
What's the difference between CSV.jl and CSVFiles.jl? New to Julia	24	8561	January 29, 2020
[ANN] New CSV.jl 0.5 Release Package Announcements data , csv	18	5363	October 20, 2019
Parsing strange CSV files General Usage question , csv	5	719	March 13, 2020
Partition a large CSV file into smaller files without loading into memory General Usage question , csv , io	6	3826	March 10, 2019
Read multiple csv files Performance filesystem	1	1489	July 28, 2020

Request for feedback on potential CSV.jl feature

Related topics