# When converting multiple .csv files to a DataFrame, what is the most effective way to identify the data entries in the DataFrame with the .csv filename?

**URL:** <https://discourse.julialang.org/t/when-converting-multiple-csv-files-to-a-dataframe-what-is-the-most-effective-way-to-identify-the-data-entries-in-the-dataframe-with-the-csv-filename/88606>\
**Category:** General Usage\
**Tags:** question, dataframes, csv\
**Created:** [October 12, 2022, 4:45am UTC](https://discourse.julialang.org/t/when-converting-multiple-csv-files-to-a-dataframe-what-is-the-most-effective-way-to-identify-the-data-entries-in-the-dataframe-with-the-csv-filename/88606 "2022-10-12T04:45:25Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![phantom](https://avatars.discourse-cdn.com/v4/letter/p/e0b2c6/32.png) [@phantom](https://discourse.julialang.org/u/phantom)\
**Post date:** [October 12, 2022, 4:45am UTC](https://discourse.julialang.org/t/when-converting-multiple-csv-files-to-a-dataframe-what-is-the-most-effective-way-to-identify-the-data-entries-in-the-dataframe-with-the-csv-filename/88606/1 "2022-10-12T04:45:25Z")

</div>

Suppose there is a Folder with the following files all with a different number of data entries:

File1 = Data1.csv  
File2 = Data2.csv  
:  
:  
Filen = Datan.csv

where n \>\> 100. I am converting these files to a single DataFrame, however I need to identify which file each entry originally came from. I was just wondering whether there was a direct way to do this in DataFrames?

Currently I am using the following approach which is not great and may be prone to mislabelling error.

First I created a vector of DataFrames with all the Data.

```julia
Files = glob("*.csv", Folder)
CombinedData = DataFrame.(CSV.File.(Files))

```

Then I gathered the names of all the files in a vector and pushed them with the data onto a new DataFrame.

```julia
Names = readdir("Users/Folder")
LabledData = DataFrame( Name = Srtring[], Data = Float64)
for k = 1:size(CombinedData)[1]
     for i = 1:size(CombinedData[k])[1]
          push!(LabledData, [Names[ k], CombinedData[k][i , :] ])
     end
end 

```

My main concern here is whether there would be a relatively high mislabelling risk if for some reason the files in the folder and Names are not aligned. (e.g. a file is not loaded, added, or removed). Given that n is large I wouldn’t be able to verify whether the data matches the label. I saw another post on the matter but it also seems to take a similar approach with readdir(). Just wondering if there was a better way to go about adding a column with the file name or identifying the data in general? Thanks!

---

<div class="post-metadata">

**Author:** ![nilshg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nilshg/32/2283_2.png) [@nilshg](https://discourse.julialang.org/u/nilshg)\
**Post date:** [October 12, 2022, 5:27am UTC](https://discourse.julialang.org/t/when-converting-multiple-csv-files-to-a-dataframe-what-is-the-most-effective-way-to-identify-the-data-entries-in-the-dataframe-with-the-csv-filename/88606/2 "2022-10-12T05:27:13Z")

</div>

`CSV.read` takes a vector of filepaths which will read all files into a single table, and a `source` kwarg which is a vector of source names (to which you can pass the vector of full file paths as well or just the file names).

---

<div class="post-metadata">

**Author:** ![digital\_carver](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/digital_carver/32/33818_2.png) [@digital\_carver](https://discourse.julialang.org/u/digital_carver)\
**Post date:** [October 12, 2022, 6:50am UTC](https://discourse.julialang.org/t/when-converting-multiple-csv-files-to-a-dataframe-what-is-the-most-effective-way-to-identify-the-data-entries-in-the-dataframe-with-the-csv-filename/88606/3 "2022-10-12T06:50:27Z")

</div>

> - source: … As a single Symbol or String, provides the column name that will be added to the parsed columns, the values of the column will be the input “name” (usually file name) of the input from whence the value was parsed.

This seems to be saying that if you pass in just the column name for the `source` keyword, the values will automatically be the filenames where the row came from. That makes it even more convenient, and more robust too.

---

<div class="post-metadata">

**Author:** ![nilshg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nilshg/32/2283_2.png) [@nilshg](https://discourse.julialang.org/u/nilshg)\
**Post date:** [October 12, 2022, 7:32am UTC](https://discourse.julialang.org/t/when-converting-multiple-csv-files-to-a-dataframe-what-is-the-most-effective-way-to-identify-the-data-entries-in-the-dataframe-with-the-csv-filename/88606/4 "2022-10-12T07:32:27Z")

</div>

Sorry I was on the phone so didn’t expand on my answer.

Here’s a full example:

```julia
julia> using CSV, DataFrames

julia> CSV.write("df1.csv", DataFrame(rand(2, 2), :auto));

julia> CSV.write("df2.csv", DataFrame(5 .* rand(2, 2), :auto));

julia> csv_files = filter(endswith(".csv"), readdir(; join = true))
2-element Vector{String}:
 "C:\\Users\\ngudat\\Documents\\df1.csv"
 "C:\\Users\\ngudat\\Documents\\df2.csv"

julia> CSV.read(csv_files, DataFrame; source = "source")
4×3 DataFrame
 Row │ x1 x2 source
     │ Float64 Float64 String
─────┼─────────────────────────────────────────────────────────
   1 │ 0.0292249 0.281584 C:\\Users\\ngudat\\Documents\\df…
   2 │ 0.220783 0.717647 C:\\Users\\ngudat\\Documents\\df…
   3 │ 3.48909 0.0183668 C:\\Users\\ngudat\\Documents\\df…
   4 │ 4.00998 3.90755 C:\\Users\\ngudat\\Documents\\df…

julia> CSV.read(csv_files, DataFrame; source = "source" => basename.(csv_files))
4×3 DataFrame
 Row │ x1 x2 source
     │ Float64 Float64 String
─────┼───────────────────────────────
   1 │ 0.0292249 0.281584 df1.csv
   2 │ 0.220783 0.717647 df1.csv
   3 │ 3.48909 0.0183668 df2.csv
   4 │ 4.00998 3.90755 df2.csv

```

I prefer to pass a pair as kwarg as I generally find the full paths less useful (although if one needs to re-parse part of the data it can of course sometimes be useful to have them in the resulting DataFrame).

---

<div class="post-metadata">

**Author:** ![phantom](https://avatars.discourse-cdn.com/v4/letter/p/e0b2c6/32.png) [@phantom](https://discourse.julialang.org/u/phantom)\
**Post date:** [October 12, 2022, 8:34am UTC](https://discourse.julialang.org/t/when-converting-multiple-csv-files-to-a-dataframe-what-is-the-most-effective-way-to-identify-the-data-entries-in-the-dataframe-with-the-csv-filename/88606/5 "2022-10-12T08:34:21Z")

</div>

awesome thanks for saving the day again! ( I was just going over one of your posts about splitting columns). In one of your earlier posts you mentioned that CSV.read() was being deprecated in favor of DataFrame(CSV.File()) so will this still work in the future or is there a planned equivalent in DataFrames?

---

<div class="post-metadata">

**Author:** ![nilshg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nilshg/32/2283_2.png) [@nilshg](https://discourse.julialang.org/u/nilshg)\
**Post date:** [October 12, 2022, 9:28am UTC](https://discourse.julialang.org/t/when-converting-multiple-csv-files-to-a-dataframe-what-is-the-most-effective-way-to-identify-the-data-entries-in-the-dataframe-with-the-csv-filename/88606/6 "2022-10-12T09:28:09Z")

</div>

Sorry for the confusion my comment is outdated and we basically came full circle on this.

Originally, `CSV` was depending on `DataFrames` to provide `CSV.read()` functionality which returned a `DataFrame` by default. As `DataFrames` is a pretty heavy dependency, `CSV.read` was at some point deprecated in favour of just having the basic parsing into a generic table structure provided by `CSV.File`, dropping the `DataFrames` dependency. This required users to do `DataFrame(CSV.File())` to get back the old functionality. After some requests from users to bring `CSV.read` back and with advances in the generic `Tables` packages, the `read` function was introduced with the requirement to provide a `sink` argument.

So in summary we went from

```julia
using CSV
CSV.read("myfile.csv") # returns DataFrame by default

```

via

```julia
using CSV, DataFrames
DataFrame(CSV.File("myfile.csv"))

```

to

```julia
using CSV, DataFrames
CSV.read("myfile.csv", DataFrame)

```

while CSV is still not at 1.0, as far as I’m aware it’s been tantalizingly close for a while now and I wouldn’t expect the API for `CSV.read` to change again once 1.0 is released, so you probably don’t have to worry about future deprecations.

---

<div class="post-metadata">

**Author:** ![phantom](https://avatars.discourse-cdn.com/v4/letter/p/e0b2c6/32.png) [@phantom](https://discourse.julialang.org/u/phantom)\
**Post date:** [October 12, 2022, 9:33am UTC](https://discourse.julialang.org/t/when-converting-multiple-csv-files-to-a-dataframe-what-is-the-most-effective-way-to-identify-the-data-entries-in-the-dataframe-with-the-csv-filename/88606/7 "2022-10-12T09:33:51Z")

</div>

awesome, really appreciate the detailed explanation.