# CSV.jl fails when reading multiple inputs at once if variable name "name" occurs

**URL:** <https://discourse.julialang.org/t/csv-jl-fails-when-reading-multiple-inputs-at-once-if-variable-name-name-occurs/97610>\
**Category:** Data\
**Tags:** question, csv\
**Created:** [April 18, 2023, 10:29am UTC](https://discourse.julialang.org/t/csv-jl-fails-when-reading-multiple-inputs-at-once-if-variable-name-name-occurs/97610 "2023-04-18T10:29:59Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![klwlevy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/klwlevy/32/25272_2.png) [@klwlevy](https://discourse.julialang.org/u/klwlevy)\
**Post date:** [April 18, 2023, 10:29am UTC](https://discourse.julialang.org/t/csv-jl-fails-when-reading-multiple-inputs-at-once-if-variable-name-name-occurs/97610/1 "2023-04-18T10:29:59Z")

</div>

Hi there,

I am importing multiple csv files using the excellent CSV.jl package. However, I fail to import all files in one go when one of the imported variables has the variable name “name”.

Is this a bug, and can someone recommend a workaround? In my real example, I cannot change the content of the csv files I am importing.

I could of course import only one file at a time, but I like the idea of importing them all in one go.

MWE:

```julia
using CSV

# Data with ordinary variable names - works
data = [
    "a,b,c\n1,2,3\n4,5,6\n",
    "a,b,c\n7,8,9\n10,11,12\n",
]
CSV.File(map(IOBuffer, data)) # Works

# Only one set of data with variable name "name" - works
data = [
    "a,name,c\n1,2,3\n4,5,6\n",
]
CSV.File(map(IOBuffer, data)) # Works

# Data where the second variable name is "name" - fails
data = [
    "a,name,c\n1,2,3\n4,5,6\n",
    "a,name,c\n7,8,9\n10,11,12\n",
]
CSV.File(map(IOBuffer, data)) # Fails

```

REPL output:

```julia
4-element CSV.File:
 CSV.Row: (a = 1, b = 2, c = 3)   
 CSV.Row: (a = 4, b = 5, c = 6)   
 CSV.Row: (a = 7, b = 8, c = 9)   
 CSV.Row: (a = 10, b = 11, c = 12)

2-element CSV.File:
 CSV.Row: (a = 1, name = 2, c = 3)
 CSV.Row: (a = 4, name = 5, c = 6)

ERROR: MethodError: Cannot `convert` an object of type SentinelArrays.ChainedVector{Int64, Vector{Int64}} to an object of type String
Closest candidates are:
  convert(::Type{String}, ::WeakRefStrings.WeakRefString) at C:\Users\B046326\.julia\packages\WeakRefStrings\31nkb\src\WeakRefStrings.jl:81
  convert(::Type{String}, ::FilePathsBase.AbstractPath) at C:\Users\B046326\.julia\packages\FilePathsBase\9kSEl\src\path.jl:117       
  convert(::Type{String}, ::String) at essentials.jl:218
  ...
Stacktrace:
 [1] CSV.File(name::SentinelArrays.ChainedVector{Int64, Vector{Int64}}, names::Vector{Symbol}, types::Vector{Type}, rows::Int64, cols::Int64, columns::Vector{CSV.Column}, lookup::Dict{Symbol, CSV.Column})
   @ CSV C:\Users\B046326\.julia\packages\CSV\b8ebJ\src\file.jl:106
 [2] CSV.File(sources::Vector{IOBuffer}; source::Nothing, kw::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
   @ CSV C:\Users\B046326\.julia\packages\CSV\b8ebJ\src\file.jl:940
 [3] CSV.File(sources::Vector{IOBuffer})
   @ CSV C:\Users\B046326\.julia\packages\CSV\b8ebJ\src\file.jl:891
 [4] top-level scope
   @ c:\Users\B046326\Levy\projects\bam_io\test\temp.jl:150

julia>

```

---

<div class="post-metadata">

**Author:** ![albheim](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/albheim/32/34660_2.png) [@albheim](https://discourse.julialang.org/u/albheim)\
**Post date:** [April 18, 2023, 11:08am UTC](https://discourse.julialang.org/t/csv-jl-fails-when-reading-multiple-inputs-at-once-if-variable-name-name-occurs/97610/2 "2023-04-18T11:08:38Z")

</div>

Seems like a bug based on aggregating the data in some temporary `f::File` that has a field `name` and then calling `f.name` when creating the return value.

> <https://github.com/JuliaData/CSV.jl/blob/cfb4ffb5d9847df4f8d5efa5129b27026a83a4a8/src/file.jl#L940>

This call will go to `getproperty` for the type that is defined to first look at some dict to see if there is a column by that name, and if not it will get the field from the struct.

> <https://github.com/JuliaData/CSV.jl/blob/cfb4ffb5d9847df4f8d5efa5129b27026a83a4a8/src/file.jl#L142-L145>

So for this case `f.name` will return a column, but it is at a position in the `File` constructor that expects a string.

Could probably be solved by using some of the getters defined at the top of the file instead of dot syntax?

> <https://github.com/JuliaData/CSV.jl/blob/cfb4ffb5d9847df4f8d5efa5129b27026a83a4a8/src/file.jl#L115-L123>

Though that would mean a fix in CSV, I don’t have any suggestion for a simple local workaround other than just load them separately.

Seems like a brittle way of overriding `getproperty` though, easy for something like this to happen.

---

<div class="post-metadata">

**Author:** ![jd-foster](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jd-foster/32/35824_2.png) [@jd-foster](https://discourse.julialang.org/u/jd-foster)\
**Post date:** [April 18, 2023, 11:47am UTC](https://discourse.julialang.org/t/csv-jl-fails-when-reading-multiple-inputs-at-once-if-variable-name-name-occurs/97610/3 "2023-04-18T11:47:23Z")

</div>

> [@klwlevy](#):
>
> can someone recommend a workaround

Some combination of the keyword `header` and a preprocessing step on the names in the header?

[`CSV.File`](https://csv.juliadata.org/stable/reading.html#CSV.File)

---

<div class="post-metadata">

**Author:** ![rocco\_sprmnt21](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rocco_sprmnt21/32/20127_2.png) [@rocco\_sprmnt21](https://discourse.julialang.org/u/rocco_sprmnt21)\
**Post date:** [April 20, 2023, 11:00am UTC](https://discourse.julialang.org/t/csv-jl-fails-when-reading-multiple-inputs-at-once-if-variable-name-name-occurs/97610/4 "2023-04-20T11:00:12Z")

</div>

Does this meet the need?

```julia
vcat(CSV.File.(map(IOBuffer, data))...)

```

---

<div class="post-metadata">

**Author:** ![albheim](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/albheim/32/34660_2.png) [@albheim](https://discourse.julialang.org/u/albheim)\
**Post date:** [April 21, 2023, 5:53am UTC](https://discourse.julialang.org/t/csv-jl-fails-when-reading-multiple-inputs-at-once-if-variable-name-name-occurs/97610/5 "2023-04-21T05:53:25Z")

</div>

Added a PR to fix this, so one workaround could be to use that branch until it gets merged.

> <https://github.com/JuliaData/CSV.jl/pull/1081>
>
> Solves #1080 and adds test for the original instance of the problem.
> 
> Would it… be worth having a check on all field names? Feels like it is very easy to accidentally use the dot syntax internally when accessing the fields, and only when someone use data where the columns share the names something would be incorrect. Maybe more than only this constructor should be tested with columns using the field names of \`File\`?

---

<div class="post-metadata">

**Author:** ![klwlevy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/klwlevy/32/25272_2.png) [@klwlevy](https://discourse.julialang.org/u/klwlevy)\
**Post date:** [April 23, 2023, 5:17pm UTC](https://discourse.julialang.org/t/csv-jl-fails-when-reading-multiple-inputs-at-once-if-variable-name-name-occurs/97610/6 "2023-04-23T17:17:19Z")

</div>

That is amazing. Today I saw that @quinnj merged the bugfix into the main branch on github so I guess there will be a new version of the package soon to resolve this. I wish I can get good enough one day to be able to do a PR myself. As of now I can just express my gratitude to @albheim and @quinnj. Thanks!

---

<div class="post-metadata">

**Author:** ![klwlevy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/klwlevy/32/25272_2.png) [@klwlevy](https://discourse.julialang.org/u/klwlevy)\
**Post date:** [April 23, 2023, 5:22pm UTC](https://discourse.julialang.org/t/csv-jl-fails-when-reading-multiple-inputs-at-once-if-variable-name-name-occurs/97610/7 "2023-04-23T17:22:12Z")

</div>

Thanks for this idea. To make progress, I made a workaround without parsing the headers at all and instead hardcoding them, like this:

```julia
CSV.read(
      # Collect needed to materialize the vector since SubDataFrames are views into the parent
      collect(subdf.desired_filepath),
      DataFrame;
      header = [
          "date",
          "code",
          "isin",
          "sedol",
          "country",
          "currency",
          "exchange",
          "sector",
          "indexshares",
          "rate",
          "price",
          "indexweight",
          #"name" # CSV.jl has a bug that does not allow this variable name when reading multiple
                                  # files in one go, see
                                  # https://discourse.julialang.org/t/csv-jl-fails-when-reading-multiple-inputs-at-once-if-variable-name-name-occurs/97610
          "security_name",
      ],
      skipto = 5,
  )

```

---

<div class="post-metadata">

**Author:** ![klwlevy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/klwlevy/32/25272_2.png) [@klwlevy](https://discourse.julialang.org/u/klwlevy)\
**Post date:** [April 24, 2023, 9:17am UTC](https://discourse.julialang.org/t/csv-jl-fails-when-reading-multiple-inputs-at-once-if-variable-name-name-occurs/97610/8 "2023-04-24T09:17:42Z")

</div>

I decided to not wait for a new version of the CSV package but to move to the main branch of CSV.jl where @albheim got his bugfix implemented.

In practice, I simply entered the package manager in the REPL and entered:

`add CSV#main`

Then I could remove my workaround and simply use:

```julia
CSV.read(
      # Collect needed to materialize the vector since SubDataFrames are views into the parent
      collect(subdf.desired_filepath),
      DataFrame;
      header = 4,
)

```

Entering `status` in the package manager now shows:

[336ed68f] CSV v0.10.9 `https://github.com/JuliaData/CSV.jl.git#main`

Open source is pretty cool when it works like this. What a great community!