CSV.write writeshortest error

I have been trying for several days to save a dataframe and cannot seem to avoid the same recurring error.

Here is the beginning of my file:

df_rtns6[1:5,1:5]

	Date	AAPL_Close	ABT_Close	ADBE_Close	AIG_Close
String	Float64	Float64	Float64	Float64
1	2000-01-04	-0.08431	-0.0285714	-0.0838894	-0.0511125
2	2000-01-05	0.0146332	-0.00183822	0.0197711	0.00237643
3	2000-01-06	-0.0865383	0.0349907	0.00816327	0.0305042
4	2000-01-07	0.0473689	0.0106762	0.048583	0.0748466
5	2000-01-10	-0.0175879	-0.00704224	0.03861	-0.0176941

and here is my write statement

CSV.write("Rtns_FF_Factors.csv",df_rtns6);

The output is always something like:

UndefVarError: writeshortest not defined

Stacktrace:
  [1] writecell(buf::Vector{UInt8}, pos::Int64, len::Int64, io::IOStream, x::Float64, opts::CSV.Options{UInt8, UInt8, Nothing, Tuple{}, typeof(CSV._identity)})
    @ CSV ~/.julia/packages/CSV/jFiCn/src/write.jl:444
  [2] (::CSV.var"#106#107"{Vector{UInt8}, Base.RefValue{Int64}, Int64, IOStream, Int64, CSV.Options{UInt8, UInt8, Nothing, Tuple{}, typeof(CSV._identity)}, UInt8, UInt8})(val::Float64, col::Int64, nm::Symbol)
    @ CSV ~/.julia/packages/CSV/jFiCn/src/write.jl:371
  [3] eachcolumn
    @ ~/.julia/packages/Tables/T7rHm/src/utils.jl:70 [inlined]
  [4] writerow(buf::Vector{UInt8}, pos::Base.RefValue{Int64}, len::Int64, io::IOStream, sch::Tables.Schema{(:Date, :AAPL_Close, :ABT_Close, :ADBE_Close, :AIG_Close, :AMGN_Close, :AMT_Close, :AMZN_Close, :AXP_Close, :BA_Close, :BAC_Close, :BK_Close, :BKNG_Close, :BLK_Close, :BMY_Close, :C_Close, :CAT_Close, :CL_Close, :CMCSA_Close, :COF_Close,

and similar, then a bit further down:

DataFrames.DataFrameRows{DataFrame}, CSV.Options{UInt8, UInt8, Nothing, Tuple{}, typeof(CSV._identity)}, Vector{UInt8}, Int64, Int64, NTuple{85, Symbol}}, io::Any, append::Bool, compress::Bool)

I checked for β€˜nothings’ and missing values and there are none.

I would be grateful for any help

This might be a version conflict between CSV and Parsers.
Update both packages to the newest version should fix this.
Downgrading Parsers to 2.4.0 should also work.

Thank you for your suggestions.

I have recently run Pkg.update(), so unfortunately I don’t think your first suggestion will work.

I never knowingly installed Parsers before. I worry a bit that downgrading might break something else(?)

In the meantime, writedlm worked easily for me, so I guess I will have to avoid CSV.jl for now.
Thanks

What version of packages are you on? Please post the output of ] st

Hi. Thanks for taking a look.

julia> Pkg.status("CSV")
Status `~/.julia/environments/v1.8/Project.toml`
βŒƒ [336ed68f] CSV v0.10.4
Info Packages marked with βŒƒ have new versions available

julia> Pkg.status("Parsers")
No Matches in `~/.julia/environments/v1.8/Project.toml`

I updated CSV, so now

julia> Pkg.status("CSV")
Status `~/.julia/environments/v1.8/Project.toml`
  [336ed68f] CSV v0.10.7

I’ll try it out and let you know (hope nothing else breaks!)

Thanks

Did you manage to fix it? I have the exact same problem

1 Like

I haven’t been able to test whether the upgrade helped yet, as I
can’t afford to lose my data by restarting just now. I did manage
to get around it using writedlm

It is strange because the same instruction was working fine a few minutes ago and now it is not, maybe there’s a bug in some Julia update? I have both the last version of CSV plus the last version of Parsers and I’m having this trouble. Do you know how to downgrade parsers?

1 Like

having some issue here as well - moving from atom over to vscode - running julia 1.7 and updated to latest packages. never had an issue prior.

Converting the entire data frame content into string type worked for me

CSV.write(β€œtemp.csv”,string.(df))

1 Like

That seems a good workaround, so long as it doesn’t add quotes to everything.
It would be lovely if someone knew how to fix this.

Can you create a MWE that illustrates the problem? Something we can copy and paste to debug.

Yes an MWE would be good - converting to string is certainly not an acceptable solution for most usecases.

Here’s an example MWE:

(jl_AHCSuc) pkg> st CSV DataFrames
Status `C:\Users\ngudat\AppData\Local\Temp\jl_AHCSuc\Project.toml`
  [336ed68f] CSV v0.10.7
  [a93c6f00] DataFrames v1.4.3

julia> using CSV, DataFrames, Dates

julia> df = DataFrame(Date = Date(2000):Day(1):Date(2000, 1, 5), AAPL_Close = randn(5), ABT_Close = randn(5), ADBE_Close = randn(5), AIG_Close = randn(5))
5Γ—5 DataFrame
 Row β”‚ Date        AAPL_Close  ABT_Close  ADBE_Close  AIG_Close
     β”‚ Date        Float64     Float64    Float64     Float64
─────┼───────────────────────────────────────────────────────────
   1 β”‚ 2000-01-01   -1.67331    0.501665   -0.471492   0.685009
   2 β”‚ 2000-01-02    0.832788  -0.47307     0.648369  -0.0362069
   3 β”‚ 2000-01-03    1.22546   -0.517133    0.505574   0.282108
   4 β”‚ 2000-01-04    1.10711   -1.23482    -0.993789   0.325811
   5 β”‚ 2000-01-05   -1.08964   -1.01078     0.306837  -0.956466

julia> CSV.write("Rtns_FF_Factors.csv", df)
"Rtns_FF_Factors.csv"

so I can’t replicate this - do you get the same error if you only write the first five rows/columns so is there a specific issue with a certain row/column of your data?

Hi Thanks for taking a look. I tried just looking at the first few rows (or columns) of the dataset.

I do recall seeing question marks (β€˜?’) showing before the β€˜Float64’ at the top, even though I made sure there were no missing values or nothings or Inf’s anywhere. I could not get rid of those, though the Matrix() command on the columns worked fine.

I will send an MWE tomorrow, (as I have to give a lecture!)

Thanks!

1 Like

Are you running your code inside of Juno? Or any other environment that might load packages in multiple stages, while changing environments?

Hi! With the following I’m able to reproduce the issue

(@v1.8) pkg> activate --temp
...
(jl_cOmpJF) pkg> add CSV@v0.9.11
...
(jl_cOmpJF) pkg> add DataFrames
...
julia> using DataFrames, CSV
julia> cd(mktempdir())

julia> df = DataFrame(:a=>rand(3))
3Γ—1 DataFrame
 Row β”‚ a
     β”‚ Float64
─────┼──────────
   1 β”‚ 0.100739
   2 β”‚ 0.346434
   3 β”‚ 0.657864

julia> CSV.write("a.csv", df)
ERROR: UndefVarError: writeshortest not defined
Stacktrace:
  [1] writecell(buf::Vector{UInt8}, pos::Int64, len::Int64, io::IOStream, x::Float64, opts::CSV.Options{UInt8, UInt8, Nothing, Tuple{}, typeof(CSV._identity)})
    @ CSV C:\Users\sebastian\.julia\packages\CSV\9LsxT\src\write.jl:444
  [2] (::CSV.var"#112#113"{Vector{UInt8}, Base.RefValue{Int64}, Int64, IOStream, Int64, CSV.Options{UInt8, UInt8, Nothing, Tuple{}, typeof(CSV._identity)}, UInt8, UInt8})(val::Float64, col::Int64, nm::Symbol)
    @ CSV C:\Users\sebastian\.julia\packages\CSV\9LsxT\src\write.jl:371
  [3] eachcolumn
    @ C:\Users\sebastian\.julia\packages\Tables\T7rHm\src\utils.jl:70 [inlined]
  [4] writerow(buf::Vector{UInt8}, pos::Base.RefValue{Int64}, len::Int64, io::IOStream, sch::Tables.Schema{(:a,), Tuple{Float64}}, row::DataFrameRow{DataFrame, DataFrames.Index}, cols::Int64, opts::CSV.Options{UInt8, UInt8, Nothing, Tuple{}, typeof(CSV._identity)})
    @ CSV C:\Users\sebastian\.julia\packages\CSV\9LsxT\src\write.jl:367
  [5] (::CSV.var"#105#106"{Bool, Bool, Tables.Schema{(:a,), Tuple{Float64}}, DataFrames.DataFrameRows{DataFrame}, CSV.Options{UInt8, UInt8, Nothing, Tuple{}, typeof(CSV._identity)}, Vector{UInt8}, Int64, Int64, Tuple{Symbol}})(io::IOStream)
    @ CSV C:\Users\sebastian\.julia\packages\CSV\9LsxT\src\write.jl:235
  [6] with(f::CSV.var"#105#106"{Bool, Bool, Tables.Schema{(:a,), Tuple{Float64}}, DataFrames.DataFrameRows{DataFrame}, CSV.Options{UInt8, UInt8, Nothing, Tuple{}, typeof(CSV._identity)}, Vector{UInt8}, Int64, Int64, Tuple{Symbol}}, io::Any,
 append::Bool, compress::Bool)
    @ CSV C:\Users\sebastian\.julia\packages\CSV\9LsxT\src\write.jl:303
  [7] #write#104
    @ C:\Users\sebastian\.julia\packages\CSV\9LsxT\src\write.jl:225 [inlined]
  [8] write(file::String, itr::DataFrame; delim::Char, quotechar::Char, openquotechar::Nothing, closequotechar::Nothing, escapechar::Char, newline::Char, decimal::Char, dateformat::Nothing, quotestrings::Bool, missingstring::String, transform::typeof(CSV._identity), bom::Bool, append::Bool, compress::Bool, writeheader::Nothing, partition::Bool, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ CSV C:\Users\sebastian\.julia\packages\CSV\9LsxT\src\write.jl:210
  [9] write(file::String, itr::DataFrame)
    @ CSV C:\Users\sebastian\.julia\packages\CSV\9LsxT\src\write.jl:153
 [10] top-level scope
    @ REPL[9]:1
1 Like

Updating CSV will surely fix the issue, but often folks are stuck at v0.9.x due to other compatibility concerns. To work around it, you can pin Parsers.jl to v2.2.4:

Here’s the β€œfixed” version from @SebastianM-C’s MWE:

(@v1.7) pkg> activate --temp
  Activating new project at `/var/folders/cb/h4ftb1y11ls5khbqltqswsl40000gn/T/jl_EsGAnk`

(jl_EsGAnk) pkg> add CSV@v0.9.11
    Updating registry at `~/.julia/registries/General.toml`
    Updating registry at `~/.julia/registries/JuliaComputingRegistry.toml`
    Updating registry at `~/.julia/registries/JuliaSimRegistry.toml`
    Updating registry at `~/.julia/registries/PumasRegistry.toml`
    Updating registry at `~/.julia/registries/TestRegistry.toml`
    Updating registry at `~/.julia/registries/githubjuliaregistryjdb.toml`
    Updating registry at `~/.julia/registries/gitlabjuliaregistryjdb.toml`
   Resolving package versions...
    Updating `/private/var/folders/cb/h4ftb1y11ls5khbqltqswsl40000gn/T/jl_EsGAnk/Project.toml`
  [336ed68f] + CSV v0.9.11
    Updating `/private/var/folders/cb/h4ftb1y11ls5khbqltqswsl40000gn/T/jl_EsGAnk/Manifest.toml`
  [336ed68f] + CSV v0.9.11
  [944b1d66] + CodecZlib v0.7.0
  [34da2185] + Compat v4.5.0
  [9a962f9c] + DataAPI v1.13.0
  [e2d170a0] + DataValueInterfaces v1.0.0
  [48062228] + FilePathsBase v0.9.20
  [842dd82b] + InlineStrings v1.3.2
  [82899510] + IteratorInterfaceExtensions v1.0.0
  [bac558e1] + OrderedCollections v1.4.1
  [69de0a69] + Parsers v2.5.1
  [2dfb63ee] + PooledArrays v1.4.2
  [91c51154] + SentinelArrays v1.3.16
  [66db9d55] + SnoopPrecompile v1.0.1
  [3783bdb8] + TableTraits v1.0.1
  [bd369af6] + Tables v1.10.0
  [3bb67fe8] + TranscodingStreams v0.9.9
  [ea10d353] + WeakRefStrings v1.4.2
  [56f22d72] + Artifacts
  [2a0f44e3] + Base64
  [ade2ca70] + Dates
  [9fa8497b] + Future
  [b77e0a4c] + InteractiveUtils
  [8f399da3] + Libdl
  [37e2e46d] + LinearAlgebra
  [56ddb016] + Logging
  [d6f4376e] + Markdown
  [a63ad114] + Mmap
  [de0858da] + Printf
  [9a3f8284] + Random
  [ea8e919c] + SHA
  [9e88b42a] + Serialization
  [8dfed614] + Test
  [cf7118a7] + UUIDs
  [4ec0a83e] + Unicode
  [e66e0078] + CompilerSupportLibraries_jll
  [4536629a] + OpenBLAS_jll
  [83775a58] + Zlib_jll
  [8e850b90] + libblastrampoline_jll

(jl_EsGAnk) pkg> add DataFrames
   Resolving package versions...
    Updating `/private/var/folders/cb/h4ftb1y11ls5khbqltqswsl40000gn/T/jl_EsGAnk/Project.toml`
  [a93c6f00] + DataFrames v1.4.3
    Updating `/private/var/folders/cb/h4ftb1y11ls5khbqltqswsl40000gn/T/jl_EsGAnk/Manifest.toml`
  [a8cc5b0e] + Crayons v4.1.1
  [a93c6f00] + DataFrames v1.4.3
  [864edb3b] + DataStructures v0.18.13
  [59287772] + Formatting v0.4.2
  [41ab1584] + InvertedIndices v1.2.0
  [b964fa9f] + LaTeXStrings v1.3.0
  [e1d29d7a] + Missings v1.0.2
  [08abe8d2] + PrettyTables v2.2.2
  [189a3867] + Reexport v1.2.2
  [a2af1166] + SortingAlgorithms v1.1.0
  [892a3eda] + StringManipulation v0.3.0
  [3fa0cd96] + REPL
  [6462fe0b] + Sockets
  [2f01184e] + SparseArrays
  [10745b16] + Statistics

(jl_EsGAnk) pkg> add Parsers@v2.2.4
   Resolving package versions...
   Installed Parsers ─ v2.2.4
    Updating `/private/var/folders/cb/h4ftb1y11ls5khbqltqswsl40000gn/T/jl_EsGAnk/Project.toml`
  [69de0a69] + Parsers v2.2.4
    Updating `/private/var/folders/cb/h4ftb1y11ls5khbqltqswsl40000gn/T/jl_EsGAnk/Manifest.toml`
  [69de0a69] ↓ Parsers v2.5.1 β‡’ v2.2.4
Precompiling project...
  4 dependencies successfully precompiled in 7 seconds (28 already precompiled)

julia> using DataFrames, CSV

julia> df = DataFrame(:a=>rand(3))
3Γ—1 DataFrame
 Row β”‚ a
     β”‚ Float64
─────┼──────────
   1 β”‚ 0.687344
   2 β”‚ 0.577659
   3 β”‚ 0.318793

julia> CSV.write("a.csv", df)
"a.csv"

Cf.

did for me too. Thanks!