Append a single line to a csv file for a multiple of times using Julia

General Usage
1

Hi, my task is to append a single line to a given csv file for a couple of times using Julia. There might be more than one ways to do it. I was thinking to use the DataFrames module and the CSV module to accomplish this. The following is the initial code, which is not working so far.

using DataFrames
using CSV

df = DataFrame(A=1:3, B=4:6, C=7:9)

for i in 1:10   # do the following for 10 times
    CSV.write("append_a_line.csv", df, header = true, append = true)
end

When running the above code, I got an error as the following:
“TypeError: in #write, in typeassert, expected Array{T,1} where T, got Bool”.

Could anyone kindly let me know what might be wrong here? Or do you have any other better ways to handle this?
Many thanks for your time and attention.

2

Hi,

So you want to append your 3x3 Dataframe 10 times correct?

Don’t know if there is a better way but you cannot use

header=true

If you look at documentation for CSV.write, you will read

•    header: pass a list of column names (Symbols or
    Strings) to use instead of the column names of the
    input table

So you can remove this parameter and it will work.
If you want to have the header printed, It seems that using append=true doesn’t create header, so I suggest that you write

using CSV

df = DataFrame(A=1:3, B=4:6, C=7:9)

CSV.write("append_a_line.csv", df, delim = ';')
for i in 1:9   # do the following for 10 times
    CSV.write("append_a_line.csv", df, delim = ';',append=true)
end

You can add the parameter

delim = ';'

such that your column will be correctly separate.

But there is maybe a nicer way to do it.

Is this answer your question?

Regards

1 Like
3

https://github.com/JuliaData/CSV.jl/issues/247
I’m following the above link basically. It seems like this issue has already been solved. However, I do not quite follow the way to solve it.

4

try this (it seems that the keyword header is not the correct one)

I assumed you only want to write the header once.

using DataFrames
using CSV

df = DataFrame(A=1:3, B=4:6, C=7:9)

fi="c:\\temp\\append_a_line.csv"
fi="append_a_line.csv"


for i in 1:10   # do the following for 10 times    
    CSV.write(fi, df, writeheader = (i==1), append = true)
end
1 Like
5

@bernhard, thank you so much for your quick reply. I tried it out and it worked well. However, it is not exactly what I needed. Instead of controlling the number of times the CSV.write function is called, I need to put this CSV.write inside of another function, say myFunction. Then, the first time when this function myFunction is called, a single line with header is written. After that, for all other times when the function myFunction is called, only a single line is written, without the header. I guess I can use a global flag variable to indicate the number of times the function myFunction is called. And then it is doable in this way. At the same time, I was wondering, are there any other more elegant way to do this then?

6

Well, do you know the header (i.e. the column names)?
If yes, you can simply do

myHeader=["colA","colB","somethingFancy"]
myHeaderReshaped=reshape(myHeader,1,length(myHeader))
using DelimitedFiles

#note: this will 'delete/overwrite' the file if it already exists
writedlm(fi,myHeaderReshaped,',') #note: last argument is the delimiter which should be the same as below

#a lot of code and nested functions......
#.....

for i in 1:10   # do the following for 10 times    
    CSV.write(fi, df, writeheader = false, append = true,sep=',')
end
1 Like
7

Nice job. But why

myHeaderReshaped=reshape(myHeader,1,length(myHeader))
8

Well, you can try the code without that line.
writedlm seems to expect an Array of dimension two. If you simply provide it with the vector, it will write the header on 3 lines.

9

Thanks for trying to resolve the issue. But I tested and the solution and it does not seem to work for me, I am wandering of I am doing something wrong. When I append the header does not write down. Strangely it worked once but then failed to work. I am pasting below my code:

   for  iClimateInventory in ClimateInventory


      Data = CSV.File(ClimatePath_Input, header=true)

      Year_Obs_Start  = convert(Vector{Float64}, Tables.getcolumn(Data, :Year_Obs_Start))
      Month_Obs_Start = convert(Vector{Float64}, Tables.getcolumn(Data, :Month_Obs_Start))
      Day_Obs_Start   = convert(Vector{Float64}, Tables.getcolumn(Data, :Day_Obs_Start))

      Year_Obs_End    = convert(Vector{Float64}, Tables.getcolumn(Data, :Year_Obs_End))
      Month_Obs_End   = convert(Vector{Float64}, Tables.getcolumn(Data, :Month_Obs_End))
      Day_Obs_End     = convert(Vector{Float64}, Tables.getcolumn(Data, :Day_Obs_End))

      Year_Sim_Start  = convert(Vector{Float64}, Tables.getcolumn(Data, :Year_Sim_Start))
      Month_Sim_Start = convert(Vector{Float64}, Tables.getcolumn(Data, :Month_Sim_Start))
      Day_Sim_Start   = convert(Vector{Float64}, Tables.getcolumn(Data, :Day_Sim_Start))

      Year_Sim_Start  = convert(Vector{Float64}, Tables.getcolumn(Data, :Year_Sim_Start))
      Month_Sim_Start = convert(Vector{Float64}, Tables.getcolumn(Data, :Month_Sim_Start))
      Day_Sim_Start   = convert(Vector{Float64}, Tables.getcolumn(Data, :Day_Sim_Start))

      Hour_Start=[9]
      Hour_Start_Sim=[9]
      Hour_End =[9]

      NEW_FILE(iClimateInventory, Year_Obs_Start, Year_Sim_Start, Year_Obs_End ,Month_Obs_Start, Month_Sim_Start ,Month_Obs_End, Day_Obs_Start ,Day_Sim_Start ,Day_Obs_End ,Hour_Start,Hour_Start_Sim ,Hour_End, i)
   end
   
return nothing
end  # function: JULES_CLIMATE
# ------------------------------------------------------------------


# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
#		FUNCTION : NEW_FILE
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
function NEW_FILE(iClimateInventory, Year_Start, Year_Start_Sim, Year_End ,Month_Start, Month_Start_Sim ,Month_End, Day_Start ,Day_Start_Sim ,Day_End ,Hour_Start, Hour_Start_Sim ,Hour_End, i)

   ClimatePath_Output = joinpath(raw"D:\DATAraw\JULES_DATA\DATE\DATE_SUMMARY.csv" )
      
   Header = [ "Station", "Year_Start" ,"Year_Start_Sim", "Year_End" , "Month_Start" ,"Month_Start_Sim" ,"Month_End" ,"Day_Start", "Day_Start_Sim" ,"Day_End" ,"Hour_Start" ,"Hour_Start_Sim", "Hour_End"]

   CSV.write(ClimatePath_Output, Tables.table([iClimateInventory Year_Start Year_Start_Sim Year_End Month_Start Month_Start_Sim Month_End Day_Start Day_Start_Sim Day_End Hour_Start Hour_Start_Sim Hour_End]), bom=true, append=true,   writeheader = true,header=Header, delim = ",")

      
   return nothing
end  # function: NEW_FILE
# ------------------------------------------------------------------