Reading text or csv files and assigning values to variable names

Chris

Understand. I will purse further tomorrow with different file formats and delimiters.

Thanks

Peter

Chris,

I have made up both “vertical” and “horizontal” *.dat files for testing (writing a *.dat file from VBA automatically puts a “,” between the data values rather than a space if you use the *.txt format. I think the “,” is a safer bet going forward). When I run the code below I keep getting an error (see below), no matter what *.dat data file I use. Any ideas?

Thanks,

Peter

using DelimitedFiles

Alpha_Leaf_Air, Latent, Sigma, e_Can = readdlm(“C:\Users\peter\Documents\Julia_Code\Learning\MyFile_Horizontal.dat”);

println(Latent)

ERROR: LoadError: BoundsError: attempt to access 1×1 Array{Any,2} at index [2]

Stacktrace:

[1] getindex at .\array.jl:809 [inlined]

[2] indexed_iterate(::Array{Any,2}, ::Int64, ::Int64) at .\tuple.jl:82

[3] top-level scope at c:\Users\peter\Documents\Julia_Code\Learning\Read_Write_Dat_File.jl:9

[4] include_string(::Function, ::Module, ::String, ::String) at .\loading.jl:1088

[5] include_string(::Module, ::String, ::String) at .\loading.jl:1096

[6] invokelatest(::Any, ::Any, ::Vararg{Any,N} where N; kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at .\essentials.jl:710

[7] invokelatest(::Any, ::Any, ::Vararg{Any,N} where N) at .\essentials.jl:709

[8] inlineeval(::Module, ::String, ::Int64, ::Int64, ::String; softscope::Bool) at c:\Users\peter.vscode\extensions\julialang.language-julia-1.0.10\scripts\packages\VSCodeServer\src\eval.jl:185

[9] (::VSCodeServer.var"#61#65"{String,Int64,Int64,String,Module,Bool,VSCodeServer.ReplRunCodeRequestParams})() at c:\Users\peter.vscode\extensions\julialang.language-julia-1.0.10\scripts\packages\VSCodeServer\src\eval.jl:144

[10] withpath(::VSCodeServer.var"#61#65"{String,Int64,Int64,String,Module,Bool,VSCodeServer.ReplRunCodeRequestParams}, ::String) at c:\Users\peter.vscode\extensions\julialang.language-julia-1.0.10\scripts\packages\VSCodeServer\src\repl.jl:124

[11] (::VSCodeServer.var"#60#64"{String,Int64,Int64,String,Module,Bool,Bool,VSCodeServer.ReplRunCodeRequestParams})() at c:\Users\peter.vscode\extensions\julialang.language-julia-1.0.10\scripts\packages\VSCodeServer\src\eval.jl:142

[12] hideprompt(::VSCodeServer.var"#60#64"{String,Int64,Int64,String,Module,Bool,Bool,VSCodeServer.ReplRunCodeRequestParams}) at c:\Users\peter.vscode\extensions\julialang.language-julia-1.0.10\scripts\packages\VSCodeServer\src\repl.jl:36

[13] (::VSCodeServer.var"#59#63"{String,Int64,Int64,String,Module,Bool,Bool,VSCodeServer.ReplRunCodeRequestParams})() at c:\Users\peter.vscode\extensions\julialang.language-julia-1.0.10\scripts\packages\VSCodeServer\src\eval.jl:110

[14] with_logstate(::Function, ::Any) at .\logging.jl:408

[15] with_logger at .\logging.jl:514 [inlined]

[16] (::VSCodeServer.var"#58#62"{VSCodeServer.ReplRunCodeRequestParams})() at c:\Users\peter.vscode\extensions\julialang.language-julia-1.0.10\scripts\packages\VSCodeServer\src\eval.jl:109

[17] #invokelatest#1 at .\essentials.jl:710 [inlined]

[18] invokelatest(::Any) at .\essentials.jl:709

[19] macro expansion at c:\Users\peter.vscode\extensions\julialang.language-julia-1.0.10\scripts\packages\VSCodeServer\src\eval.jl:27 [inlined]

[20] (::VSCodeServer.var"#56#57")() at .\task.jl:356

in expression starting at c:\Users\peter\Documents\Julia_Code\Learning\Read_Write_Dat_File.jl:9

(Attachment MyFile_Horizontal.dat is missing)

(Attachment MyFile_Vertical.dat is missing)

Hi Peter,

you probably just need to put the delimiter into readdlm, e.g.

a,b,c = readdlm("data.dat", ',')
1 Like

Paul and Chris, yes that works for the “horizontal file” - thanks. Still having problems with the “vertical file” I have tried without delimiters (only one value on each line) and with ‘\n’ to capture the end of line but to no avail. Any ideas? Thanks Peter

Chris and Paul

Now both vertical and horizontal files working - many thanks. Problems with the vertical file were self inflicted as I edited the horizontal in the Vscode environment to a vertical file - it all looked good in this environment BUT when I viewed the file using notepad the file was still in horizontal format. Once I generated the file directly in notepad all was well. Sounds strange I know - a warning to others.
Onwards and up!

Peter

Sure, glad you’ve got it sorted out.

By the way, I should point out that DelimitedFiles isn’t as powerful or fast as the CSV.jl module and that should be preferred for “serious” amounts of CSV-like data. But DelimitedFiles.readdlm() is installed by default and it’s ok for quick simple tasks like this.

Chris,

Thanks. I have a bit of familiarity with CSV – will adopt going forward as much of what I have to do is read and write data to spreadsheets. A more immediate problem, is how to write data from Julia to a text file (reverse of the read problem just solved). The code below writes each of the four items to a separate line rather than all on one line. I have tried putting the four outputs into a 1 x 4 array but the outputs are still printed one on each line…. Rrrr. Any ideas?

using DelimitedFiles

Alpha_Leaf_Air, Latent, Sigma, e_Can = readdlm(“C:\Users\peter\Documents\Julia_Code\Learning\MyFile_Horizontal.txt”,‘,’);

println(Alpha_Leaf_Air,", “, Latent,”, “, Sigma,”, ", e_Can)

#Alpha_Leaf_Air, Latent, Sigma, e_Can = readdlm(“C:\Users\peter\Documents\Julia_Code\Learning\MyFile_Vertical.txt”);

#println(Latent,", ",Sigma)

x = Array{Float64}(undef,1, 4) # N given explicitly

x=[Alpha_Leaf_Air, Latent, Sigma, e_Can];

writedlm(“C:\Users\peter\Documents\Julia_Code\Learning\delim_file.txt”,x,‘,’)

Hi Peter, to make it easier to understand your questions here, do read Please read: make it easier to help you — especially as regards to formatting guidelines, as your code will have formatting mangled if you just paste it in. Also, it’s appreciated if you can post a complete and minimal example, or if not possible at least something self-contained. If you’re trying to read text files, that should include some small example content of the file in question. Without a complete example, we can be left guessing your intent.

1 Like

If you are reading and writing spreadsheet-like files, you really want to move to CSV.jl, possibly in combination with DataFrames.jl in which case

df = CSV.read("MyFile_Horizontal.txt", DataFrame)

returns a DataFrame where presumably your columns would be Alpha_Leaf_Air, Latent etc. (depending on how your horizontal/vertical file is structured exactly), and

CSV.write("out.csv", df)

would write this table back out to disk as a csv.

1 Like

How about something simple, such as:

open("myfile.txt", "w") do f
    print(f, 45,' ', 232.0,' ',22,' ',3,' ',44.342,' ',324)
end;

a,b,c,d,e,f=parse.(Float64, split(readline("myfile.txt"), ' '))


println(a, ' ', b, ' ', c, ' ' , d, ' ', e)

Will do. Thanks for tip

1 Like

I have just tried this and it will read a line of comma separated data and then print is out. Thanks. However, what I need is to assign each of the values that I read in to variable names, Alpha_Leaf_Air, Latent, Sigma, e_Can … In the Julia program I will use these variables for calculation to create new variables/results. These results will then be written to a *.csv file and ultimately read by my VBA program for plotting and others that need to access the VBA program.

So my specific problems are with the CSV usage are:

  1. Reading a text file (horizontal data) and assigning each value to a variable name (I have tried the technique that successfully works when using DelimitedFiles but to no avail when using CSV). Is there a way to deconstruct df in your code into the various variable names?
  2. Writing results to a file a in horizontal format. Again, how to specify a list of variables to write data in a horizontal form? I am yet to work this out using DelimitedFiles as well
    Thanks Peter

The deconstruction works the same way as shown above if you use the eachcol iterator:

julia> using DataFrames

julia> df = DataFrame(rand(5, 5), :auto)
5×5 DataFrame
 Row │ x1         x2        x3         x4         x5        
     │ Float64    Float64   Float64    Float64    Float64   
─────┼──────────────────────────────────────────────────────
   1 │ 0.386535   0.657961  0.397478   0.0570275  0.114564
   2 │ 0.513711   0.210151  0.0745352  0.520141   0.392656
   3 │ 0.428348   0.552453  0.741062   0.0550832  0.572661
   4 │ 0.404761   0.761791  0.362254   0.765474   0.0324307
   5 │ 0.0781592  0.544008  0.588868   0.161822   0.136142

julia> a, b, c, d, e = eachcol(df);

julia> a
5-element Vector{Float64}:
 0.3865349792714514
 0.5137106547733481
 0.42834760490136525
 0.40476068514594465
 0.07815920354162742

The other way of course would be to just work with the DataFrame columns, so use df.x1 (or df.a if your input file has headers that have the same names as the variables you want to construct).

If you work directly with the columns, then writing out would work as described. If you have variables you want to write out, you can either wrap them in a DataFrame or any object that satisfies the Tables.jl interface, the simplest being a NamedTuple, so e.g.

CSV.write("out.csv", (Alpha_Leaf_Air = Alpha_Leaf_Air, Latent = Latent))

Chris I hope this code is not mangled:


 a,b,c,d = readdlm("C:\\Users\\peter\\Documents\\Julia_Code\\Learning\\MyFile_Horizontal.txt",',');

  x = a,b,c,d

 writedlm("C:\\Users\\peter\\Documents\\Julia_Code\\Learning\\delim_file.txt",x,',')

Please note that I can read a comma separated horizontal file and correctly assign the values to a, b, c and d.
MyFile_Horizontal.txt is:
5, 2450000, 0.0000000567, 1
I am unable to write the data back in horizontal form, rather it appears in vertical form as below:
5.0
2.45e6
5.67e-8
1.0

I have tried putting the individual variable names a, b, c and d into the writedlm statement but this is not accepted. What am I missing?

Thanks again. Peter

I take your point about using NamedTuples, and in due course I will use to output data/values calculated in the Julia code. My immediate issue is to solve the trivial problem below. Please note my data input file has no headings at this time, rather just the numbers:
5,2450000,.0000000567,1

Without headings how to assign a, b, c and d to the appropriate values? With the eachcol(df) I get a “missing error”, presumably because there are no headings. Sorry to be a pest … such a trivial problem it is becoming a bother for everyone I am sure.


 using DataFrames

 df = CSV.read("C:\\Users\\peter\\Documents\\Julia_Code\\Learning\\MyFile_Horizontal.txt", DataFrame)

 a, b, c, d = eachcol(df);

 print(a)

 CSV.write("C:\\Users\\peter\\Documents\\Julia_Code\\Learning\\out.csv",df)

For writedlm to write the values on the same line, you need to do something like this:

julia> using DelimitedFiles

julia> a,b,c,d = readdlm("MyFile_Horizontal.txt", ',');

julia> writedlm("delim_file.txt", [a b c d], ", ")

shell> cat delim_file.txt
5.0, 2.45e6, 5.67e-8, 1.0

Explanation: writedlm respects the rows/columns of a matrix. For other types of collections, it iterates and writes one value per line. From the documentation:

Write A (a vector, matrix, or an iterable collection of iterable rows) as text to f

So it iterates on the tuple created with x = a,b,c,d, and you get values on different lines.

3 Likes

And using CSV and DataFrames:

julia> using DataFrames, CSV

julia> df = CSV.read("MyFile_Horizontal.txt",  DataFrame, header=[:a, :b, :c, :d])
1×4 DataFrame
│ Row │ a     │ b       │ c       │ d     │
│     │ Int64 │ Int64   │ Float64 │ Int64 │
├─────┼───────┼─────────┼─────────┼───────┤
│ 1   │ 5     │ 2450000 │ 5.67e-8 │ 1     │

julia> CSV.write("delim_file.txt", df, header=false)
"delim_file.txt"

shell> cat delim_file.txt
5,2450000,5.67e-8,1

You can also call CSV.read with header=false to let it choose column names.

Edit: I forgot about storing the DataFrame values in individual variables:

julia> a,b,c,d = df[1,:];

julia> CSV.write("delim_file.txt", Tables.table([a b c d]), header=false)
"delim_file.txt"

shell> cat delim_file.txt
5.0,2.45e6,5.67e-8,1.0

but for just a few variables from a single line of text I would rather stay with DelimitedFiles.

5 Likes

I like this solution. I have added a couple of extra lines to that reflect the intermediate operations on a, b,c … and then print out the new result. Many thanks for your clear explanation. It seems a lot simpler than using CSV. Regards Peter


 a,b,c,d = readdlm("C:\\Users\\peter\\Documents\\Julia_Code\\Learning\\MyFile_Horizontal.txt",',');

  x = a+c

  y = b+c

  println(x,", ",y)

 writedlm("C:\\Users\\peter\\Documents\\Julia_Code\\Learning\\delim_file.txt",[x y],',')

I changed your program a little to reflect intermediate actions. It seems to work. If I wanted to print a header x, y on the output file what is the best way? If I use header = true then the headings become column1 and column 2 not x and y. Thanks Peter


 using DataFrames

 df = CSV.read("C:\\Users\\peter\\Documents\\Julia_Code\\Learning\\MyFile_Horizontal.txt", DataFrame, header=[:a, :b, :c, :d])

 a, b, c, d = eachcol(df);

  x= a + d 

  y=c + d

 CSV.write("C:\\Users\\peter\\Documents\\Julia_Code\\Learning\\out.csv",Tables.table([x y]),header=false)

Thanks, much nicer!

You can set header = ["x", "y"] for this:

julia> using Tables, CSV

julia> CSV.write("file.csv", Tables.table([1 2]), header=["x", "y"])
"file.csv"

julia> print(read("file.csv", String))
x,y
1,2

Personally I think that, if you can choose your file format you might not want CSV if this is a configuration file with a header and never more than one row of values.

Instead, an ini-like format like TOML is preferable to a tabular format for configuration because it’s more flexible. For example to create some configuration and write it to the file:

julia> using TOML

julia> config = Dict("x"=>1, "y"=>2, "z"=>[4,5,6])
Dict{String,Any} with 3 entries:
  "x" => 1
  "z" => [4, 5, 6]
  "y" => 2

julia> open(io->TOML.print(io, config), "config.toml", write=true)

Here’s the content of the generated “config.toml”

julia> print(read("config.toml", String))
x = 1
z = [4, 5, 6]
y = 2

Now you can read it back in and refer to the variables by name, relative to the config Dict:

julia> config_read = open(io->TOML.parse(io), "config.toml")
Dict{String,Any} with 3 entries:
  "x" => 1
  "z" => [4, 5, 6]
  "y" => 2

julia> config_read["x"]
1

julia> config_read["y"]
2

julia> config_read["z"]
3-element Array{Int64,1}:
 4
 5
 6

Notice how this allows a lot of flexibility in having nested structures of Dicts and Arrays for your program configuration.

2 Likes