Reading text or csv files and assigning values to variable names

New to Julia
1

Hi, I have several large text files write out from a VBA program. The files contain numbers only (integers, floating point). I want to read these files into a Julia program. In FORTRAN or VBA you would just use Read(“File.txt (or .csv)”) a, b, c …z where a to z are the variable names. The numerical values in the text file would be automatically assigned to a, b, c in the order they are read.

Is there an equivalent in Julia?

I have spent hours googling but to no avail. The code below works but all the inputs are read into a string s which I guess could be decomposed. Surely there is an easier way as I literally have 100’s of inputs I want to assign to variable names (and possible named tuples)

open(“C:\Users\peter\Documents\Julia_Code\Learning\MyFile.txt”,“r”) do f

 #line_number 
 line = 0   #

# read till end of file 
while ! eof(f)   

   # read a new / next line for every iteration            
    s = readline(f)  
      
   line += 1
    
   println("$s")

end
2

If you want to work with CSV files, you definitely should check out GitHub - JuliaData/CSV.jl: Utility library for working with CSV and other delimited files in the Julia programming language

1 Like
3

If you want to assign individual variable names to each item in the file you can do the following. Imagine you have a file called “test.txt” with the content

1
123.123
-100.0

Now just use:

julia> using DelimitedFiles

julia> a,b,c = readdlm("test.txt");

julia> a
1.0

julia> b
123.123

julia> c
-100.0
2 Likes
4

Chris,

That looks interesting and should work with some tweaking I am thinking. My file has all values on ONE line, with spaces between the values, as follows:

1 123.123 -100 …….

6

This should work regardless of whether the values are on a single line or on separate lines.

What’s going on here is that readdlm() returns a 1xN or Nx1 matrix and the variables on the left hand side are assigned to elements of this array in iteration order (from first to last). This isn’t about files, it works for any array:

julia> values = [1,2,3]
3-element Array{Int64,1}:
 1
 2
 3

julia> a, b, c = values
3-element Array{Int64,1}:
 1
 2
 3

julia> a
1

julia> b
2

julia> c
3
1 Like
7

Chris

Understand. I will purse further tomorrow with different file formats and delimiters.

Thanks

Peter

8

Chris,

I have made up both “vertical” and “horizontal” *.dat files for testing (writing a *.dat file from VBA automatically puts a “,” between the data values rather than a space if you use the *.txt format. I think the “,” is a safer bet going forward). When I run the code below I keep getting an error (see below), no matter what *.dat data file I use. Any ideas?

Thanks,

Peter

using DelimitedFiles

Alpha_Leaf_Air, Latent, Sigma, e_Can = readdlm(“C:\Users\peter\Documents\Julia_Code\Learning\MyFile_Horizontal.dat”);

println(Latent)

ERROR: LoadError: BoundsError: attempt to access 1×1 Array{Any,2} at index [2]

Stacktrace:

[1] getindex at .\array.jl:809 [inlined]

[2] indexed_iterate(::Array{Any,2}, ::Int64, ::Int64) at .\tuple.jl:82

[3] top-level scope at c:\Users\peter\Documents\Julia_Code\Learning\Read_Write_Dat_File.jl:9

[4] include_string(::Function, ::Module, ::String, ::String) at .\loading.jl:1088

[5] include_string(::Module, ::String, ::String) at .\loading.jl:1096

[6] invokelatest(::Any, ::Any, ::Vararg{Any,N} where N; kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at .\essentials.jl:710

[7] invokelatest(::Any, ::Any, ::Vararg{Any,N} where N) at .\essentials.jl:709

[8] inlineeval(::Module, ::String, ::Int64, ::Int64, ::String; softscope::Bool) at c:\Users\peter.vscode\extensions\julialang.language-julia-1.0.10\scripts\packages\VSCodeServer\src\eval.jl:185

[9] (::VSCodeServer.var"#61#65"{String,Int64,Int64,String,Module,Bool,VSCodeServer.ReplRunCodeRequestParams})() at c:\Users\peter.vscode\extensions\julialang.language-julia-1.0.10\scripts\packages\VSCodeServer\src\eval.jl:144

[10] withpath(::VSCodeServer.var"#61#65"{String,Int64,Int64,String,Module,Bool,VSCodeServer.ReplRunCodeRequestParams}, ::String) at c:\Users\peter.vscode\extensions\julialang.language-julia-1.0.10\scripts\packages\VSCodeServer\src\repl.jl:124

[11] (::VSCodeServer.var"#60#64"{String,Int64,Int64,String,Module,Bool,Bool,VSCodeServer.ReplRunCodeRequestParams})() at c:\Users\peter.vscode\extensions\julialang.language-julia-1.0.10\scripts\packages\VSCodeServer\src\eval.jl:142

[12] hideprompt(::VSCodeServer.var"#60#64"{String,Int64,Int64,String,Module,Bool,Bool,VSCodeServer.ReplRunCodeRequestParams}) at c:\Users\peter.vscode\extensions\julialang.language-julia-1.0.10\scripts\packages\VSCodeServer\src\repl.jl:36

[13] (::VSCodeServer.var"#59#63"{String,Int64,Int64,String,Module,Bool,Bool,VSCodeServer.ReplRunCodeRequestParams})() at c:\Users\peter.vscode\extensions\julialang.language-julia-1.0.10\scripts\packages\VSCodeServer\src\eval.jl:110

[14] with_logstate(::Function, ::Any) at .\logging.jl:408

[15] with_logger at .\logging.jl:514 [inlined]

[16] (::VSCodeServer.var"#58#62"{VSCodeServer.ReplRunCodeRequestParams})() at c:\Users\peter.vscode\extensions\julialang.language-julia-1.0.10\scripts\packages\VSCodeServer\src\eval.jl:109

[17] #invokelatest#1 at .\essentials.jl:710 [inlined]

[18] invokelatest(::Any) at .\essentials.jl:709

[19] macro expansion at c:\Users\peter.vscode\extensions\julialang.language-julia-1.0.10\scripts\packages\VSCodeServer\src\eval.jl:27 [inlined]

[20] (::VSCodeServer.var"#56#57")() at .\task.jl:356

in expression starting at c:\Users\peter\Documents\Julia_Code\Learning\Read_Write_Dat_File.jl:9

(Attachment MyFile_Horizontal.dat is missing)

(Attachment MyFile_Vertical.dat is missing)

9

Hi Peter,

you probably just need to put the delimiter into readdlm, e.g.

a,b,c = readdlm("data.dat", ',')
2 Likes
10

Paul and Chris, yes that works for the “horizontal file” - thanks. Still having problems with the “vertical file” I have tried without delimiters (only one value on each line) and with ‘\n’ to capture the end of line but to no avail. Any ideas? Thanks Peter

11

Chris and Paul

Now both vertical and horizontal files working - many thanks. Problems with the vertical file were self inflicted as I edited the horizontal in the Vscode environment to a vertical file - it all looked good in this environment BUT when I viewed the file using notepad the file was still in horizontal format. Once I generated the file directly in notepad all was well. Sounds strange I know - a warning to others.
Onwards and up!

Peter

12

Sure, glad you’ve got it sorted out.

By the way, I should point out that DelimitedFiles isn’t as powerful or fast as the CSV.jl module and that should be preferred for “serious” amounts of CSV-like data. But DelimitedFiles.readdlm() is installed by default and it’s ok for quick simple tasks like this.

13

Chris,

Thanks. I have a bit of familiarity with CSV – will adopt going forward as much of what I have to do is read and write data to spreadsheets. A more immediate problem, is how to write data from Julia to a text file (reverse of the read problem just solved). The code below writes each of the four items to a separate line rather than all on one line. I have tried putting the four outputs into a 1 x 4 array but the outputs are still printed one on each line…. Rrrr. Any ideas?

using DelimitedFiles

Alpha_Leaf_Air, Latent, Sigma, e_Can = readdlm(“C:\Users\peter\Documents\Julia_Code\Learning\MyFile_Horizontal.txt”,‘,’);

println(Alpha_Leaf_Air,", “, Latent,”, “, Sigma,”, ", e_Can)

#Alpha_Leaf_Air, Latent, Sigma, e_Can = readdlm(“C:\Users\peter\Documents\Julia_Code\Learning\MyFile_Vertical.txt”);

#println(Latent,", ",Sigma)

x = Array{Float64}(undef,1, 4) # N given explicitly

x=[Alpha_Leaf_Air, Latent, Sigma, e_Can];

writedlm(“C:\Users\peter\Documents\Julia_Code\Learning\delim_file.txt”,x,‘,’)

14

Hi Peter, to make it easier to understand your questions here, do read Please read: make it easier to help you — especially as regards to formatting guidelines, as your code will have formatting mangled if you just paste it in. Also, it’s appreciated if you can post a complete and minimal example, or if not possible at least something self-contained. If you’re trying to read text files, that should include some small example content of the file in question. Without a complete example, we can be left guessing your intent.

1 Like
15

If you are reading and writing spreadsheet-like files, you really want to move to CSV.jl, possibly in combination with DataFrames.jl in which case

df = CSV.read("MyFile_Horizontal.txt", DataFrame)

returns a DataFrame where presumably your columns would be Alpha_Leaf_Air, Latent etc. (depending on how your horizontal/vertical file is structured exactly), and

CSV.write("out.csv", df)

would write this table back out to disk as a csv.

1 Like
16

How about something simple, such as:

open("myfile.txt", "w") do f
    print(f, 45,' ', 232.0,' ',22,' ',3,' ',44.342,' ',324)
end;

a,b,c,d,e,f=parse.(Float64, split(readline("myfile.txt"), ' '))


println(a, ' ', b, ' ', c, ' ' , d, ' ', e)
17

Will do. Thanks for tip

1 Like
18

I have just tried this and it will read a line of comma separated data and then print is out. Thanks. However, what I need is to assign each of the values that I read in to variable names, Alpha_Leaf_Air, Latent, Sigma, e_Can … In the Julia program I will use these variables for calculation to create new variables/results. These results will then be written to a *.csv file and ultimately read by my VBA program for plotting and others that need to access the VBA program.

So my specific problems are with the CSV usage are:

  1. Reading a text file (horizontal data) and assigning each value to a variable name (I have tried the technique that successfully works when using DelimitedFiles but to no avail when using CSV). Is there a way to deconstruct df in your code into the various variable names?
  2. Writing results to a file a in horizontal format. Again, how to specify a list of variables to write data in a horizontal form? I am yet to work this out using DelimitedFiles as well
    Thanks Peter
19

The deconstruction works the same way as shown above if you use the eachcol iterator:

julia> using DataFrames

julia> df = DataFrame(rand(5, 5), :auto)
5×5 DataFrame
 Row │ x1         x2        x3         x4         x5        
     │ Float64    Float64   Float64    Float64    Float64   
─────┼──────────────────────────────────────────────────────
   1 │ 0.386535   0.657961  0.397478   0.0570275  0.114564
   2 │ 0.513711   0.210151  0.0745352  0.520141   0.392656
   3 │ 0.428348   0.552453  0.741062   0.0550832  0.572661
   4 │ 0.404761   0.761791  0.362254   0.765474   0.0324307
   5 │ 0.0781592  0.544008  0.588868   0.161822   0.136142

julia> a, b, c, d, e = eachcol(df);

julia> a
5-element Vector{Float64}:
 0.3865349792714514
 0.5137106547733481
 0.42834760490136525
 0.40476068514594465
 0.07815920354162742

The other way of course would be to just work with the DataFrame columns, so use df.x1 (or df.a if your input file has headers that have the same names as the variables you want to construct).

If you work directly with the columns, then writing out would work as described. If you have variables you want to write out, you can either wrap them in a DataFrame or any object that satisfies the Tables.jl interface, the simplest being a NamedTuple, so e.g.

CSV.write("out.csv", (Alpha_Leaf_Air = Alpha_Leaf_Air, Latent = Latent))
20

Chris I hope this code is not mangled:


 a,b,c,d = readdlm("C:\\Users\\peter\\Documents\\Julia_Code\\Learning\\MyFile_Horizontal.txt",',');

  x = a,b,c,d

 writedlm("C:\\Users\\peter\\Documents\\Julia_Code\\Learning\\delim_file.txt",x,',')

Please note that I can read a comma separated horizontal file and correctly assign the values to a, b, c and d.
MyFile_Horizontal.txt is:
5, 2450000, 0.0000000567, 1
I am unable to write the data back in horizontal form, rather it appears in vertical form as below:
5.0
2.45e6
5.67e-8
1.0

I have tried putting the individual variable names a, b, c and d into the writedlm statement but this is not accepted. What am I missing?

Thanks again. Peter

21

I take your point about using NamedTuples, and in due course I will use to output data/values calculated in the Julia code. My immediate issue is to solve the trivial problem below. Please note my data input file has no headings at this time, rather just the numbers:
5,2450000,.0000000567,1

Without headings how to assign a, b, c and d to the appropriate values? With the eachcol(df) I get a “missing error”, presumably because there are no headings. Sorry to be a pest … such a trivial problem it is becoming a bother for everyone I am sure.


 using DataFrames

 df = CSV.read("C:\\Users\\peter\\Documents\\Julia_Code\\Learning\\MyFile_Horizontal.txt", DataFrame)

 a, b, c, d = eachcol(df);

 print(a)

 CSV.write("C:\\Users\\peter\\Documents\\Julia_Code\\Learning\\out.csv",df)