# Slow and Memory intensive For Loop
**URL:** https://discourse.julialang.org/t/slow-and-memory-intensive-for-loop/112304
**Category:** Performance
**Tags:** question
**Created:** [March 29, 2024, 5:13pm UTC](https://discourse.julialang.org/t/slow-and-memory-intensive-for-loop/112304 "2024-03-29T17:13:51Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@soumen261](https://discourse.julialang.org/u/soumen261)
#### Post date: [March 29, 2024, 5:13pm UTC](https://discourse.julialang.org/t/slow-and-memory-intensive-for-loop/112304/1 "2024-03-29T17:13:51Z")
Hi Julians,
I am still a novice in Julia. I am writing a code to import/read a total 16 data (.txt) files each having 504\*1025 matrix. Essentially trying to make a 3D data cube from the 2D datasets using a For loop. But it takes 6.9 seconds for the code with 22 M allocation. I will really appreciate any suggestions and advice to make the code faster. Please see the code below:
```julia
using Gtk4
using DelimitedFiles
function LoopScans()
file_path = open_dialog("My Open dialog")
file_location, filename = dirname(file_path), basename(file_path)
Rawdata = readdlm(file_path)'
file_name, file_ext = split(filename, ".")
filestring, ScanIndex = match(r"[A-Za-z]+", file_name).match, parse(Int, match(r"\d+", file_name).match)
files = readdir(file_location)
# Filter files based on the pattern
matching_files = filter(f -> occursin(filestring * r"\d+\." * file_ext, f), files)
isempty(matching_files) && throw(ArgumentError("No files found matching pattern"))
sample_data = readdlm(joinpath(file_location, matching_files[1]))'
allScans = Array{Float64, 3}(undef, size(sample_data)..., length(matching_files))
for (ii, file) in enumerate(matching_files)
allScans[:, :, ii] = readdlm(joinpath(file_location, file))'
end
return allScans
end
@time LoopScans()
```
* * *
```julia
6.968473 seconds (23.50 M allocations: 961.212 MiB, 3.03% gc time, 5.37% compilation time)
1025×504×16 Array{Float64, 3}:
```
* * *
---
### Author:  [@ufechner7](https://discourse.julialang.org/u/ufechner7)
#### Post date: [March 29, 2024, 5:34pm UTC](https://discourse.julialang.org/t/slow-and-memory-intensive-for-loop/112304/2 "2024-03-29T17:34:06Z")
Welcome to the forum!
Some remarks:
1. please, enclose any code you share in triple back-ticks like ```
2. if you call a function the first time it gets compiled, so call it a second time to determine the execution time
3. avoid global variables if you need fast execution time
> [@Please read: make it easier to help you](https://discourse.julialang.org/t/please-read-make-it-easier-to-help-you/14757):
>
> Welcome to the Julia Discourse! We are enthusiastic about helping Julia programmers, both beginner and experienced. This public service announcement (PSA) outlines best practices when asking for help. Following these points makes it easier for us to help you and more likely you’ll get a prompt, useful answer. Keywords are highlighted to make it easier to refer to specific points. Choose a descriptive title that captures the key part of your question, eg “plots with multiple axes” instead of …
Finally, in this line:
```julia
allScans[:, :, ii] = readdlm(joinpath(file_location, file))
```
try to replace `=` with `.=` to avoid allocations.
---
### Author:  [@stevengj](https://discourse.julialang.org/u/stevengj)
#### Post date: [March 29, 2024, 9:20pm UTC](https://discourse.julialang.org/t/slow-and-memory-intensive-for-loop/112304/3 "2024-03-29T21:20:47Z")
> [@ufechner7](#):
>
> try to replace `=` with `.=` to avoid allocations.
That makes no difference in this example. `a[:, :, i] = function_returns_array()` allocates an array for the right-hand-side and then writes it in-place into a slice of `a`. Changing `=` to `.=` does the same thing.
---
### Author:  [@sgaure](https://discourse.julialang.org/u/sgaure)
#### Post date: [March 29, 2024, 10:28pm UTC](https://discourse.julialang.org/t/slow-and-memory-intensive-for-loop/112304/4 "2024-03-29T22:28:23Z")
It depends on where the time is spent. If it’s spent on reading from the file system, there is little you can do. If it’s spent on parsing the input into floats, you can read in parallel, if you’ve got more than one cpu/core. Remember to start julia with threads, e.g. `$ julia -t auto` on linux.
```julia
using Base.Threads
...
...
@threads for i in eachindex(matching_files)
allScans[:, :, i] = readdlm(joinpath(file_location, matching_file[i]))
end
```
(note that `@threads` is a bit picky about what you loop over, it should be a vector).
On my box with 8 cpus this gets the time down from 2.7 to 0.6 seconds. But, this depends on disk speed, if the files are in the disk cache, cpu speed, etc, etc.
---
### Author:  [@stevengj](https://discourse.julialang.org/u/stevengj)
#### Post date: [March 29, 2024, 10:49pm UTC](https://discourse.julialang.org/t/slow-and-memory-intensive-for-loop/112304/5 "2024-03-29T22:49:13Z")
> [@sgaure](#):
>
> If it’s spent on parsing the input into floats, you can read in parallel, if you’ve got more than one cpu/core
You should also try a more optimized file-import package like CSV.jl rather than the DelimitedFiles stdlib (which is simple and convenient but relatively slow).
---
### Author:  [@soumen261](https://discourse.julialang.org/u/soumen261)
#### Post date: [March 30, 2024, 5:18pm UTC](https://discourse.julialang.org/t/slow-and-memory-intensive-for-loop/112304/6 "2024-03-30T17:18:49Z")
Thank you all for your suggestions and advice. Using the CSV.jl and threading as suggested by @stevengj and @sgaure does indeed improve the performance. For my Intel Core i7-12700 with 20 cores, the time is reduced from 6.96 seconds to 2.18 seconds (49.92 M allocations: 1.441 GiB). The allocations and memory usage seems too high though. Below is the latest code:
```julia
using DataFrames
using CSV
using Gtk4
using Base.Threads
function LoopScans()::Tuple{Array{Float64,3}, String}
file_path = open_dialog("Select data scans")
file_location, filename = dirname(file_path), basename(file_path)
file_name, file_ext = split(filename, ".")
filestring, ScanIndex = match(r"[A-Za-z]+", file_name).match, parse(Int, match(r"\d+", file_name).match)
files = readdir(file_location)
matching_files = filter(f -> occursin(filestring * r"\d+\." * file_ext, f), files) # Filter files based on the pattern
ScanNumber=length(matching_files)
isempty(matching_files) && throw(ArgumentError("No files found matching pattern"))
Rawdata = DataFrame(CSV.File(joinpath(file_location, matching_files[1])))
if size(Rawdata, 1) > size(Rawdata, 2);
Rawdata = Rawdata;
else
Rawdata = permutedims(Rawdata);
end
allScans = Array{Float64, 3}(undef, size(Rawdata)..., ScanNumber)
@threads for (ii, file) in collect(enumerate(matching_files))
allScans[:, :, ii] .= permutedims(DataFrame(CSV.File(joinpath(file_location, file))))
end
return allScans, file_location
end
@time LoopScans()
```
2.183225 seconds (49.92 M allocations: 1.441 GiB, 3.95% gc time)
---
### Author:  [@ufechner7](https://discourse.julialang.org/u/ufechner7)
#### Post date: [March 30, 2024, 10:44pm UTC](https://discourse.julialang.org/t/slow-and-memory-intensive-for-loop/112304/7 "2024-03-30T22:44:51Z")
> [@soumen261](#):
>
> The allocations and memory usage seems too high though.
You could also try [Introduction · InMemoryDatasets](https://sl-solution.github.io/InMemoryDatasets.jl/stable/) .
The author(s) write:
_The package performance is tuned with two goals in mind, a) low overhead of allowing missing values everywhere, and b) the following priorities - in order of importance:_
- _Low compilation time_
- _Memory efficiency_
- _High performance_
I used it with good success, but if it is better for your use case I do not know.