Good day.
I have 12 cores. So nthread() gives 12.
when I used "Threads.@threads for ", only 8 thread was created.
RAM memory was utilized for up to 85%.
Can memory limitations and the number of thread created be connected?
thanks.
Good day.
I have 12 cores. So nthread() gives 12.
when I used "Threads.@threads for ", only 8 thread was created.
RAM memory was utilized for up to 85%.
Can memory limitations and the number of thread created be connected?
thanks.
Please show what code you ran.
function main()
files=readdir()
Threads.@threads for file in files
if file[end-1:end]=="gz"
println("threadid-",threadid(), " ", file)
df=ParseFile(file)
println("threadid-",threadid(), " ","Parsefile")
df=KeepUniqueId(df)
println("threadid-",threadid(), " ","KeepUnique")
dfNews=ParseSubjects(df)
println("threadid-",threadid(), " ","ParseSubjests")
filename=file[1:end-7]
CSV.write("$file.csv",dfNews)
end
end
end
The DataFrame size is around 1Gb.
See the first item here: Please read: make it easier to help you
How many files are in that directory?
Around 10-12. it’s more than a number of threads.
Maybe I got the answer
“Threads.@threads for” tries distribute evenly a number of threads since when I have exactly 12 files, 12 threads were created.