Most of the time spent in `readdlm` a txt file

General Usage Performance
1

Hi there,

I’m my attempt to learn Julia, I’m solving project euler problems. In one of them, I found the performance not particularly good, when compared with other languages, e.g., Python.

The code in question is the following:

function problemXXX()
    file = "../src/files/ProblemXXX.txt"
    names = sort(readdlm(file, ',', String)[:])

    result = 0
    for (idx, name) in enumerate(names)
        result += sum(Int(c) - 64 for c in name) * idx
    end
    return result
end

@time ProblemXXX()
  0.352722 seconds (262.48 k allocations: 13.976 MiB, 98.79% compilation time)

If I just read the file and return the first item, without doing any calculation, I got this result:

function problemXXX()
    file = "../src/files/ProblemXXX.txt"
    names = sort(readdlm(file, ',', String)[:])
    return names[1]
end

@time ProblemXXX()
  0.289460 seconds (133.07 k allocations: 7.343 MiB, 99.56% compilation time)

It means that most of the time is just spent in reading the file?

The file, provided by the problem, is just a single line string of 46k names surrounded by double quotes and separated by commas, e.g., “MARY”, “AARON”, “DAVID”, …

Any ideas of how to improve the speed, memory allocations?

Thx in advance :slight_smile:

2

The answer is in here: You are also measuring compilation time (which also includes the time it takes to compile the readdlm functions. If you run this line with @time a second time, it will be much faster to run.
It is usually recommended to keep the Julia session open as long as possible to not recompile everything all over again each time you want something done. If it is absolutely neccessary, there are a few workarounds like PackageCompiler.jl (and DaemonMode.jl although I’ve never tried it).

1 Like