How to improve a Generator to be more memory efficient when it is collected?

camilogarciabotero
Fri, 13 Jan 2023 18:10:11 +0000
Performance

Hi all,

I wrote a function that searches through a sequence (a BioSequences.LongDNA but could also be a String) for start codons (predefined as constants subsequence searches) and generates ranges of indices corresponding to the start of start codons of the sequence. Later, when two conditions are met (no “premature” stops and length), then it generates stop range. It is fast when defined, but very slow when collected and allocates a lot of memory for a small sequence:

const STOPCODONS = [dna"TAG", dna"TAA"), dna"TGA"]
const STARTCODON = ExactSearchQuery(dna"ATG", iscompatible)
const EXTENDED_STARTCODONS = ExactSearchQuery(dna"DTG", iscompatible)

function eachcodon(sequence::LongDNA)
    seqbound = length(sequence) - 2
    return(sequence[i:i+2] for i in 1:3:seqbound)
end

function hasprematurestop(sequence::LongDNA)::Bool
    stop_codon_count = 0
    @inbounds for codon in eachcodon(sequence)
        if codon ∈ STOPCODONS
            stop_codon_count += 1
        end
    end
    stop_codon_count > 1
end

function locationgenerator(sequence::LongDNA; alternative_start::Bool=false, min_len::Int64=6)
    seqbound = length(sequence) - 2
    startcodons = alternative_start ? EXTENDED_STARTCODONS : STARTCODON
    start_codon_indices = findall(startcodons, sequence)
    @inbounds begin
        location = (i.start:j+2 for i in start_codon_indices for j in i.start:3:seqbound if sequence[j:j+2] ∈ STOPCODONS && !hasprematurestop(sequence[i.start:j+2]) && length(i.start:j+2) >= min_len)
    end
    return location
end

Here a small benchmark:

using BioSequences
using BenchmarkTools
seq = randdnaseq(1*10^4) # only 10Kb string

@benchmark collect(locationgenerator(seq))

BenchmarkTools.Trial: 5 samples with 1 evaluation.
 Range (min … max):  1.197 s …   1.250 s  ┊ GC (min … max): 5.80% … 6.28%
 Time  (median):     1.222 s              ┊ GC (median):    6.42%
 Time  (mean ± σ):   1.220 s ± 21.299 ms  ┊ GC (mean ± σ):  6.24% ± 0.33%

  █      █                  █      █                      █
  █▁▁▁▁▁▁█▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁█▁▁▁▁▁▁█▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁█ ▁
  1.2 s          Histogram: frequency by time        1.25 s <

 Memory estimate: 1.29 GiB, allocs estimate: 28430466.

Any suggestions on how to improve this?

22 posts - 5 participants