# How to improve a Generator to be more memory efficient when it is collected?

**URL:** https://discourse.julialang.org/t/how-to-improve-a-generator-to-be-more-memory-efficient-when-it-is-collected/92932
**Category:** Performance
**Tags:** question, regex, optimization, iterators
**Created:** [January 13, 2023, 6:10pm UTC](https://discourse.julialang.org/t/how-to-improve-a-generator-to-be-more-memory-efficient-when-it-is-collected/92932 "2023-01-13T18:10:11Z")
**Posts on this page:** 2
**Page:** 2

<div class="post-metadata">

### Author: ![rocco\_sprmnt21](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rocco_sprmnt21/32/20127_2.png) [@rocco\_sprmnt21](https://discourse.julialang.org/u/rocco_sprmnt21)
#### Post date: [January 16, 2023, 6:24pm UTC](https://discourse.julialang.org/t/how-to-improve-a-generator-to-be-more-memory-efficient-when-it-is-collected/92932/22 "2023-01-16T18:24:10Z")

</div>

two observations:

1. the “+1” instead of “+3” I don’t think changes the times much, but it certainly doesn’t change the result. As I certainly can’t find an ATG startcodon following this one before position(A)+3;
2. the takewhile function is also in the IterTools module. Therefore, having avoided the use of drop, the IterTools package is enough.  
For times and allocations I did these tests…

```julia
julia> seq = randdnaseq(10^8)
100000000nt DNA Sequence:
AACATGAGCGTGGTTTTATTTGGGAGGACGATATTATTC…TCCCGCGGCAACTCCCGTACAATCAGAGTTGGCGTACCG

julia> using BenchmarkTools

julia> @time collect(locationiterator(seq))
  2.170984 seconds (16.12 M allocations: 842.458 MiB, 11.20% gc time, 7.71% compilation time)
1563050-element Vector{UnitRange{Int64}}:
 4:264
 45:152

julia> sseq=string(seq)
"AACATGAGCGTGGTTTTATTTGGGAGGACGATATTATTCCCGAGATGCGGTGTGAAAAAACAAATATTAAAACATTCAGCTTCTGTTGTACACAAGTCGCATCGCATTGTGGCCCGGAGGGAAAGGAGGTGTGCGTTGTGGATGATCAATAAAATCCGTCATCCACCCCGCCAGTTAAATTATATAGGCTCTCTCAGAGCACCTGACAGAGCACTCGGCGGTTGGATCGGGATGGCAGAT" ⋯ 99999520 bytes ⋯ "GTATCGGATCCAGGTCTAGTAGAGTCGTATGGACCATAGCGGCTCAGGTTAAAACCCAAAGGCTCTGGGGAAACATCGTGGTTTATCGGCAGTCGTCGACCAGGACAGTCAAGTGACTCTAAGTCCTAGTAGCTCAAACCGCGCGGATGCTGTAACGCTTTCTGGTTGGGCCACTGTAGAGACTCCGACACATTCCTCCCTTCCCGCGGCAACTCCCGTACAATCAGAGTTGGCGTACCG"

julia> ptrn=r"ATG(?:[AGTC][AGTC][AGTC])*?(TAG|TAA|TGA)"
r"ATG(?:[AGTC][AGTC][AGTC])*?(TAG|TAA|TGA)"

julia> f(R) = findnext(ptrn,sseq,first(R)+3)
f (generic function with 1 method)

julia> itr = takewhile(!isnothing, iterated(f, findfirst(ptrn,sseq)))   
IterTools.TakeWhile{IterTools.Iterated{UnitRange{Int64}, typeof(f)}}(Base.var"#97#98"{typeof(isnothing)}(isnothing), IterTools.Iterated{UnitRange{Int64}, typeof(f)}(f, 4:264))

julia> @btime [R for R in itr]
  949.594 ms (9378307 allocations: 324.01 MiB)
1563050-element Vector{UnitRange{Int64}}:
 4:264
 45:152
 142:264

julia> function se(sseq)
           eos=true
           R=-2:0
           v=UnitRange[]
           while eos
               R=f(R)
               (eos=!isnothing(R)) && push!(v,R)
           end
           v
       end
se (generic function with 1 method)

julia> @btime se(sseq)
  751.423 ms (4689159 allocations: 138.16 MiB)
1563050-element Vector{UnitRange}:
 4:264
 45:152
 142:264
 232:264

```

---

<div class="post-metadata">

### Author: ![camilogarciabotero](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/camilogarciabotero/32/35000_2.png) [@camilogarciabotero](https://discourse.julialang.org/u/camilogarciabotero)
#### Post date: [January 23, 2023, 6:26pm UTC](https://discourse.julialang.org/t/how-to-improve-a-generator-to-be-more-memory-efficient-when-it-is-collected/92932/23 "2023-01-23T18:26:18Z")

</div>

Thanks for answering this post. I used all your suggestions and especially @Dan implementation and reformulate it in terms of `BioSequences` types. I couldn’t make @rocco_sprmnt21 `se` function to work with `LonDNA` strings…

I ported these suggestions to [`GeneFinder.jl`](https://github.com/camilogarciabotero/GeneFinder.jl) package, particularly to the [`locationfinder`](https://github.com/camilogarciabotero/GeneFinder.jl/blob/36c64faad2636fa21e71e9a8e585a5812cd8e538/src/algorithms/findorfs.jl#L7) function.

I added you guys in the changelog with the Discourse profile because I couldn’t find your GH profiles but feel free to share it here or open a PR in the changelog.

Best.

[Previous page](https://discourse.julialang.org/t/how-to-improve-a-generator-to-be-more-memory-efficient-when-it-is-collected/92932.md?page=1)