Lazy.jl list overhead

General Usage Performance
1

I was experimenting with Lazy.jl and was wondering why there was significant overhead for accessing the first item of a list with a sufficiently complicated comprehension.

function testlist(n)
     l = @lazy [x^2 for x in 1:n if isodd(x)]
     first(drop(l, 1))
end

But running this for different values of n slows down significantly for higher values of n and eventually hits a stack overflow error.

julia> using BenchmarkTools
julia> @benchmark testlist(1)
  599.435 ns (17 allocations: 552 bytes)

julia> @btime testlist(100)
  4.529 μs (156 allocations: 17.09 KiB)
9
julia> @btime testlist(1000)
  151.500 μs (1507 allocations: 1.04 MiB)
9
julia> testlist(100000)
ERROR: StackOverflowError:

Since I am just calling first on the tail of the original list to get the second element, I thought the execution would stop early on. I found similar problems comparing equality of lists with one being very short, but the length of the longer list slowed down the runtime. Is this expected behavior for Lazy.jl and is there a way to do this without experiencing such a slowdown or error?

I know that this can already be achieved with generators, but they don’t behave the same way as lazy lists in other contexts. See generator example below:

function testgenerator(n)
       l = (x^2 for x in 1:n if isodd(x))
       safefirst(l) = isempty(l) ? [] : first(l)
       safefirst(drop(l, 1))
end

julia> @btime testgenerator(1)
  0.001 ns (0 allocations: 0 bytes)

julia> @btime testgenerator(100)
  13.828 ns (0 allocations: 0 bytes)
9
julia> @btime testgenerator(100000)
  13.627 ns (0 allocations: 0 bytes)
9

Edit
I also realized that with this alternative way of constructing the list, the code works without slowing down or stack overflows. I suppose this is the correct way to use Lazy.jl, but I’m still wondering why the other method fails, and if there is a way to use comprehensions without these problems.

function testlist2(n)
     l = @>> Lazy.range(1,n) map(x -> x^2) filter(isodd)
     first(tail(l))
end

julia> @btime testlist2(1)
  1.890 μs (35 allocations: 768 bytes)

julia> @btime testlist2(100)
  2.344 μs (64 allocations: 1.32 KiB)
9

julia> @btime testlist2(10000)
  2.389 μs (64 allocations: 1.32 KiB)
9

julia> @btime testlist2(1000000)
  2.389 μs (64 allocations: 1.32 KiB)
9