Question on lazy map function

Can anyone please point me to overview of using (maybe different kinds) of map/reduce/filter functions in Julia (and it’s standard packages)?

Here is is what I’m missing now: I want any(map(x -> x > 0, big_rand_array)) to lazy evaluate without computing on all elements, yet it appears to allocates big resulting array of map and only then passes it to any.

(this is “first steps” question)
Thank you very much,

1 Like

Hi there,

you may be interested in this thread Best Data Manipulation packages 2020-09 [video]

Lazy.jl can give you lazy evaluation: GitHub - MikeInnes/Lazy.jl: I was gonna maintain this package, but then I got high

And then there is Transducers.jl and the things that use it, not lazy AFAIK but pretty cool: GitHub - JuliaFolds/Transducers.jl: Efficient transducers for Julia

1 Like

One option is to use Iterators (Iteration utilities · The Julia Language)

a = Iterators.filter(x -> x > 0, big_rand_array)
b = Iterators.take(a, 1)
c = length(collect(b)) > 0

Will lazily iterate over your list and return true if any matches are found.

3 Likes

You can use a generator, like

any((x > 0 for x in -5:5)) # true
3 Likes

Many of these reduction functions can take a function as the first argument: any(>(0), randn(2)) ought to be efficient. There is a lazy map, but here you may as well combine them: any(f, Iterators.map(g, xs)) == any(f∘g, xs).

5 Likes

@ericphanson, why not single parentheses?

In Julia v1.5, Iterators.map does not appear to be lazy:

julia> Iterators.map(x -> 2x, Iterators.repeated(1, 3))
3-element Array{Int64,1}:
 2
 2
 2

Oh I forgot which version, again, sorry, it’s in 1.6 then. It is however identical to writing the generator:

julia> Iterators.map(sqrt, 1:10)
Base.Generator{UnitRange{Int64}, typeof(sqrt)}(sqrt, 1:10)

julia> (sqrt(x) for x in  1:10)
Base.Generator{UnitRange{Int64}, typeof(sqrt)}(sqrt, 1:10)
3 Likes

True, forgot you don’t need two sets there

2 Likes

Transducers are lazy.

julia> using Transducers

julia> Iterators.repeated(1, 3) |> Map(x -> 2x)
Base.Iterators.Take{Base.Iterators.Repeated{Int64}} |>
    Map(Main.λ❓)

julia> Iterators.repeated(1, 3) |> Map(x -> 2x) |> collect
3-element Vector{Int64}:
 2
 2
 2
3 Likes

FYI, there’s Compat.Iterators.map for older Julia.

Somewhat unfortunately, as of lowering: remove incorrect comprehension eta reduction by vtjnash · Pull Request #39139 · JuliaLang/julia · GitHub it’s not the case:

julia> (sqrt(x) for x in 1:10)
Base.Generator{UnitRange{Int64}, var"#1#2"}(var"#1#2"(), 1:10)

(but it’s required for correctness)

1 Like