# How to leverage internals of \`collect\`

**URL:** <https://discourse.julialang.org/t/how-to-leverage-internals-of-collect/7493>\
**Category:** General Usage\
**Tags:** question\
**Created:** [December 4, 2017, 10:18am UTC](https://discourse.julialang.org/t/how-to-leverage-internals-of-collect/7493 "2017-12-04T10:18:50Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [December 4, 2017, 10:18am UTC](https://discourse.julialang.org/t/how-to-leverage-internals-of-collect/7493/1 "2017-12-04T10:18:50Z")

</div>

As part of on API I am working on, one of the higher-level functions collects results from an iterative algorithm. The results use user-defined functions, which are ideally type stable, but don’t have to be. Code could look like

```julia
"""
    get_result(state)

Return the result from state. Defined by the user.
"""
function get_result end

"""
    update_state(rng, state)

Return a new state updated randomly. Uses user-defined components.
"""
function update_state(rng, state) end

function collect_results(state, N, rng = Base.Random.GLOBAL_RNG)
    results = FIXME_initialize(...)
    for _ in 1:N
        FIXME_add!(results, get_result(state))
        state = update_state(rng, state)
    end
    results
end

```

Lines with `FIXME` is where I need help.

I could somehow cram this into a form for `collect` with closures, but I would prefer to keep the loop. Looking at the sources, the internals of `collect` have mechanisms for initializing a container with a type, then dealing with the possibility of getting something else instead. Is there a chance of using those? An example would help.

Ideally, my hope is that the compiler can sometimes figure out types for `get_result`, then it would be nice to get efficient code, but that’s a best case scenario I would not want to rely on, and want the right container type even if that does not work.

---

<div class="post-metadata">

**Author:** ![tkoolen](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkoolen/32/1603_2.png) [@tkoolen](https://discourse.julialang.org/u/tkoolen)\
**Post date:** [December 4, 2017, 1:02pm UTC](https://discourse.julialang.org/t/how-to-leverage-internals-of-collect/7493/2 "2017-12-04T13:02:09Z")

</div>

> [@Tamas\_Papp](#):
>
> I could somehow cram this into a form for collect with closures, but I would prefer to keep the loop.

Could you provide more info on why you prefer the loop?

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [December 4, 2017, 2:28pm UTC](https://discourse.julialang.org/t/how-to-leverage-internals-of-collect/7493/3 "2017-12-04T14:28:11Z")

</div>

I prefer a loop because I have a state, and a somewhat complicated update mechanism for it (naturally, not shown in the MWE). As I said, I could shoehorn it into a `collect` with a closure, but that would not be the natural idiom. Loops are natural for many problems in Julia, now I just want to `collect` the result.

---

<div class="post-metadata">

**Author:** ![tkoolen](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkoolen/32/1603_2.png) [@tkoolen](https://discourse.julialang.org/u/tkoolen)\
**Post date:** [December 4, 2017, 3:38pm UTC](https://discourse.julialang.org/t/how-to-leverage-internals-of-collect/7493/4 "2017-12-04T15:38:32Z")

</div>

I suppose whether or not something is more natural is a bit subjective. I would argue that digging into the internals of `collect` is perhaps less natural.

How about the `map` `do` syntax? See [`stevengj`’s suggestion for a similar problem](https://discourse.julialang.org/t/ridiculous-idea-types-from-the-future/5457/2).

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [December 4, 2017, 3:53pm UTC](https://discourse.julialang.org/t/how-to-leverage-internals-of-collect/7493/5 "2017-12-04T15:53:28Z")

</div>

Thanks, but my primary intention here is not to debate whether this is “natural”, but access this mechanism. It is of course possible that the interface is yet not ready for use outside `Base`, in which case I will just open an issue after thinking more about this.

---

<div class="post-metadata">

**Author:** ![rdeits](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rdeits/32/286_2.png) [@rdeits](https://discourse.julialang.org/u/rdeits)\
**Post date:** [December 4, 2017, 4:06pm UTC](https://discourse.julialang.org/t/how-to-leverage-internals-of-collect/7493/6 "2017-12-04T16:06:31Z")

</div>

As suggested by `stevegj` in that thread linked above, I’ve found the `map-do` construction pretty useful for this kind of situation. In your case, that might look like:

```julia
function collect_results(state, N, rng = Base.Random.GLOBAL_RNG)
    map(1:N) do _
        get_result(update_state(rng, state))
    end
end

```

which keeps the loop structure you’re looking for while also letting you simple omit both `FIXME`s.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [December 4, 2017, 4:11pm UTC](https://discourse.julialang.org/t/how-to-leverage-internals-of-collect/7493/7 "2017-12-04T16:11:41Z")

</div>

Thanks. Thinking about this, perhaps the only thing I cannot do with a `map` is `break` or `return` early. In this particular case I don’t need it though.

---

<div class="post-metadata">

**Author:** ![rdeits](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rdeits/32/286_2.png) [@rdeits](https://discourse.julialang.org/u/rdeits)\
**Post date:** [December 4, 2017, 4:21pm UTC](https://discourse.julialang.org/t/how-to-leverage-internals-of-collect/7493/8 "2017-12-04T16:21:07Z")

</div>

Ah, yes, I agree, and I have wanted to be able to break inside a `map` as well.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [December 6, 2017, 10:00am UTC](https://discourse.julialang.org/t/how-to-leverage-internals-of-collect/7493/9 "2017-12-06T10:00:42Z")

</div>

I looked at `Base.collect` more carefully and came to the conclusion that what I want cannot be done in a way that fits well with the features that make Julia performant.

Specifically, [`Base.collect_to!`](https://github.com/JuliaLang/julia/blob/beb752629afe18539845826b61ac8b3859cba1d8/base/array.jl#L553-L572) is how one of the “worst case” scenarios is handled: when it finds that it can’t store an element because of type restrictions, it widens, copies, and calls itself recursively.

In order to achieve something similar in a loop with a mutable structure, I can imagine a

```julia
mutable struct Container
    elements::Vector # note the unspecified element type
end

```

and a variant of `push!` that tries pushing to `.elements`, and widens and copies if necessary. This way we keep the same `Container`, but its `.element` [does not have a concrete type](https://docs.julialang.org/en/latest/manual/performance-tips/#Avoid-fields-with-abstract-containers-1).

Or alternatively, I can envision a

```julia
container = store(container, new_element)

```

which may or may not return its first argument, if widening and copying if necessary. This runs into [changing variable types](https://docs.julialang.org/en/latest/manual/performance-tips/#Avoid-changing-the-type-of-a-variable-1), again problematic.

Anyhow, this is not a critique of Julia, just my notes on what I understood in connection with this question. I just have to organize my code differently.