# Threaded for loop over arbitrary indexable objects

**URL:** <https://discourse.julialang.org/t/threaded-for-loop-over-arbitrary-indexable-objects/26957>\
**Category:** Internals & Design\
**Tags:** question\
**Created:** [July 29, 2019, 9:24pm UTC](https://discourse.julialang.org/t/threaded-for-loop-over-arbitrary-indexable-objects/26957 "2019-07-29T21:24:15Z")\
**Posts on this page:** 17\
**Page:** 1

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [July 29, 2019, 9:24pm UTC](https://discourse.julialang.org/t/threaded-for-loop-over-arbitrary-indexable-objects/26957/1 "2019-07-29T21:24:15Z")

</div>

Currently the following code fails:

```julia
julia> Threads.@threads for c in "ssf"
       end

Error thrown in threaded loop on thread 0: MethodError(f=typeof(Base.unsafe_getindex)(), args=("ssf", 1), world=0x000000000000643a)

```

(the original problem was posted in DataFrames.jl [here](https://github.com/JuliaData/DataFrames.jl/issues/1905))

The reason is that `Base.unsafe_getindex` is used internally and it is defined only for a limited number of collection types.

With Julia 1.3 I expect that threads will be used more often.

So my question is if it is intentional and required to be restricted or could be allowed for a wider range of types that support `getindex` (as I suppose it could)?

EDIT: the example given above is probably not best, as strings are not indexable with step 1, but see the linked DataFrames.jl example where `GroupedDataFrame` is properly indexable with 1-increment, but on purpose we do not make it a subtype of `AbstractArray`.

EDIT2: probably the simplest example is with a `Tuple`:

```julia
julia> @Threads.threads for i in (1,2,3)
       end

Error thrown in threaded loop on thread 0: MethodError(f=typeof(Base.unsafe_getindex)(), args=((1, 2, 3), 1), world=0x00000000000063fd)

```

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [September 9, 2019, 11:51am UTC](https://discourse.julialang.org/t/threaded-for-loop-over-arbitrary-indexable-objects/26957/2 "2019-09-09T11:51:39Z")

</div>

Why not use (loosely)

```julia
@sync begin
    for i in (1,2,3)
        Threads.@spawn work(i)
    end
end

```

and let the dynamic scheduler do its job?

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [September 9, 2019, 12:07pm UTC](https://discourse.julialang.org/t/threaded-for-loop-over-arbitrary-indexable-objects/26957/3 "2019-09-09T12:07:13Z")

</div>

This is only possible in Julia 1.3 and, e.g. in DataFrames.jl we try to support Julia 1.0 LTS.

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [September 9, 2019, 12:11pm UTC](https://discourse.julialang.org/t/threaded-for-loop-over-arbitrary-indexable-objects/26957/4 "2019-09-09T12:11:00Z")

</div>

Just run things serially in 1.0?

---

<div class="post-metadata">

**Author:** ![nalimilan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nalimilan/32/147_2.png) [@nalimilan](https://discourse.julialang.org/u/nalimilan)\
**Post date:** [September 9, 2019, 12:11pm UTC](https://discourse.julialang.org/t/threaded-for-loop-over-arbitrary-indexable-objects/26957/5 "2019-09-09T12:11:18Z")

</div>

I guess Compat could define some no-op macros to allow that. But AFAICT the issue was that users wanted to write that, and it failed. Should it?

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [September 9, 2019, 12:15pm UTC](https://discourse.julialang.org/t/threaded-for-loop-over-arbitrary-indexable-objects/26957/6 "2019-09-09T12:15:32Z")

</div>

> [@bkamins](#):
>
> in DataFrames.jl we try to support Julia 1.0 LTS.

> [@nalimilan](#):
>
> But AFAICT the issue was that users wanted to write that, and it failed.

I’m a bit confused if this topic is about doing threading inside DataFrames.jl or just a general question about users doing threads on arbitrary objects.

---

<div class="post-metadata">

**Author:** ![nalimilan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nalimilan/32/147_2.png) [@nalimilan](https://discourse.julialang.org/u/nalimilan)\
**Post date:** [September 9, 2019, 12:16pm UTC](https://discourse.julialang.org/t/threaded-for-loop-over-arbitrary-indexable-objects/26957/7 "2019-09-09T12:16:57Z")

</div>

It’s not about writing threaded code inside DataFrames.jl. It’s about whether/how things like this should work for users:

```julia
  Threads.@threads for sdf ∈ groupby(df, :grpcol)
    sdf.x3 .= -1.
  end

```

But the example with a tuple is simpler:

```julia
@Threads.threads for i in (1,2,3)
end

```

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [September 9, 2019, 12:20pm UTC](https://discourse.julialang.org/t/threaded-for-loop-over-arbitrary-indexable-objects/26957/8 "2019-09-09T12:20:40Z")

</div>

I used `String` as an example, to avoid going into the details of design and intended usage of `GroupedDataFrame` object in DataFrames.jl (as the core of the issue in the given question is the same).

Essentially we are investigating how we can speed up split-apply-combine ecosystem in DataFrames.jl using threading.

---

<div class="post-metadata">

**Author:** ![nalimilan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nalimilan/32/147_2.png) [@nalimilan](https://discourse.julialang.org/u/nalimilan)\
**Post date:** [September 9, 2019, 12:35pm UTC](https://discourse.julialang.org/t/threaded-for-loop-over-arbitrary-indexable-objects/26957/9 "2019-09-09T12:35:29Z")

</div>

> [@bkamins](#):
>
> Essentially we are investigating how we can speed up split-apply-combine ecosystem in DataFrames.jl using threading.

Just to clarify: this will require internal changes in DataFrames which are well beyond the scope of this issue of course.

---

<div class="post-metadata">

**Author:** ![nilshg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nilshg/32/2283_2.png) [@nilshg](https://discourse.julialang.org/u/nilshg)\
**Post date:** [September 10, 2019, 8:35am UTC](https://discourse.julialang.org/t/threaded-for-loop-over-arbitrary-indexable-objects/26957/10 "2019-09-10T08:35:30Z")

</div>

This actually bit me yesterday, when I tried doing (you might remember the discussion on Slack) something like:

```julia
@Threads.threads for (n, r) in enumerate(eachrow(df))
   ...
end

```

So I’m also interested in the broader question of which iterators are expected to received threads support? I naively expected this to work out of the box for everything and couldn’t quite work out what the `unsafe_getindex` error message was telling me.

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [September 10, 2019, 11:56am UTC](https://discourse.julialang.org/t/threaded-for-loop-over-arbitrary-indexable-objects/26957/11 "2019-09-10T11:56:28Z")

</div>

`Threads.@threads` statically schedule​ the work so it needs random access into what is looped over. It could work probably be made to work for tuples but for simplicity, I have always just looped over a vector of int and then indexed into whatever I need based on that.

---

<div class="post-metadata">

**Author:** ![nilshg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nilshg/32/2283_2.png) [@nilshg](https://discourse.julialang.org/u/nilshg)\
**Post date:** [September 10, 2019, 12:27pm UTC](https://discourse.julialang.org/t/threaded-for-loop-over-arbitrary-indexable-objects/26957/12 "2019-09-10T12:27:36Z")

</div>

Is this just a case of missing methods for iterables other than an integer range or is there a more fundamental issue that prevents this from working?

To me the ability to write for loops over general iterables without having to do integer indexing is one of the great Julia features that allows for clean and readable code, and it’d be great if throwing in `@Threads.threads` after the fact would _justwork_™

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [September 10, 2019, 12:32pm UTC](https://discourse.julialang.org/t/threaded-for-loop-over-arbitrary-indexable-objects/26957/13 "2019-09-10T12:32:08Z")

</div>

For `@threads` it is fundamental in that it schedules all the work on threads statically.

If you want to do dynamic scheduling then use the `Threads.@spawn` functionality in 1.3+. Is there an issue with that?

---

<div class="post-metadata">

**Author:** ![nilshg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nilshg/32/2283_2.png) [@nilshg](https://discourse.julialang.org/u/nilshg)\
**Post date:** [September 10, 2019, 12:46pm UTC](https://discourse.julialang.org/t/threaded-for-loop-over-arbitrary-indexable-objects/26957/14 "2019-09-10T12:46:15Z")

</div>

Sorry I feel like I should be doing a bit more reading on how the new threading works rather than wasting your time with basic questions - I haven’t tried the `@spawn` route, so far all my parallel code uses `SharedArrays` and then `@sync @distributed for` loops, and my mental model was that `@threads` replaces this and allows me to work with “normal” Arrays given it’s shared memory parallelism.

I’m still not sure what is so bad about random acces for general iterators, in my use case yesterday I was iterating through DataFrame rows, and for my taks it didn’t matter in which order the rows were processed. I might be missing something fundamental about what random access and static scheduling mean…

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [September 10, 2019, 12:56pm UTC](https://discourse.julialang.org/t/threaded-for-loop-over-arbitrary-indexable-objects/26957/15 "2019-09-10T12:56:30Z")

</div>

@kristoffer.carlsson - for me the core of the question is why `@threads` requires an iterable to support `Base.unsafe_getindex` and not just `getindex` and `length` (which already guarantees that the work can be statically scheduled).

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [September 10, 2019, 1:04pm UTC](https://discourse.julialang.org/t/threaded-for-loop-over-arbitrary-indexable-objects/26957/16 "2019-09-10T13:04:30Z")

</div>

> [@bkamins](#):
>
> requires an iterable to support `Base.unsafe_getindex` and not just `getindex` and `length` (which already guarantees that the work can be statically scheduled).

Yeah, I realize this and I don’t think there is any reason. It was added a long time ago in [Remove @threads for calls and blocks, since these require · JuliaLang/julia@a312aad · GitHub](https://github.com/JuliaLang/julia/commit/a312aad47095ff1601d4a27afe5391184218e78d) with the description:

> Generalize `@threads` to work on any 1D range with random-access subscripting. Long term, we can generalize it to more general ranges similar to how `@simd` works

Previously, I think it only worked on unit ranges.

---

<div class="post-metadata">

**Author:** ![tkf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkf/32/17635_2.png) [@tkf](https://discourse.julialang.org/u/tkf)\
**Post date:** [September 10, 2019, 7:28pm UTC](https://discourse.julialang.org/t/threaded-for-loop-over-arbitrary-indexable-objects/26957/17 "2019-09-10T19:28:56Z")

</div>

Just FYI [`reduce`](https://tkf.github.io/Transducers.jl/dev/manual/#Base.reduce) defined in Transducers.jl can be used to do something like `@threads for` for not only arbitrary indexable collections but also any collections that can [`halve`](https://github.com/tkf/Transducers.jl/blob/c6dbc4ebdeafc06d9574f4e5f854527f2e36a61c/src/reduce.jl#L43-L49) itself (…in principle; ATM only `AbstractArray` is supported). Roughly speaking,

```julia
Threads.@threads for x in xs
    use(x)
end

```

is equivalent to

```julia
reduce(Map(identity), xs; init=nothing) do _, x
    use(x)
end

```

It also support (deterministic) [“`break`”](https://tkf.github.io/Transducers.jl/dev/manual/#Transducers.reduced) which cannot be done with `@threads for` AFAICT.