# How to speed up the for-loop with dataframe access

**URL:** <https://discourse.julialang.org/t/how-to-speed-up-the-for-loop-with-dataframe-access/79447>\
**Category:** Performance\
**Tags:** dataframes\
**Created:** [April 13, 2022, 6:15pm UTC](https://discourse.julialang.org/t/how-to-speed-up-the-for-loop-with-dataframe-access/79447 "2022-04-13T18:15:13Z")\
**Posts on this page:** 6\
**Page:** 2

<div class="post-metadata">

**Author:** ![goerch](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/goerch/32/29122_2.png) [@goerch](https://discourse.julialang.org/u/goerch)\
**Post date:** [April 13, 2022, 10:47pm UTC](https://discourse.julialang.org/t/how-to-speed-up-the-for-loop-with-dataframe-access/79447/22 "2022-04-13T22:47:14Z")

</div>

> [@pdeffebach](#):
>
> As @bkamins referenced above, the type stabilities are causing all the problems.

Thanks for chiming in, @bkamins comments are way over my head. This looks very good, indeed. Is it sensible to talk about the generalized case (finding all duplicates) or should we leave it at this?

---

<div class="post-metadata">

**Author:** ![kfrb](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kfrb/32/29557_2.png) [@kfrb](https://discourse.julialang.org/u/kfrb)\
**Post date:** [April 13, 2022, 11:19pm UTC](https://discourse.julialang.org/t/how-to-speed-up-the-for-loop-with-dataframe-access/79447/23 "2022-04-13T23:19:42Z")

</div>

@goerch Your solution looks much better! I actually just wanted to point out that the calculation should be embedded in a function and didn’t do any benchmarks or tests.

@pdeffebach `DataFramesMeta.jl` looks very impressive! I will definitely take a closer look at the package!

---

<div class="post-metadata">

**Author:** ![goerch](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/goerch/32/29122_2.png) [@goerch](https://discourse.julialang.org/u/goerch)\
**Post date:** [April 13, 2022, 11:28pm UTC](https://discourse.julialang.org/t/how-to-speed-up-the-for-loop-with-dataframe-access/79447/24 "2022-04-13T23:28:58Z")

</div>

> [@kfrb](#):
>
> I actually just wanted to point out that the calculation should be embedded in a function and didn’t do any benchmarks or tests.

No problem here. My intention is to understand what can and can’t be done when transferring common SQL knowledge to `DataFrames` and for now I’ve failed to construct the necessary self join in this example (and I suspect that in a case like this we are on our own to construct indices).

---

<div class="post-metadata">

**Author:** ![qsong](https://avatars.discourse-cdn.com/v4/letter/q/d07c76/32.png) [@qsong](https://discourse.julialang.org/u/qsong)\
**Post date:** [April 14, 2022, 6:17am UTC](https://discourse.julialang.org/t/how-to-speed-up-the-for-loop-with-dataframe-access/79447/25 "2022-04-14T06:17:03Z")

</div>

To understand @bkamins brief comments above, I found [another one from him](https://discourse.julialang.org/t/allocation-free-type-hiding/66609/15), which provides the details.

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [April 14, 2022, 6:42am UTC](https://discourse.julialang.org/t/how-to-speed-up-the-for-loop-with-dataframe-access/79447/26 "2022-04-14T06:42:17Z")

</div>

Consider the following minimal example:

```julia
function f1(df)
    s = 0
    n = nrow(df)
    for i in 1:n
        for j in 1:n
            s += df.x[i] * df.y[j]
        end
    end
    return s
end

f2(df) = helper(df.x, df.y, nrow(df))

function helper(x, y, n)
    s = 0
    for i in 1:n
        for j in 1:n
            s += x[i] * y[j]
        end
    end
    return s
end

```

and now we benchmark it:

```julia
julia> using BenchmarkTools

julia> df = DataFrame(x=1:10^3, y=1:10^3);

julia> @btime f1($df)
  114.867 ms (3952700 allocations: 60.31 MiB)
250500250000

julia> @btime f2($df)
  490.200 μs (3 allocations: 48 bytes)
250500250000

```

And as @pdeffebach commented in DataFramesMeta.jl there are convenience macros that simplify this process.

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [April 14, 2022, 6:44am UTC](https://discourse.julialang.org/t/how-to-speed-up-the-for-loop-with-dataframe-access/79447/27 "2022-04-14T06:44:18Z")

</div>

Another option is:

```julia
julia> function f3(df)
           s = 0
           n = length(df.x)
               for i in 1:n
                   for j in 1:n
                       s += df.x[i] * df.y[j]
                   end
           end
           return s
       end;

julia> @btime f3(Tables.columntable($df))
  571.700 μs (9 allocations: 320 bytes)
250500250000

```

[Previous page](https://discourse.julialang.org/t/how-to-speed-up-the-for-loop-with-dataframe-access/79447.md?page=1)
