# Fast iteration over rows of a DataFrame

**URL:** <https://discourse.julialang.org/t/fast-iteration-over-rows-of-a-dataframe/24612>\
**Category:** Performance\
**Created:** [May 26, 2019, 1:26am UTC](https://discourse.julialang.org/t/fast-iteration-over-rows-of-a-dataframe/24612 "2019-05-26T01:26:32Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![robsmith11](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/robsmith11/32/29641_2.png) [@robsmith11](https://discourse.julialang.org/u/robsmith11)\
**Post date:** [May 26, 2019, 1:26am UTC](https://discourse.julialang.org/t/fast-iteration-over-rows-of-a-dataframe/24612/1 "2019-05-26T01:26:32Z")

</div>

I want to read a CSV file and do some path-dependent calculations involving multiple columns (i.e., no vectorization allowed in the actual problem.)

I was surprised to see that in a simple example, DataFrames introduce over 100x overhead. Is there a faster way to iterate over rows? Or alternatively, a way to parse a CSV directly into a NamedTuple?

```julia
julia> d = [(a=rand(),b=rand()) for _ in 1:10^6];

julia> df = DataFrame(d);

julia> function f(xs)
        s = 0.0;
        for x in xs
         s += x.a * x.b
        end
        s
       end
f (generic function with 1 method)

julia> function g(xs)
        s = 0.0
        for x in eachrow(xs)
         s += x.a * x.b
        end
        s
       end
g (generic function with 1 method)

julia> @btime f($d)
  577.269 μs (0 allocations: 0 bytes)
249855.20496448214

julia> @btime g($df)
  105.782 ms (6998979 allocations: 122.05 MiB)
249855.20496448386

```

---

<div class="post-metadata">

**Author:** ![Elrod](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/elrod/32/22461_2.png) [@Elrod](https://discourse.julialang.org/u/Elrod)\
**Post date:** [May 26, 2019, 3:26am UTC](https://discourse.julialang.org/t/fast-iteration-over-rows-of-a-dataframe/24612/2 "2019-05-26T03:26:49Z")

</div>

If all the entries can be promoted to the same type, you may get better performance with [readdlm](https://docs.julialang.org/en/v1/stdlib/DelimitedFiles/index.html#DelimitedFiles.readdlm-Tuple%7BAny,AbstractChar,Type,AbstractChar%7D), which returns a matrix.

---

<div class="post-metadata">

**Author:** ![bernhard](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bernhard/32/2619_2.png) [@bernhard](https://discourse.julialang.org/u/bernhard)\
**Post date:** [May 26, 2019, 4:00pm UTC](https://discourse.julialang.org/t/fast-iteration-over-rows-of-a-dataframe/24612/3 "2019-05-26T16:00:37Z")

</div>

the slow down is indeed surprising.  
Seems like you should just use a Matrix instead of a DataFrame (if your use case allows this)

the function below is about 38% faster than yours, but still way slower than the matrix

```julia
function u(xs)
        s = 0.0;
        @inbounds for i=1:size(xs,1)
            s+=xs[i,:a]*xs[i,:b]
        end
        s
       end

```

---

<div class="post-metadata">

**Author:** ![robsmith11](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/robsmith11/32/29641_2.png) [@robsmith11](https://discourse.julialang.org/u/robsmith11)\
**Post date:** [May 26, 2019, 4:34pm UTC](https://discourse.julialang.org/t/fast-iteration-over-rows-of-a-dataframe/24612/4 "2019-05-26T16:34:43Z")

</div>

My actual use case does involve multiple types of columns.

I’ve opened an issue on the DataFrames.jl github to see if they have any ideas:  
[https://github.com/JuliaData/DataFrames.jl/issues/1827](https://github.com/JuliaData/DataFrames.jl/issues/1827)

EDIT: The reason is that iteration over the rows of a DataFrame is type-unstable, hence the slow down. I’m not a big fan of Julia’s DataFrame API anyway, so I’ll just stick with NamedTuples.

---

<div class="post-metadata">

**Author:** ![bernhard](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bernhard/32/2619_2.png) [@bernhard](https://discourse.julialang.org/u/bernhard)\
**Post date:** [May 26, 2019, 7:19pm UTC](https://discourse.julialang.org/t/fast-iteration-over-rows-of-a-dataframe/24612/5 "2019-05-26T19:19:10Z")

</div>

Ok. If your use case is time critical, you may want to work with several vectors instead of a dataframe. Also encoding the character data in Ints or similar might speed up the process

---

<div class="post-metadata">

**Author:** ![robsmith11](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/robsmith11/32/29641_2.png) [@robsmith11](https://discourse.julialang.org/u/robsmith11)\
**Post date:** [December 22, 2019, 5:23am UTC](https://discourse.julialang.org/t/fast-iteration-over-rows-of-a-dataframe/24612/6 "2019-12-22T05:23:10Z")

</div>

For anyone else googling this, the solution is to use [IndexedTables.jl](https://github.com/JuliaComputing/IndexedTables.jl).

Unlike DataFrames.jl, it is type-stable when iterating over rows, so the performance is just as fast as working with raw vectors.

I’m surprised IndexedTables.jl isn’t more popular. It seems to have most of the same features without the performance pitfalls.

---

<div class="post-metadata">

**Author:** ![bashonubuntu](https://avatars.discourse-cdn.com/v4/letter/b/f19dbf/32.png) [@bashonubuntu](https://discourse.julialang.org/u/bashonubuntu)\
**Post date:** [December 22, 2019, 6:00am UTC](https://discourse.julialang.org/t/fast-iteration-over-rows-of-a-dataframe/24612/7 "2019-12-22T06:00:50Z")

</div>

Also look at [https://github.com/piever/JuliaDBMeta.jl](https://github.com/piever/JuliaDBMeta.jl)

---

<div class="post-metadata">

**Author:** ![nalimilan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nalimilan/32/147_2.png) [@nalimilan](https://discourse.julialang.org/u/nalimilan)\
**Post date:** [December 26, 2019, 9:43pm UTC](https://discourse.julialang.org/t/fast-iteration-over-rows-of-a-dataframe/24612/8 "2019-12-26T21:43:18Z")

</div>

For reference, you don’t even need to use IndexedTables. You can just pass a type-stable iterators to a function. In the OP, call `g(Tables.columntable(df))` instead of `g(df)` to pass `g` a named tuple of vectors, and then replace `eachrow(xs)` with `Tables.rows(xs)`.

(IndexedTables is great if you have few columns, but with hundreds of columns of different types it’s going to stress the compiler.)

---

<div class="post-metadata">

**Author:** ![msekino](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/msekino/32/12889_2.png) [@msekino](https://discourse.julialang.org/u/msekino)\
**Post date:** [February 16, 2020, 2:01pm UTC](https://discourse.julialang.org/t/fast-iteration-over-rows-of-a-dataframe/24612/9 "2020-02-16T14:01:59Z")

</div>

Thank you for your useful information!  
Finally, I could iterate over rows using:

- tbl = Tables.rowtable(df)
- for row in Tables.rows(tbl)

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [February 16, 2020, 2:57pm UTC](https://discourse.julialang.org/t/fast-iteration-over-rows-of-a-dataframe/24612/10 "2020-02-16T14:57:47Z")

</div>

Actually the recommended way after DataFrames 0.21 will be released (or on current #master) is:

```julia
for row in Tables.namedtupleiterator(df)

```

if you need performance but you are willing to pay the cost of compilation.

If your computation is small and you want to avoid compilation cost (which for very wide tables can be significant) use what you have indicated above:

```julia
for row in eachrow(df)

```

---

<div class="post-metadata">

**Author:** ![matthieu](https://avatars.discourse-cdn.com/v4/letter/m/da6949/32.png) [@matthieu](https://discourse.julialang.org/u/matthieu)\
**Post date:** [February 16, 2020, 3:23pm UTC](https://discourse.julialang.org/t/fast-iteration-over-rows-of-a-dataframe/24612/11 "2020-02-16T15:23:31Z")

</div>

If you want performance, you cannot just use `for row in Tables.namedtupleiterator(df)`, right? You would still need to pass the rows iterator as a function argument:

```julia
function g(rows)
   s = 0.0
   for row in rows
      s += row.a * row.b
   end
   s
 end
# compile once
g(eachrow(df))
# faster but recompiles for each dataframe
g(Tables.namedtupleiterator(df))

```

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [February 16, 2020, 4:20pm UTC](https://discourse.julialang.org/t/fast-iteration-over-rows-of-a-dataframe/24612/12 "2020-02-16T16:20:25Z")

</div>

Yes - I was too brief. Thank you for correcting. You need a barrier function as you have indicated.

In this specific case one could also write the following to get a barrier:

```julia
mapreduce(row -> row.a+row.b, +, Tables.namedtupleiterator(df), init=0.0)

```

---

<div class="post-metadata">

**Author:** ![aaowens](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aaowens/32/12101_2.png) [@aaowens](https://discourse.julialang.org/u/aaowens)\
**Post date:** [February 16, 2020, 4:37pm UTC](https://discourse.julialang.org/t/fast-iteration-over-rows-of-a-dataframe/24612/13 "2020-02-16T16:37:37Z")

</div>

To get around the long compilation times, can we subset the DataFrame to only fetch the columns we want?, ie, `Tables.namedtupleiterator(df[!, [:a, :b]])`

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [February 16, 2020, 5:05pm UTC](https://discourse.julialang.org/t/fast-iteration-over-rows-of-a-dataframe/24612/14 "2020-02-16T17:05:16Z")

</div>

Sure - but I did not want to complicate the code with another change. Actually the fastest way would probably be just:

```julia
Tables.rows((a=df.a, b=df.b))

```

---

<div class="post-metadata">

**Author:** ![Marc.Cox](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/marc.cox/32/7514_2.png) [@Marc.Cox](https://discourse.julialang.org/u/Marc.Cox)\
**Post date:** [June 30, 2020, 3:20pm UTC](https://discourse.julialang.org/t/fast-iteration-over-rows-of-a-dataframe/24612/15 "2020-06-30T15:20:56Z")

</div>

Hi Rob @robsmith11 and all on this thread - might you have an insight or new approach here ?

So I took up the solution of using `IndexedTables.jl` for _Fast Iterations over rows of a Dataframe_  
here. And then I proceeded to attempt to use IndexedTables for multidimensional 2D,3D (scatter) Plots  
using IndexedTables to graph N-Dimensional data; but ran into issues trying to  
collect the iterable `for iter in eachindex(keys(tab_t1.index.columns))`  
because `tab_t1.index.columns` returns  
`ERROR: LoadError: type IndexedTable has no field index`  
as you’ll see when you run the Julia pseudocode listed here

> [@Looking for ways to generalize using IndexedTables to Plot / Graph N-Dimensional data](https://discourse.julialang.org/t/looking-for-ways-to-generalize-using-indexedtables-to-plot-graph-n-dimensional-data/42266):
>
> Hi All, I’m looking for ways to generalize using IndexedTables to graph N-Dimensional data, but I’m stuck on how to collect the “for iter in eachindex(keys(tab\_t1.index.columns))”. In that regard, would you please see MWE pseudocode snippet below and let me know any suggestions you might have ? using IndexedTables using Plots # 2D = 1Dy Dependent var equation of 1Dx INDependent var tab\_t1 = table( (x=1:5, y=randn(5)); pkey = [:x] ) str\_current\_equation\_approximated = "(x=1:5, y=randn(5))" …

Any new insights or approaches to getting the keys from `tab_t1.index.columns` ;  
-or- another way to automatically graph general multidimensional 2D,3D,(? and 4D like [www.wolframalpha.com](http://www.wolframalpha.com) ?) scatter Plots is appreciated.

TY  
-Marc