# Can this be made faster?

**URL:** <https://discourse.julialang.org/t/can-this-be-made-faster/78064>\
**Category:** Performance\
**Tags:** dataframes\
**Created:** [March 18, 2022, 2:42am UTC](https://discourse.julialang.org/t/can-this-be-made-faster/78064 "2022-03-18T02:42:07Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![r0b0ty](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/r0b0ty/32/27250_2.png) [@r0b0ty](https://discourse.julialang.org/u/r0b0ty)\
**Post date:** [March 18, 2022, 2:42am UTC](https://discourse.julialang.org/t/can-this-be-made-faster/78064/1 "2022-03-18T02:42:07Z")

</div>

I have two _DataFrames_. The size of the first is (7390, 2), represented below by a **small** (3, 2) _array_. The size of the second is quite a bit larger, at (482244, 8), represented below by a **larger** (7, 4) _array_.

I need to compare the values of **EACH** row in the **small** array with the values of **ALL** rows of the **large** array.

Obviously, it doesn’t take long in this example, but using my very large-sized _DataFrames_ (and comparing against 8 values in the large one), it takes about 10 minutes. Unfortunately, I have to do this over 24 times. Not knowing any better, I used a `for` loop, but wonder if there’s a better way. **Any suggestions?**

```julia
function inbound(x::Float64, y::Float64, bounds::Matrix{Float64})
    lastrow, _ = size(bounds)
    return any(x .>= bounds[1:lastrow, 1] .&&
               x .<= bounds[1:lastrow, 3] .&&
               y .>= bounds[1:lastrow, 2] .&&
               y .<= bounds[1:lastrow, 4])
end

```

```julia
julia> smallarray = [2.2 7.5
                     3.5 8.1
                     1.7 3.6];

```

```julia
julia> nrow_smallarray, _ = size(smallarray);

```

```julia
julia> largearray = [1.6 3.5 1.8 3.7
                     2.0 8.5 2.2 8.7
                     3.0 9.5 3.2 9.7
                     4.0 10.5 4.2 10.7
                     5.0 11.5 5.2 11.7
                     6.0 12.5 6.2 12.7
                     7.0 13.5 7.2 13.7];

```

```julia
julia> result = Vector{Bool}(undef, nrow_smallarray);

```

```julia
julia> for i in 1:nrow_smallarray
           result[i] = inbound(smallarray[i, 1], smallarray[i, 2], largearray)
       end

```

```julia
julia> result
3-element Vector{Bool}:
 0
 0
 1

```

---

<div class="post-metadata">

**Author:** ![CameronBieganek](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cameronbieganek/32/6915_2.png) [@CameronBieganek](https://discourse.julialang.org/u/CameronBieganek)\
**Post date:** [March 18, 2022, 4:05am UTC](https://discourse.julialang.org/t/can-this-be-made-faster/78064/2 "2022-03-18T04:05:40Z")

</div>

A couple things to notice:

- When you slice an array like `bounds[1:lastrow, 1]`, it actually makes a copy. So you are making a bunch of copies of large vectors when you run on your full data set.
- An expression like `any(a .≤ x .≤ b)` will broadcast the comparison operators over the full vector `x`, and _then_ apply the `any` function. It’s more efficient to use the `any(f, x)` form, because that form will short-circuit. In other words, it will return early as soon as the predicate `f` finds finds a `true` case.

You can address both of those issues with code like this:

```julia
function foo(smallarray, largearray)
    map(eachrow(smallarray)) do (x, y)
        any(eachrow(largearray)) do (a, c, b, d)
            a ≤ x ≤ b && c ≤ y ≤ d
        end
    end
end

```

Notes:

- I chose to use `map` because I find it more convenient than a `for` loop—you don’t have to preallocate an array. But the performance for `map` and a `for` loop should be similar.
- Notice the handy pattern matching / destructuring syntax in the `do` blocks that allows you to unpack an iterable object (in this case a row) into smaller named parts.
- `eachrow` is a non-allocating iterator over the rows of a matrix.

Here’s a benchmark:

```julia
julia> using BenchmarkTools

julia> A = rand(7390, 2); B = rand(482244, 4);

julia> @btime foo($A, $B);
  14.208 ms (1 allocation: 7.38 KiB)

```

(EDIT: This benchmark might not be the most representative of your problem, since for this random data set `any` will short-circuit quite quickly on each iteration of `map`.)

I would recommend using plain arrays for this if you can. It’s not as fast with a `DataFrame` because row iteration of `DataFrame`s is not very efficient:

```julia
julia> dfA = DataFrame(A, :auto); dfB = DataFrame(B, :auto);

julia> @btime foo($dfA, $dfB);
  2.013 s (67294279 allocations: 1.19 GiB)

```

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [March 18, 2022, 6:23am UTC](https://discourse.julialang.org/t/can-this-be-made-faster/78064/3 "2022-03-18T06:23:24Z")

</div>

or use a specialized function for data frames:

```julia
julia> function foo2(smalldf, largedf)
           map(Tables.namedtupleiterator(smalldf)) do x
               any(Tables.namedtupleiterator(largedf[!, 1:4])) do y
                   y[1] ≤ x[1] ≤ y[2] && y[3] ≤ x[2] ≤ y[4]
               end
           end
       end
foo2 (generic function with 1 method)

julia> @btime foo2($dfA, $dfB);
  23.745 ms (206935 allocations: 12.75 MiB)

```

In general, `Tables.namedtupleiterator` makes a loop type stable. As @CameronBieganek commented, by default `DataFrame` is optimized for column oriented operations.

---

<div class="post-metadata">

**Author:** ![r0b0ty](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/r0b0ty/32/27250_2.png) [@r0b0ty](https://discourse.julialang.org/u/r0b0ty)\
**Post date:** [March 18, 2022, 2:53pm UTC](https://discourse.julialang.org/t/can-this-be-made-faster/78064/4 "2022-03-18T14:53:12Z")

</div>

Thank you, gentlemen. I love learning new “tricks” to help improve my skills and can’t wait to try this out. My _DataFrames_ include other data types (Strings and Bools), so I could extract the Floats into arrays to do this comparison for quicker results. Thanks again for your input and have a great weekend.

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [March 18, 2022, 2:57pm UTC](https://discourse.julialang.org/t/can-this-be-made-faster/78064/5 "2022-03-18T14:57:45Z")

</div>

to get column names which are floats automatically write `names(df, AbstractFloat)`, e.g.:

```julia
julia> df = DataFrame(x=1:3, y=["a", "b", "c"], z1=1.0:3.0, z2=4.0:6.0)
3×4 DataFrame
 Row │ x y z1 z2
     │ Int64 String Float64 Float64
─────┼─────────────────────────────────
   1 │ 1 a 1.0 4.0
   2 │ 2 b 2.0 5.0
   3 │ 3 c 3.0 6.0

julia> names(df, AbstractFloat)
2-element Vector{String}:
 "z1"
 "z2"

```

---

<div class="post-metadata">

**Author:** ![r0b0ty](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/r0b0ty/32/27250_2.png) [@r0b0ty](https://discourse.julialang.org/u/r0b0ty)\
**Post date:** [March 19, 2022, 1:00am UTC](https://discourse.julialang.org/t/can-this-be-made-faster/78064/6 "2022-03-19T01:00:39Z")

</div>

FYI, when using the approach offered by @CameronBieganek, the processing time was incredibly reduced when applying it to actual data. A single run went from 10 minutes to **~12 seconds!**

```julia
julia> size(z1_0_matrix)
(7390, 2)

julia> size(nok_20m_matrix)
(482244, 4)

julia> @btime result = inbound(z1_0_matrix, nok_20m_matrix)
  12.636 s (1 allocation: 7.38 KiB)
7390-element Vector{Bool}:
 0
 0
 0
 0
 0
 0
 0
 0
 0
 0
 0
 ⋮
 0
 0
 0
 0
 0
 0
 0
 0
 0
 0
 0

```

There are, in fact, expected `true` results (not just zeros).

```julia
julia> count(>(0), result)
497

```

For completeness (and curiosity’s sake), I’ll also do it as @bkamins suggested (knowing that we expect it to be somewhat slower).

Thanks, again.