# Help speed up SimpleGraph creation

**URL:** <https://discourse.julialang.org/t/help-speed-up-simplegraph-creation/48754>\
**Category:** Graphs\
**Tags:** lightgraphs\
**Created:** [October 21, 2020, 3:12pm UTC](https://discourse.julialang.org/t/help-speed-up-simplegraph-creation/48754 "2020-10-21T15:12:10Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![mthelm85](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mthelm85/32/224164_2.png) [@mthelm85](https://discourse.julialang.org/u/mthelm85)\
**Post date:** [October 21, 2020, 3:12pm UTC](https://discourse.julialang.org/t/help-speed-up-simplegraph-creation/48754/1 "2020-10-21T15:12:10Z")

</div>

Here’s a MWE of what I’m attempting to do:

```julia
using DataFrames
using LightGraphs
using ProgressMeter

df = DataFrame(a = rand(1:50_000, 150_000), b = rand(1:25_000, 150_000)) |> unique!
vs = unique(vcat(df.a, df.b))
n = length(vs)
g = SimpleGraph(n)

p = Progress(n)
Threads.@threads for v in vertices(g)
    connected_vs = filter(row -> row.a == vs[v], df).b
    for cv in connected_vs
        add_edge!(g, v, findfirst(x -> x == cv, vs))
    end
    next!(p)
end

```

This example will take 10 - 15 minutes to complete on my machine. My real problem is larger and ETA is about 50 minutes. Surely there’s a more performant way?? 🤞

EDIT: It looks like the `filter` function is a primary culprit. Changing to this yields much better results:

```julia
df = DataFrame(a = rand(1:5_000, 15_000), b = rand(1:2_500, 15_000)) |> unique!
vs = unique(vcat(df.a, df.b))
n = length(vs)
g = SimpleGraph(n)

Threads.@threads for v in vertices(g)
    for cv in df[df.a .== vs[v], :b]
        add_edge!(g, v, findfirst(x -> x == cv, vs))
    end
end

```

---

<div class="post-metadata">

**Author:** ![lmiq](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lmiq/32/18314_2.png) [@lmiq](https://discourse.julialang.org/u/lmiq)\
**Post date:** [October 21, 2020, 3:54pm UTC](https://discourse.julialang.org/t/help-speed-up-simplegraph-creation/48754/2 "2020-10-21T15:54:57Z")

</div>

> [@mthelm85](#):
>
> `filter` function

Indeed. If the data was ordered relative to the `a` fields, you could use just a slice instead, and that would be much faster.

---

<div class="post-metadata">

**Author:** ![mthelm85](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mthelm85/32/224164_2.png) [@mthelm85](https://discourse.julialang.org/u/mthelm85)\
**Post date:** [October 21, 2020, 3:59pm UTC](https://discourse.julialang.org/t/help-speed-up-simplegraph-creation/48754/3 "2020-10-21T15:59:09Z")

</div>

> [@lmiq](#):
>
> If the data was ordered relative to the `a` fields, you could use just a slice instead

I don’t see a `Base.slice`, is there a package that exports a `slice` function?

---

<div class="post-metadata">

**Author:** ![lmiq](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lmiq/32/18314_2.png) [@lmiq](https://discourse.julialang.org/u/lmiq)\
**Post date:** [October 21, 2020, 5:54pm UTC](https://discourse.julialang.org/t/help-speed-up-simplegraph-creation/48754/4 "2020-10-21T17:54:04Z")

</div>

What I meant was that you could use something like

```julia
i = 1 
for v in vertices(g)
     j = findlast(x->x==v,df.a)
     if j != nothing
         cv = df.b[i:j]
         i = j + 1
     end
end

```

This appears to be about ~3 times faster than using `cv = df[df.a .== vs[1], :b]`, but I noticed  
that this alternative appears to be fast enough, isn’t it?

(I didn’t test the code to guarantee that it exactly returns the same thing you want).

---

<div class="post-metadata">

**Author:** ![mthelm85](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mthelm85/32/224164_2.png) [@mthelm85](https://discourse.julialang.org/u/mthelm85)\
**Post date:** [October 21, 2020, 5:58pm UTC](https://discourse.julialang.org/t/help-speed-up-simplegraph-creation/48754/5 "2020-10-21T17:58:56Z")

</div>

I see, thanks! Yes, just changing the `find` function took care of it.