# Performance with MongoDB 

**URL:** <https://discourse.julialang.org/t/performance-with-mongodb/7577>\
**Category:** Performance\
**Tags:** first-steps\
**Created:** [December 6, 2017, 6:31pm UTC](https://discourse.julialang.org/t/performance-with-mongodb/7577 "2017-12-06T18:31:02Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Massimo\_Redaelli](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/massimo_redaelli/32/2859_2.png) [@Massimo\_Redaelli](https://discourse.julialang.org/u/Massimo_Redaelli)\
**Post date:** [December 6, 2017, 6:31pm UTC](https://discourse.julialang.org/t/performance-with-mongodb/7577/1 "2017-12-06T18:31:02Z")

</div>

I recently changed job and I am now working in a company doing data science stuff with heavy emphasis on geospatial timeseries.  
I work with two statisticians that have so far implemented everything in R, and occasionally some Python.

After familiarizing myself with R and realizing just how horribly slow it is, I looked around for alternatives and got very excited about Julia - although I’m afraid we cannot adopt it because the Geo libraries are still at an early development level.

in any case, I wanted to try Julia anyway, so I rewrote in it a small Python script that basically consists on three nested for loops, each of which iterates over MongoDB queries.

I was surprised to find out that the resulting Julia script is about 50% slower than the equivalent Python script.  
I tried to follow all the guidelines in the “performance tips” page, but to no avail.

I can certainly post the code, if it can help, but is it possible that this is intrinsically a bad benchmark because of the use of Dicts?  
Should I try constructing the BSON directly instead?

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [December 6, 2017, 6:42pm UTC](https://discourse.julialang.org/t/performance-with-mongodb/7577/2 "2017-12-06T18:42:22Z")

</div>

> [@Massimo\_Redaelli](#):
>
> but is it possible that this is intrinsically a bad benchmark because of the use of Dicts?

Possibly. It’s hard to know without seeing the code what is going on.

---

<div class="post-metadata">

**Author:** ![Massimo\_Redaelli](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/massimo_redaelli/32/2859_2.png) [@Massimo\_Redaelli](https://discourse.julialang.org/u/Massimo_Redaelli)\
**Post date:** [December 7, 2017, 8:54am UTC](https://discourse.julialang.org/t/performance-with-mongodb/7577/3 "2017-12-07T08:54:26Z")

</div>

Here’s the code.

```julia
function date_with_duplicates_for_grid(grid_id:: Int32, client:: MongoClient)
 command_simple(client, "xxx", OrderedDict(
   "aggregate" => "yyy",
   "pipeline" => [
     Dict("\$match" => Dict("id_grid" => grid_id)),
     Dict("\$group" => Dict(
       "_id" => Dict("date" => "\$date", "id_grid"=> "\$id_grid"),
       "count" => Dict("\$sum" => 1))
     ),
     Dict("\$match" => Dict("count" => Dict("\$gt" => 1))),
     Dict("\$project" => Dict("_id.date" => 1))
   ]
  )
 )
end

function measurements_for(grid_id:: Int32, date:: DateTime, cdata:: MongoCollection)
 find(cdata, Dict("id_grid" => grid_id, "date" => date), Dict("_id" => 1, "rain" => 1))
end

function run(grids:: MongoCursor, client:: MongoClient)
 cdata = MongoCollection(client, "xxx", "yyy")
 for grid in grids
   grid_id = grid["id_grid"] :: Int32
   for doc in date_with_duplicates_for_grid(grid_id, client)["result"]
     d = doc["_id"]["date"] :: DateTime
     mes = [m:: BSONObject for m in measurements_for(grid_id, d, cdata)]
     vals = unique([m["rain"] :: Real for m in mes])
     if( length(vals) > 1 )
       println("More than one!")
     else
       for m in mes[2:end]
         id = m["_id"]
         delete(cdata, Dict("_id" => id))
       end
     end
   end
 end
end

```

---

<div class="post-metadata">

**Author:** ![Massimo\_Redaelli](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/massimo_redaelli/32/2859_2.png) [@Massimo\_Redaelli](https://discourse.julialang.org/u/Massimo_Redaelli)\
**Post date:** [December 15, 2017, 6:53pm UTC](https://discourse.julialang.org/t/performance-with-mongodb/7577/4 "2017-12-15T18:53:36Z")

</div>

Any suggestion?

---

<div class="post-metadata">

**Author:** ![dfdx](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dfdx/32/120_2.png) [@dfdx](https://discourse.julialang.org/u/dfdx)\
**Post date:** [December 15, 2017, 9:46pm UTC](https://discourse.julialang.org/t/performance-with-mongodb/7577/5 "2017-12-15T21:46:48Z")

</div>

I don’t see anything obviously slow, but you can benchmark each part with [BenchmarkTools.jl](https://github.com/JuliaCI/BenchmarkTools.jl) and find out what exactly performs poorly.

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [December 15, 2017, 10:05pm UTC](https://discourse.julialang.org/t/performance-with-mongodb/7577/6 "2017-12-15T22:05:30Z")

</div>

It seems `@profile` is more apt here?

---

<div class="post-metadata">

**Author:** ![dfdx](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dfdx/32/120_2.png) [@dfdx](https://discourse.julialang.org/u/dfdx)\
**Post date:** [December 15, 2017, 11:49pm UTC](https://discourse.julialang.org/t/performance-with-mongodb/7577/7 "2017-12-15T23:49:08Z")

</div>

Right, I’ve been drilling through my own code with `@btime` for so lang that I’ve forgotten there’s more standard way 🙂 +1 for `@profile`.