# Benchmark and develop at the same time

**URL:** <https://discourse.julialang.org/t/benchmark-and-develop-at-the-same-time/25339>\
**Category:** Tooling\
**Tags:** benchmark\
**Created:** [June 16, 2019, 2:49am UTC](https://discourse.julialang.org/t/benchmark-and-develop-at-the-same-time/25339 "2019-06-16T02:49:27Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![jlapeyre](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jlapeyre/32/4514_2.png) [@jlapeyre](https://discourse.julialang.org/u/jlapeyre)\
**Post date:** [June 16, 2019, 2:49am UTC](https://discourse.julialang.org/t/benchmark-and-develop-at-the-same-time/25339/1 "2019-06-16T02:49:28Z")

</div>

I’d like to run a benchmark suite on two branches of a repo of a module to check for regressions, and work on the module at the same time. It seems like this requires two copies of the repo. (Or copies of the branches). Any ideas ? I’ve been working on it for the past 9 hours and have not yet succeeded. Most of the time was spent in benchmarking while I thought the problem was solved when in fact it was not. For example, if Julia loads the module from the original location, and I fortuitously checkout the correct branch at the correct time, that is in sync with the benchmark script, it will appear that the scripts are working and are loading the module from the copied repo. Here are some ideas

1. I guess that syncing or copying the repo to another user’s directory (or another machine) would be rather foolproof. I did not try this yet. Barring a solution, this is what I will do next.

2. Use some kind of `Pkg` magic to choose the location from which the module is loaded. I re-read all the documentation on `LOAD_PATH` and `Pkg` I could find. I saw the word “stack”. That got me excited, but it only occurred once.

3. Just manipulate `LOAD_PATH`. This is what I have been doing. In several places, one is advised to `push!` a directory to the path in order to load a module living there. I’ll add a couple of things here that I did not find online.  
a) the directory that you push must be the top-level directory of the package containing the module (Or at least, it cannot be the parent directory). The other paths in `LOAD_PATH` have a special meaning. Furthermore `Base.load_path()` (undocumented) returns an array of paths (as strings). These work differently than the path you pushed. One of them is a path to a `Project.toml`. Another is the `stdlib` directory; that is, the parent directory to the top-level directories of the packages in the standard library. This might fool you into thinking you can do the same with a benchmarking directory.

b) I did not find an equivalent to POSIX `which` to find out where a module is without loading it. There are a few ideas online to find where a module is after it is loaded. For example `dirname(Base.functionloc(MyModule.eval, Tuple{Nothing})[1])`. Since I want to do a lot of benchmarking on this package, I just did `const topleveldir = @ __DIR__ ` in the module, and then read it after loading the module.

c) The paths in `LOAD_PATH` are searched in order. This should have been obvious, and became clear after `b)` was solved. So, you need to use `pushfirst!` rather than `push!`.

d) After getting b) and c) correct, Julia will try to load the package/module from the specified directory. But it fails to compile. Several error messages about missing dependencies of the dependencies of `MyModule` are printed.

I suspect that `LOAD_PATH` is not the way to shadow a module. It’s at best a hack with respect to `Pkg`.

---

<div class="post-metadata">

**Author:** ![baggepinnen](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/baggepinnen/32/693_2.png) [@baggepinnen](https://discourse.julialang.org/u/baggepinnen)\
**Post date:** [June 16, 2019, 10:25am UTC](https://discourse.julialang.org/t/benchmark-and-develop-at-the-same-time/25339/2 "2019-06-16T10:25:44Z")

</div>

If one branch does not change, can’t you just save the benchmark result to disc and compare against that?

---

<div class="post-metadata">

**Author:** ![jlapeyre](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jlapeyre/32/4514_2.png) [@jlapeyre](https://discourse.julialang.org/u/jlapeyre)\
**Post date:** [June 16, 2019, 4:56pm UTC](https://discourse.julialang.org/t/benchmark-and-develop-at-the-same-time/25339/3 "2019-06-16T16:56:08Z")

</div>

Yes that’s the way to do it. I was working with inherited scripts. This should solve most or all of the problem.