# What's the best practice in Julia for research (often changed codes)?

**URL:** <https://discourse.julialang.org/t/whats-the-best-practice-in-julia-for-research-often-changed-codes/58892>\
**Category:** General Usage\
**Tags:** design-pattern\
**Created:** [April 9, 2021, 12:18am UTC](https://discourse.julialang.org/t/whats-the-best-practice-in-julia-for-research-often-changed-codes/58892 "2021-04-09T00:18:01Z")\
**Posts on this page:** 7\
**Page:** 2

<div class="post-metadata">

**Author:** ![iHany](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ihany/32/18151_2.png) [@iHany](https://discourse.julialang.org/u/iHany)\
**Post date:** [April 10, 2021, 1:56pm UTC](https://discourse.julialang.org/t/whats-the-best-practice-in-julia-for-research-often-changed-codes/58892/21 "2021-04-10T13:56:22Z")

</div>

Yes, actually I’d tried to use PackageCompiler very early, and I might have been confused with how to use it.  
I’ll try your suggestion (without `replace_default=true`).  
Thanks 🙂

EDIT: do you execute the precompilation code in your custom environment (pacakge)? Or, in default env?

---

<div class="post-metadata">

**Author:** ![ffevotte](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ffevotte/32/6587_2.png) [@ffevotte](https://discourse.julialang.org/u/ffevotte)\
**Post date:** [April 10, 2021, 4:31pm UTC](https://discourse.julialang.org/t/whats-the-best-practice-in-julia-for-research-often-changed-codes/58892/22 "2021-04-10T16:31:33Z")

</div>

> [@iHany](#):
>
> do you execute the precompilation code in your custom environment (pacakge)? Or, in default env?

I usually have a dedicated `build` sub-directory defining its own environment (much like how things are traditionally set up for using `Documenter` in the `docs` subdirectory)

```julia
MyProject
├── build
│ │ # PackageCompiler-specific environment
│ ├── Project.toml # add PackageCompiler
│ ├── Manifest.toml # dev ..
│ │
│ ├── make.jl # script calling `create_sysimage`
│ └── precomp.jl
│
│ # Normal project environment
├── Project.toml
├── Manifest.toml
│
└── src
    └── MyProject.jl

```

I’ve actually been thinking of adding a plugin to set this up in `PkgTemplates`; now might be a good time to do it 🙂

---

<div class="post-metadata">

**Author:** ![jzr](https://avatars.discourse-cdn.com/v4/letter/j/eb9ed0/32.png) [@jzr](https://discourse.julialang.org/u/jzr)\
**Post date:** [April 10, 2021, 6:59pm UTC](https://discourse.julialang.org/t/whats-the-best-practice-in-julia-for-research-often-changed-codes/58892/23 "2021-04-10T18:59:54Z")

</div>

> [@ffevotte](#):
>
> But when `Foo_v2` comes into play, no new `bar` method is introduced (unless `Revise` also detects changes to the definition of `bar` itself, in which case it redefines the correct method)

I thought redefining the method would be good and Revise should do that, no?

---

<div class="post-metadata">

**Author:** ![ffevotte](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ffevotte/32/6587_2.png) [@ffevotte](https://discourse.julialang.org/u/ffevotte)\
**Post date:** [April 10, 2021, 7:19pm UTC](https://discourse.julialang.org/t/whats-the-best-practice-in-julia-for-research-often-changed-codes/58892/24 "2021-04-10T19:19:39Z")

</div>

IIUC, when `Revise` sees something like

```julia
bar(::Foo) = 1

```

it does not keep track of the fact that the method definition depends on what `Foo` happens to be bound to at the time when the method definition was evaluated. This makes sense because Julia types are normally constant. But if `Foo` happens to be a variable and it gets re-bound to a new value (a different version of the type), then Revise currently has no way of knowing that it should re-evaluate the definition of this particular method of `bar`.

However, Revise will re-evaluate the definition of `bar(::Foo)` if its source code changes.

You can try and see what happens when you put some code like this in a package

```julia
module MyPackage
   struct Foo_v1 end
   Foo = Foo_v1

   bar(::Foo) = 1
end

```

and then in a REPL:

```julia
julia> using Revise
julia> using MyPackage
[Info: Precompiling MyPackage [ae553910-327c-57b0-bdb1-851592e5e7e1]

julia> MyPackage.bar(MyPackage.Foo())
1

# At this point, edit `MyPackage.jl` to replace `Foo_v1` with `Foo_v2`
# Revise will be happy with such changes, but...

julia> MyPackage.bar(MyPackage.Foo())
ERROR: MethodError: no method matching bar(::MyPackage.Foo_v2)
Closest candidates are:
  bar(::MyPackage.Foo_v1) at /tmp/MyPackage.jl/src/MyPackage.jl:5
Stacktrace:
 [1] top-level scope
   @ REPL[3]:1

```

Revise did account for the change in the `Foo` constructor (the source code of its definition was updated to point to `Foo_v2`), but not `bar` (its source code was left untouched).

---

<div class="post-metadata">

**Author:** ![jzr](https://avatars.discourse-cdn.com/v4/letter/j/eb9ed0/32.png) [@jzr](https://discourse.julialang.org/u/jzr)\
**Post date:** [April 10, 2021, 7:39pm UTC](https://discourse.julialang.org/t/whats-the-best-practice-in-julia-for-research-often-changed-codes/58892/25 "2021-04-10T19:39:03Z")

</div>

> [@ffevotte](#):
>
> Revise currently has no way of knowing that it should re-evaluate the definition of this particular method of `bar` .

That seems like it would be a useful feature for Revise. Is there a downside?

---

<div class="post-metadata">

**Author:** ![ffevotte](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ffevotte/32/6587_2.png) [@ffevotte](https://discourse.julialang.org/u/ffevotte)\
**Post date:** [April 10, 2021, 8:12pm UTC](https://discourse.julialang.org/t/whats-the-best-practice-in-julia-for-research-often-changed-codes/58892/26 "2021-04-10T20:12:00Z")

</div>

> [@jzr](#):
>
> That seems like it would be a useful feature for Revise.

I think everybody would agree with you!

> [@jzr](#):
>
> Is there a downside?

We’ve more or less reached the limits of my understanding of how these things work, but here are two discussions that might be of interest to you (and lead to other places where such matters are discussed)

> <https://github.com/timholy/Revise.jl/issues/18>
>
> Since \`struct\`s cannot be redefined, attempting to do so currently gives a \`fail…ure to evaluate changes\` warning.
> 
> I wonder if this can be worked around by triggering a reload of the entire module in this case.

> <https://github.com/JuliaLang/julia/issues/40399>
>
> I've been doing some work that has a fairly big dependency stack (100-200 packag…es) and as such takes a few minutes to warm up. Because of this non-Revise-able code is extremely painful, so I'd like to take another look at https://github.com/timholy/Revise.jl/issues/18. This has some fairly long history of course, but to avoid getting buried in the discussion, I figured a new issue for my specific proposal would be best.
> 
> One solution that was previously discussed (in https://github.com/JuliaLang/julia/pull/22721) was to have backedges for bindings, but this was rejected as too expensive. My proposal aims to basically do this, but shift the cost of backedge tracking to the redefinition stage where it is more acceptable.
> 
> In particular, my proposal is:
> \- Make each module's binding table partitioned by world ages. Introducing a new binding does not raise the world age, but bindings do record the world age where they were introduced.
> \- Change binding lookup to only match bindings that are visible to the current world age
> \- When wanting to redefine a binding (e.g. to replace a struct definition by an updated struct definition of the same name), we do a GC-like walk of all methods in the system and look at their source. If they contain a GlobalRef for the binding being replaced, we truncate the world bounds of any associated method instances (in effect having the GlobalRef serve as "forward edges" that get re-validated by the redefinition). This operation would increase the world age.
> \- We add an array for explicit forward edges from non-syntactic resolutions of bindings during inference. This can basically happen for two reasons:
> 1. Something like \`getfield(Main, :foo)\` which will get resolved via the getfield\_tfunc.
> 2. Generated functions
> 
> cc @JeffBezanson @vtjnash
> 
> If this proposal seems reasonable, I'd hope to nerdsnipe @timholy into taking charge :)

---

<div class="post-metadata">

**Author:** ![thisrod](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/thisrod/32/10642_2.png) [@thisrod](https://discourse.julialang.org/u/thisrod)\
**Post date:** [April 20, 2021, 1:04am UTC](https://discourse.julialang.org/t/whats-the-best-practice-in-julia-for-research-often-changed-codes/58892/27 "2021-04-20T01:04:58Z")

</div>

> [@Henrique\_Becker](#):
>
> Your idea does not work for parametric types because:
> 
> ```julia
> ...
> Foo{T} = Foo1{T}
> ...
> 
> ```

That doesn’t work, but it wasn’t my idea. 🙂 As far as I can see, the `Foo = Foo1` version works fine with a parametric type. The whole point is for `Foo` to be a variable, so that reevaluating the method definitions adds extra methods instead of changing the existing methods.

[Previous page](https://discourse.julialang.org/t/whats-the-best-practice-in-julia-for-research-often-changed-codes/58892.md?page=1)