# Optional dependencies / Requires.jl

**URL:** <https://discourse.julialang.org/t/optional-dependencies-requires-jl/3294>\
**Category:** Internals & Design\
**Created:** [April 20, 2017, 4:40am UTC](https://discourse.julialang.org/t/optional-dependencies-requires-jl/3294 "2017-04-20T04:40:04Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![vchuravy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/vchuravy/32/8_2.png) [@vchuravy](https://discourse.julialang.org/u/vchuravy)\
**Post date:** [April 20, 2017, 4:40am UTC](https://discourse.julialang.org/t/optional-dependencies-requires-jl/3294/1 "2017-04-20T04:40:04Z")

</div>

I recently started thinking about conditional modules, while working on making sure `Requires.jl` works on 0.6 and I thought it would be helpful to recap discussions I had with @jameson and @tkelman. This follows in similar veins to [julep: conditional modules · Issue #6195 · JuliaLang/julia · GitHub](https://github.com/JuliaLang/julia/issues/6195) and [extern keyword (cf conditional modules) · Issue #15705 · JuliaLang/julia · GitHub](https://github.com/JuliaLang/julia/issues/15705#issuecomment-254264419)

### Status quo: `Requires.jl`

In [`Requires.jl`](https://github.com/mikeInnes/Requires.jl) you can defer the loading of a piece of code with the `@require` macro.

```julia-auto
using Requires

@require DataFrames begin
  println("DataFrames loaded")
end

println("Before using")
using DataFrames
println("After using")

```

In order to provide this feature (which is quite heavily used to provide some notion of optional dependencies), `Requires.jl` currently has to overwrite `Base.require` which is frowned upon and requires jumping through some major hoops in order to call the original version of `Base.require` that does all the heavy lifting. The dynamic nature of this also makes it impossible to precompile these codeblocks.

### First alternative approach: Keeping it dynamic

The minimal change to `Requires.jl` would be to extend `Base.require` to notify through a callback that a module was finished loading. I experimented with this a little bit in [https://github.com/JuliaLang/julia/tree/vc/loading\_callbacks](https://github.com/JuliaLang/julia/tree/vc/loading_callbacks) and [GitHub - vchuravy/Requires.jl at vc/patchedbase](https://github.com/vchuravy/Requires.jl/tree/vc/patchedbase), but this still runs into the problem that it is not precompile friendly and so if an optional dependency is later installed we won’t invalidate the cache file.

### Second alternative approach: Static and precompile friendly

The second approach that is a lot more involved in terms of required base changes is to change `Requires.jl` dynamic delayed loading to a more straight-forward, check if module is loadable, if yes compile code, if not don’t compile code, but record that dependency and invalidate the cache if that dependency gets installed.

`@require mod expr` will evaluate to something akin to:

```julia-auto
if Base.find_in_node($(String(mod)), nothing, 1) !== nothing
   return expr
else
  return quote
      register_optional_dependency(mod)
  end
end

```

The big question mark for this approach is `register_optional_dependency`. I started experimenting a little bit in that direction, but currently that code crashes and burns when precompiling.

- [GitHub - vchuravy/julia at vc/better\_requires](https://github.com/vchuravy/julia/tree/vc/better_requires)
- [GitHub - vchuravy/Requires.jl at vc/static](https://github.com/vchuravy/Requires.jl/tree/vc/static)

mostly because I tried to use `uuid == 0` to mean optional in the cache file module list.

I would be keen on hearing thoughts from others.

---

<div class="post-metadata">

**Author:** ![tkelman](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkelman/32/692_2.png) [@tkelman](https://discourse.julialang.org/u/tkelman)\
**Post date:** [April 20, 2017, 4:53am UTC](https://discourse.julialang.org/t/optional-dependencies-requires-jl/3294/2 "2017-04-20T04:53:51Z")

</div>

Haven’t looked at the patches yet, but re:

> invalidate the cache if that dependency gets installed

why would it need to be at install time of the dependency rather than at load time for the package that has conditional dependencies? If we store a separate class of dependency in the `.ji` file for conditional dependencies, and record at precompile time which of them were or were not available at that time, we can check when that same `.ji` file is next loaded, whether or not the available set has changed. If you install some conditional dependency, but then remove it before you next try to load the package that conditionally used it, the original cache would still be valid, right?

---

<div class="post-metadata">

**Author:** ![vchuravy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/vchuravy/32/8_2.png) [@vchuravy](https://discourse.julialang.org/u/vchuravy)\
**Post date:** [April 20, 2017, 4:55am UTC](https://discourse.julialang.org/t/optional-dependencies-requires-jl/3294/3 "2017-04-20T04:55:10Z")

</div>

Yes absolutely. The cache would be invalidate once the package is loaded again.

---

<div class="post-metadata">

**Author:** ![vchuravy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/vchuravy/32/8_2.png) [@vchuravy](https://discourse.julialang.org/u/vchuravy)\
**Post date:** [April 20, 2017, 10:20am UTC](https://discourse.julialang.org/t/optional-dependencies-requires-jl/3294/4 "2017-04-20T10:20:23Z")

</div>

The version with the static approach is now usable at:

- [GitHub - vchuravy/julia at vc/better\_requires](https://github.com/vchuravy/julia/tree/vc/better_requires)
- [GitHub - vchuravy/Requires.jl at vc/static](https://github.com/vchuravy/Requires.jl/tree/vc/static)

Please note that it is still prototype functionality and the code will need to be cleaned up. To test it create a package that either calls `jl_module_register_optional` or that contains:

```julia-auto
module TestRequires

__precompile__ ()

using Requires

@require WeakRefStrings begin
  f(x) = "Here!, Here!"
end

# package code goes here

end # module

```

##### What works?

- Precompiling the package, without the optional dependency installed
- Installing the optional dependency will invalidate the cache, when the package is next loaded.

##### What doesn’t work?

- Removing the optional dependency, will throw:

```julia-auto
julia> using TestRequires
ERROR: ArgumentError: Module WeakRefStrings not found in current path.
Run `Pkg.add("WeakRefStrings")` to install the WeakRefStrings package.

```

Instead it should probably invalidate the cache again.

Please give it a try and poke wholes into the design.

---

<div class="post-metadata">

**Author:** ![MikeInnes](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mikeinnes/32/3656_2.png) [@MikeInnes](https://discourse.julialang.org/u/MikeInnes)\
**Post date:** [April 20, 2017, 11:58am UTC](https://discourse.julialang.org/t/optional-dependencies-requires-jl/3294/5 "2017-04-20T11:58:12Z")

</div>

> [@vchuravy](#):
>
> The dynamic nature of this also makes it impossible to precompile these codeblocks.

FWIW, I haven’t found this to be a problem in practice, and I don’t know of any package where being able to precompile these blocks would give a significant speedup; `@requires` blocks are generally only a few lines of header-like code.

Being able to auto-import packages is a nice feature of the static approach which would be useful for optional backends. It wouldn’t be able to handle all cases that the dynamic approach can; for example, if you load a package then install and load a conditional dependency of that package without restarting. I think the only full solution is conditional sub-modules that are separately precompiled and handled specially by the package system.

I would personally favour the support for `require` hooks in base as a simple short-term solution, and then figure out something better. (I’m not sure why you think this would cause an issue with cache files though; the require blocks aren’t cached so there’s nothing to change if the set of available deps changes.)

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [April 20, 2017, 12:47pm UTC](https://discourse.julialang.org/t/optional-dependencies-requires-jl/3294/6 "2017-04-20T12:47:05Z")

</div>

> [@MikeInnes](#):
>
> I don’t know of any package where being able to precompile these blocks would give a significant speedup; @requires blocks are generally only a few lines of header-like code.

Plots.jl. It lazy loads every backend. That would likely be helped a lot by some kind of precompilation. It doesn’t use Requires.jl though: just straight `@eval`

---

<div class="post-metadata">

**Author:** ![vchuravy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/vchuravy/32/8_2.png) [@vchuravy](https://discourse.julialang.org/u/vchuravy)\
**Post date:** [April 20, 2017, 1:03pm UTC](https://discourse.julialang.org/t/optional-dependencies-requires-jl/3294/7 "2017-04-20T13:03:56Z")

</div>

I think what I call _static_ and _dynamic_ approach are orthogonal to each other, sometimes you will want to do something in reaction to a package being loaded and other times you will want to use precompilation in order to define conditional modules.

Why do you think that dynamic conditional sub-modules are necessary? Purely for the user experience of not having to restart? One could probably do something akin to.

```julia
if Base.find_in_node(String(mod), nothing, 1) !== nothing
   return expr
else
  return quote
      register_optional_dependency(mod)
      # dynamic code loading, in case module becomes available later
      end
      listenmod(mod) do
         eval(expr) 
      end
  end
end

```

to get the effect you were thinking about. Also none of these are changes that have to go into Requires.jl 🙂 I just used that as a starting point to understand the design space better.

---

<div class="post-metadata">

**Author:** ![MikeInnes](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mikeinnes/32/3656_2.png) [@MikeInnes](https://discourse.julialang.org/u/MikeInnes)\
**Post date:** [April 20, 2017, 1:38pm UTC](https://discourse.julialang.org/t/optional-dependencies-requires-jl/3294/8 "2017-04-20T13:38:40Z")

</div>

I don’t think the dynamic approach is necessary or better as such; just chiming with what you’ve said, which is that they are independent features with slightly different tradeoffs. Your static proposal is a good one and may well be better overall. In principle they could both co-exist, although I suspect the overlap is enough that it’s better to pick one.

I guess I should revise my preference to “whatever’s easiest”. Adding a hook to `require` is trivial to implement compared to having a cache dependency on a non-existent file, but then bikeshedding with Base is part of the implementation cost too 🙂

---

<div class="post-metadata">

**Author:** ![MikeInnes](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mikeinnes/32/3656_2.png) [@MikeInnes](https://discourse.julialang.org/u/MikeInnes)\
**Post date:** [April 20, 2017, 2:17pm UTC](https://discourse.julialang.org/t/optional-dependencies-requires-jl/3294/9 "2017-04-20T14:17:40Z")

</div>

I haven’t looked at Plots in detail, but most of the time Julia’s late binding means that you can write a bulk of the heavy lifting code without the library you’re using being loaded. The main exception is adding new methods, for which you obviously need the function and any types to be available. But then you can use requires _just_ for that one definition that ties together the two packages, i.e.

```julia
function real_foo(x)
  Foo.bar(x)
end

@require Foo begin
  using Foo
  Foo.foo(x::MyType) = real_foo(x)
end

```

In Plots’ case, you could probably get almost all of the benefits of precompilation like this:

```julia
foo_code = parse("foo.jl")
load_foo() = eval(foo_code)
# instead of include("foo_code.jl") directly

```

---

<div class="post-metadata">

**Author:** ![jeff.bezanson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jeff.bezanson/32/48_2.png) [@jeff.bezanson](https://discourse.julialang.org/u/jeff.bezanson)\
**Post date:** [April 20, 2017, 3:27pm UTC](https://discourse.julialang.org/t/optional-dependencies-requires-jl/3294/10 "2017-04-20T15:27:38Z")

</div>

How do we express version requirements for optional dependencies?

---

<div class="post-metadata">

**Author:** ![lobingera](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lobingera/32/211_2.png) [@lobingera](https://discourse.julialang.org/u/lobingera)\
**Post date:** [April 20, 2017, 3:41pm UTC](https://discourse.julialang.org/t/optional-dependencies-requires-jl/3294/11 "2017-04-20T15:41:35Z")

</div>

… like for required dependencies (?). Should this be different from your point of view?

---

<div class="post-metadata">

**Author:** ![jeff.bezanson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jeff.bezanson/32/48_2.png) [@jeff.bezanson](https://discourse.julialang.org/u/jeff.bezanson)\
**Post date:** [April 20, 2017, 4:35pm UTC](https://discourse.julialang.org/t/optional-dependencies-requires-jl/3294/12 "2017-04-20T16:35:31Z")

</div>

I’m not sure, but doesn’t that mean all the optional dependencies are actually required, from the package manager’s point of view?

---

<div class="post-metadata">

**Author:** ![lobingera](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lobingera/32/211_2.png) [@lobingera](https://discourse.julialang.org/u/lobingera)\
**Post date:** [April 20, 2017, 4:45pm UTC](https://discourse.julialang.org/t/optional-dependencies-requires-jl/3294/13 "2017-04-20T16:45:54Z")

</div>

I have the feeling this was discussed before, but stil…

Required dependencies enable the package to do something, without the dependencies the package cannot do anything. Optional dependencies enable the package to do some more.

A plotting package might require a package providing colormaps. If more colormap packages are installed, you can use them additionally.

---

<div class="post-metadata">

**Author:** ![jeff.bezanson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jeff.bezanson/32/48_2.png) [@jeff.bezanson](https://discourse.julialang.org/u/jeff.bezanson)\
**Post date:** [April 20, 2017, 4:49pm UTC](https://discourse.julialang.org/t/optional-dependencies-requires-jl/3294/14 "2017-04-20T16:49:17Z")

</div>

IIUC, the only way we have to express version requirements is the `REQUIRE` file. But that can only express required dependencies. Do we need to add optional dependency support to `REQUIRE`?

---

<div class="post-metadata">

**Author:** ![JaredCrean2](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jaredcrean2/32/3574_2.png) [@JaredCrean2](https://discourse.julialang.org/u/JaredCrean2)\
**Post date:** [April 20, 2017, 4:51pm UTC](https://discourse.julialang.org/t/optional-dependencies-requires-jl/3294/15 "2017-04-20T16:51:31Z")

</div>

Has there been any discussion of expanding the data available to the package manager? `Pkg.add("Pkgname")` doesn’t allow the user to provide more information about how to build the package.

For a point of comparision, the C/C++ libraries I use (particularly PETSc) have lots of configuration options available, of the form `configure --with-feature-x --without-dependency-y`.

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [April 20, 2017, 4:56pm UTC](https://discourse.julialang.org/t/optional-dependencies-requires-jl/3294/16 "2017-04-20T16:56:55Z")

</div>

[https://github.com/JuliaLang/julia/issues/20082](https://github.com/JuliaLang/julia/issues/20082)

---

<div class="post-metadata">

**Author:** ![lobingera](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lobingera/32/211_2.png) [@lobingera](https://discourse.julialang.org/u/lobingera)\
**Post date:** [April 20, 2017, 5:03pm UTC](https://discourse.julialang.org/t/optional-dependencies-requires-jl/3294/17 "2017-04-20T17:03:36Z")

</div>

Didn’t i read

```julia
...
[package.Required]
uuid = "85241492-0f92-400a-8719-bdc0424991f7"
versions = ["1.2-1.3", "!1.2.5"]

[package.Optional]
...

```

in the Pkg3 description? Adding this right now to REQUIRE is too much of a workaround (hack?)

---

<div class="post-metadata">

**Author:** ![tkelman](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkelman/32/692_2.png) [@tkelman](https://discourse.julialang.org/u/tkelman)\
**Post date:** [April 20, 2017, 5:20pm UTC](https://discourse.julialang.org/t/optional-dependencies-requires-jl/3294/18 "2017-04-20T17:20:54Z")

</div>

Would need a way of expressing “if installed, must satisfy these version constraints.” If the version constraints aren’t satisfiable, would the alternative be disallowing the optional package from being installed at all?

Or we could include the version preconditions in the “if installed [and appropriate version], enable this block of code” check.

---

<div class="post-metadata">

**Author:** ![lobingera](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lobingera/32/211_2.png) [@lobingera](https://discourse.julialang.org/u/lobingera)\
**Post date:** [April 20, 2017, 5:26pm UTC](https://discourse.julialang.org/t/optional-dependencies-requires-jl/3294/19 "2017-04-20T17:26:56Z")

</div>

I’d vote for the second.

---

<div class="post-metadata">

**Author:** ![tobias.knopp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tobias.knopp/32/7551_2.png) [@tobias.knopp](https://discourse.julialang.org/u/tobias.knopp)\
**Post date:** [April 20, 2017, 5:35pm UTC](https://discourse.julialang.org/t/optional-dependencies-requires-jl/3294/20 "2017-04-20T17:35:06Z")

</div>

> [@jeff.bezanson](#):
>
> How do we express version requirements for optional dependencies?

I think this is a very important point.

Wouldn’t Plots be much more naturally be organized as follows

```julia
using Plots #this only has abstract stubs 
using PlotsPyPlot #Pulls in the implementation of Plots

```

[Next page](https://discourse.julialang.org/t/optional-dependencies-requires-jl/3294.md?page=2)