# CategoricalArrays forces OpenSpecFun\_jll recompilation

**URL:** <https://discourse.julialang.org/t/categoricalarrays-forces-openspecfun-jll-recompilation/87759>\
**Category:** Internals & Design\
**Tags:** method-invalidation\
**Created:** [September 24, 2022, 7:56pm UTC](https://discourse.julialang.org/t/categoricalarrays-forces-openspecfun-jll-recompilation/87759 "2022-09-24T19:56:13Z")\
**Posts on this page:** 6\
**Page:** 2

<div class="post-metadata">

**Author:** ![ToucheSir](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/touchesir/32/14411_2.png) [@ToucheSir](https://discourse.julialang.org/u/ToucheSir)\
**Post date:** [September 26, 2022, 5:39am UTC](https://discourse.julialang.org/t/categoricalarrays-forces-openspecfun-jll-recompilation/87759/21 "2022-09-26T05:39:46Z")

</div>

So I’m understanding this correctly, does the proposed `@require @eval using ...` pattern essentially give us the lazy glue code loading Requires is currently used for while not incurring most of the drawbacks of using Requires? Or is it providing something else/something far less powerful? If the former, that sounds like an amazing stopgap until we get proper conditional deps.

---

<div class="post-metadata">

**Author:** ![mkitti](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mkitti/32/12459_2.png) [@mkitti](https://discourse.julialang.org/u/mkitti)\
**Post date:** [September 26, 2022, 5:47am UTC](https://discourse.julialang.org/t/categoricalarrays-forces-openspecfun-jll-recompilation/87759/22 "2022-09-26T05:47:03Z")

</div>

A pure “Core” module to me only has types and no methods. In practice, you probably need some constructors and methods for the “Core” module to have any utility.

In one sense I agree with you that there should be few precompile statements in a “Core” package but that is mostly because there should not be much to precompile there to begin with.

> [@davidanthoff](#):
>
> So that sounds interesting! I guess in that case the idea would be that we could define four new packages (one for each of the `@require` integration points that CategoricalArrays currently has), put all the code that is currently in these `@require` blocks into these packages, and then only load these new integration packages in the `@require` clause? If that would solve the problem, it might be the least involved to get going?

Yes, that is essentially what I was proposing above.

> [@ToucheSir](#):
>
> So I’m understanding this correctly, does the proposed `@require @eval using ...` pattern essentially give us the lazy glue code loading Requires is currently used for while not incurring most of the drawbacks of using Requires? Or is it providing something else/something far less powerful? If the former, that sounds like an amazing stopgap until we get proper conditional deps.

It provides an opportunity to do precompilation caching of the glue code. One has to make sure that the interface package does indeed include precompile statements. As far as Julia is concerned the interface package is just a normal package.

---

<div class="post-metadata">

**Author:** ![nalimilan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nalimilan/32/147_2.png) [@nalimilan](https://discourse.julialang.org/u/nalimilan)\
**Post date:** [September 26, 2022, 7:05am UTC](https://discourse.julialang.org/t/categoricalarrays-forces-openspecfun-jll-recompilation/87759/23 "2022-09-26T07:05:09Z")

</div>

I guess as a short-term measure we could at least turn RecipesBase and StructTypes into plain dependencies.

For SentinelArrays, I wonder whether we could find a solution by adding an API to ArrayInterfaceCore. Indeed, the problem to solve is just that CategoricalArrays defines a `copyto!` method dispatching on the destination array, while SentinelArrays defines one on the source array. But all that’s needed is to call the general fallback defined in SentinelArrays, there’s no particular interaction between these two packages. If we could add an API to indicate that SentinelArrays should take precedence all would work fine.

Regarding JSON.jl, the only line behind `@requires` is `JSON.lower(x::CategoricalValue) = JSON.lower(unwrap(x))`. I don’t understand how it can causes invalidations. Care to explain?

---

<div class="post-metadata">

**Author:** ![mkitti](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mkitti/32/12459_2.png) [@mkitti](https://discourse.julialang.org/u/mkitti)\
**Post date:** [September 26, 2022, 7:10am UTC](https://discourse.julialang.org/t/categoricalarrays-forces-openspecfun-jll-recompilation/87759/24 "2022-09-26T07:10:39Z")

</div>

> [@nalimilan](#):
>
> Regarding JSON.jl, the only line behind `@requires` is `JSON.lower(x::CategoricalValue) = JSON.lower(unwrap(x))`. I don’t understand how it can causes invalidations. Care to explain?

I’m not sure if the issue is that it’s causing invalidations. Rather there is no ability to cache the type inferred code anywhere for that method, potentially lengthening load times.

---

<div class="post-metadata">

**Author:** ![ToucheSir](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/touchesir/32/14411_2.png) [@ToucheSir](https://discourse.julialang.org/u/ToucheSir)\
**Post date:** [September 26, 2022, 2:59pm UTC](https://discourse.julialang.org/t/categoricalarrays-forces-openspecfun-jll-recompilation/87759/25 "2022-09-26T14:59:37Z")

</div>

How does this look in practice? Can packages keep their `@require PkgName=UUID using GluePkg` statements at the top level, or does this require changes to code using Requires?

---

<div class="post-metadata">

**Author:** ![tim.holy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tim.holy/32/52_2.png) [@tim.holy](https://discourse.julialang.org/u/tim.holy)\
**Post date:** [September 26, 2022, 4:47pm UTC](https://discourse.julialang.org/t/categoricalarrays-forces-openspecfun-jll-recompilation/87759/26 "2022-09-26T16:47:58Z")

</div>

The `@require` statements have to go in ` __init__ `, as usual. But with `@eval` I don’t think there’s any barrier to doing this today, without any need for changes to Requires.jl. I’ve generally not bothered simply because when I’ve used Requires it’s mostly been to load a small amount of code, but there are cases where the precompilation would be very desirable.

[Previous page](https://discourse.julialang.org/t/categoricalarrays-forces-openspecfun-jll-recompilation/87759.md?page=1)