It may be my stupidity, but I do not seem to find exactly what I should pass to Base.require.
The docs are not very clear on what I should pass as first parameter (is it the current module? is it Main?). And the only mention with a small tidbit of code in the module documentation treat Base.require as it had only the second parameter.
I think the docs didn’t fully update when the into argument was added. Note that there is still a 1-argument method.
In brief, the 2-argument version is used for handling dependencies. If PkgA depends on PkgB but you haven’t said using PkgB, then PkgB will not be available as a name in Main. But PkgA needs to know about it.
Base.require is definitely not something I think of as “first steps”—in general there should be no need to call it directly. What are you trying to do?
I did not see the Base.require with just one argument still existed.
I think I maybe misfiled the question, XD, as I had the feeling I was not understanding something I believed to be basic I filed it as first steps.
I have some code that implements many mathematical models and that I will make available as a package in the future. This code includes a module that makes easier to a package user (but mostly to myself) to call some model over some file using some solver. I do not want to add all possible solvers as dependencies of the package just for this. I want that based on the “use this solver” parameter given by the user, the code dynamically loads the specific solver package (assuming the user must have installed if they told the method to use it).
I did not study it fully, but I knew about its existence. It seems overkill for what I want. I do not want to a part of my code to be loaded only if some package is available. The part that changes from solver to solver is small (thanks to JuMP) I just want to not need to define them as dependencies, nor have Julia print a message to the user saying that there is a bug with my code because I import packages that are not dependencies.
I’m not 100% sure, but I think that this is not currently possible. In order to import a package and use its types, it needs to be a dependency. This is why there are a bunch of <Domain>Base.jl packages that define abstract types that other packages can use for interoperability without taking on massive dependencies.
I’ve found it works better to just require the user to pass a solver (just as JuMP itself does); the alternative is to depend on solvers and/or use Requires.jl, neither of which is ideal (especially with non-free solvers that the user might not have). I think things like choosing solver-specific parameters are better handled in the documentation (e.g. say to use such-and-such solver, do using Solver and pass the solver with these parameters). Or, possibly, depend on just one default (ideally open source) solver and require the user to pass a solver if they want to use a non-default one.
I think you can use packages that are not dependencies, otherwise Requires.jl would not be possible.
I have not tested my specific approach yet, but I think I can require a package that is not a dependency and risk having an exception if it is not present in the system. The idea here is that this require will only execute if the user explicitly orders it to use some solver, and all code that depends on such specific package will be just below the require call. The method that uses the non-dependency package will not even be called if the user do not order the use of the specific solver.
This really does not work for this specific case, as what I want is to provide such solver selection and configuration as flags of a script I will be the main user, but other users/scientists may find useful too. The core functionality of my package uses the approach you describe (all model building methods take a model object as a parameter and do not meddle with the specific solver), but this non-core functionality is a command-line script that should be ready to use the solver the user wants with the most common parameters among the solvers without having the user to install all solvers for this. I use it to do my experiments, and it would be an easy way to other scientists reproduce my results. If they want to pass additional flags they can mess with the code.
I would also like to point that the require method that takes just one parameter is not documented even if yet present in the code
The docstrings of require is heavily outdated (https://github.com/JuliaLang/julia/issues/27822)
The way I would address that situation is to have the command-line script outside of the package code, in a separate folder with its own Project and Manifest files. Then that script can load the package as well as the solver you used and do the computation. That way the Manifest will help make it reproducible. If reproducibility is the goal, I don’t think optional dependencies are a good idea since then the script does slightly different things depending on their environment.
I think that’s because Requires.jl makes a macro that allows you to write code that’s not actually evaluated when you load your package.
By all means, try and report back. My confidence is only at ~60% on this. But I’m pretty certain that if you include things that aren’t defined, even if they’re inside functions or other blocks, julia typically throws the error when it tries to compile.
I had a lot to work to do before answering your question, XD. Seems like I would be able to do what I wanted (i.e., not use Requires.jl but use a package that is not a hard dependency of mine) if my use case was simpler but, unfortunately, my use case is more convoluted. My idea was using the code below:
function import_if_necessary(module_sym :: Symbol) :: Nothing
if !Base.isdefined(@__MODULE__, module_sym)
Base.include_string(@__MODULE__, "import $(module_sym)")
end
end
function empty_configured_model(::Val{:CPLEX}, p_args)
Base.invokelatest(import_if_necessary, :CPLEX)
Base.invokelatest(cplex_empty_configured_model, p_args)
end
The method cplex_empty_configured_model in question called methods defined by package/module CPLEX inside it. This worked. The problem was in the function that called optimize!, it did not work even putting the call to optimize! in the same module, calling it with Base.invokelatest and, therefore, giving it access to the CPLEX module. As optimize! is not a CPLEX method but calls a MathOptInterface method that has an specialization defined in CPLEX.jl, this workaround did not in fact work. If I just wanted to build the model using a specific solver, it would have worked, but as I need to call optimize! the problem becomes harder.
I had to settle for using Requires.jl and asking the user to manually add an “import SolverPackageX” (with the solver package they will use) in the script before calling a method that collects all command-line arguments and do what needs to be done (i.e., the user has to manually add a line to a 5-line script or Requires.jl will also not solve my problem).
If you are OK with invokelatest you can use require like this
withpkg(f, pkgid::Base.PkgId) = Base.invokelatest(f, Base.require(pkgid))
withpkg(Base.PkgId(Base.UUID(0x682c06a0de6a54aba142c8b1cf79cde6), "JSON")) do JSON
JSON.print(Dict("hello" => "world"))
end
This is likely a “bad” strategy, but sometimes it’s handy.
This does not require the package to be importable:
(@v1.x) pkg> rm JSON
Updating `~/.julia/environments/v1.x/Project.toml`
[682c06a0] - JSON v0.21.0
Updating `~/.julia/environments/v1.x/Manifest.toml`
[no changes]
julia> withpkg(f, pkgid::Base.PkgId) = Base.invokelatest(f, Base.require(pkgid))
withpkg (generic function with 1 method)
julia> withpkg(Base.PkgId(Base.UUID(0x682c06a0de6a54aba142c8b1cf79cde6), "JSON")) do JSON
JSON.print(Dict("hello" => "world"))
end
{"hello":"world"}
julia> using JSON
ERROR: ArgumentError: Package JSON not found in current path:
- Run `import Pkg; Pkg.add("JSON")` to install the JSON package.
Stacktrace:
[1] require(::Module, ::Symbol) at ./loading.jl:892