About Pkg Apps and Extentions

General Usage
, ,
1

While reading 6. Apps · Pkg.jl, I was wondering if one could define a app (with its own submodule) inside of a package or if it needed to be another package altogether. In particular, what I would ideally like to do is

  • In an already developped package Foo
  • That has an extension FooBarExt
  • Define an app FooBarApp that uses Foo and Bar and therefore loads FooBarExt as well

This is mostly to keep things well organized and avoid multiplying repos and increase maintenance load.

2

If you want to specify an app that depends on Foo and Bar in Foo’s Project.toml, then you need Foo or a submodule to import Bar, so it no longer makes sense for the extension FooBarExt to exist. What would make sense is the app being based on a new package that depends on Foo and Bar, thus loading FooBarExt. But you probably wouldn’t put FooBar in the name of the package or the app by convention.

1 Like
3

Maybe I misunderstood how these Apps work, but basically my use case is that Foo handles a specific type of files and Bar handles another one, and I have a FooBarExt that handles conversion from Foo to Bar format. And I have a similar approach for other Baz … packages that define similar extensions.

So ideally I’d like to facilitate conversions like Foo <-> Bar or Foo <-> Baz, but if I understand correctly your answer the only way to do that is to define a new package say FooConvert that has Foo, Bar and all the Bazs as dependencies.
However, please correct me if this is wrong, I should be able to make a FooBarConvert app that only loads Foo and Bar (and therefore FooBarExt) only so that Baz is never involved, and a separate similar one for Baz.

Or maybe I’m just overthinking this and I shoud just define an app that loads everything and that handles the proper format conversions by a specific argument ?

4

First off, I think I’m getting thrown off by the language here. Just so we’re on the same page:

  • Julia modules have a PascalCase naming convention, while command line interfaces have a kebab-case naming convention, though multiple words and thus hyphens are often avoided. So when I’m reading FooBarApp or FooBarConvert, my instinct is to treat it as an underlying Julia module, not the app based on it.
  • To be perfectly clear, Julia packages can import and load each other, and extensions are automatically loaded in environments if their set of packages were loaded. Currently, Pkg Apps are specified in a Julia package based on 1) the package or a submodule, and 2) default julia process flags. Strictly speaking, an App doesn’t directly load other Julia packages, its underlying Julia module does, though we can speak more loosely.

Back to the topic at hand, a Pkg App specified in the package Foo can only involve conversion with Bar or Baz if Foo also imports them. That clearly contradicts Foo, Bar, and Baz being independent dependencies with all the conversions implemented in extensions. I can’t presume what you need, but your ideas sound correct if they involve making derivative packages.

1 Like
5

just a note that you can have multiple packages in one repo, in case that’s helpful

6

So if I put things in a very concrete example from the documentation I could do

Case 1 : one app that loads everything (even if not used)

# src/FooConvertApp.jl
module FooConvertApp

using Foo
using Bar # Triggers loading of FooBarExt
using Baz # Triggers loading of FooBazExt

function (@main)(ARGS)
    try 
        bar2foo(ARGS[1],ARGS[2])
        return 0
    catch ex
        return 1
    end
end

end # module

# Project.toml

# standard fields here

[apps]
foo-convert = {}

And then call

foo-convert file.bar file.foo

This app contains all dependencies and loads everything, so might lead to increased TTFX.

Case 2 : create a submodule per conversion and an associated app

# src/FooConvertApp.jl
module FooConvertApp

using Foo
include("src/FooBar.jl")
include("src/FooBaz.jl")

end # module

# src/FooBar.jl
module FooBar
using Bar # Triggers loading of FooBarExt

function (@main)(ARGS)
    try 
        bar2foo(ARGS[1],ARGS[2])
        return 0
    catch ex
        return 1
    end
end

end # submodule

# Project.toml

# standard fields here

[apps]
foobar-convert = { submodule = "FooBar"}
foobaz-convert = { submodule = "FooBaz"}

And then call

foobar-convert file.bar file.foo

I was not sure if include("src/FooBaz.jl") was not making this completely irrelevant, but I just found conditional submodule loading · Issue #2536 · JuliaLang/julia · GitHub that IIUC say exactly that organizing things this way avoids to load unnecessary sumbodules.

If this works, it should reduce TTFX as it should avoid loading unnecessary packages

Case 3 : repeat case 1 for each Foo / BaX couple

Is this clearer or am I completely misunderstanding how this works ?

7

Yes, thanks for the heads-up.
I know I can have in general multiple packages in a single repo using the subdir option (I think), although I never tried it.
I’m mostly developing them on a Gitlab self-hosted instance and for some reason I can’t have my CI testing and deploying multiple documentations in the same repo.
It’s probably a lack of skill on my part or maybe a limitation of Gitlab, but thanks anyway, maybe I should retry that as well.

8

In Case 2, every app loads the same package and all of its submodules. There’s currently no mechanism for only loading select submodules of a Julia package. I don’t know if that is impossible, but it’s a bit harder for a language where submodules are specified in the parent module’s expression than a language where a parent module doesn’t need to explicitly reference submodules.

Case 1 or Case 3 seems more feasible, but again, I’m not sure what you really need. I can say people have never really shied away from monolithic command-line tools; if we really needed the granularity, we’d write something and use the libraries ourselves.

9

All right, thanks for the help. I’ll attempt Case 1 then!