# Is there a way to make a submodule "open"
**URL:** https://discourse.julialang.org/t/is-there-a-way-to-make-a-submodule-open/49641
**Category:** General Usage
**Tags:** module, scope
**Created:** [November 5, 2020, 8:58pm UTC](https://discourse.julialang.org/t/is-there-a-way-to-make-a-submodule-open/49641 "2020-11-05T20:58:46Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@mhinsch](https://discourse.julialang.org/u/mhinsch)
#### Post date: [November 5, 2020, 8:58pm UTC](https://discourse.julialang.org/t/is-there-a-way-to-make-a-submodule-open/49641/1 "2020-11-05T20:58:46Z")
The scenario: I am writing a DSLish set of macros that create a few functions that again call code the user has provided verbatim (in the macro call), so something like this:
```nohighlight
@mymacro Bla
myfun1()
myfun2()
end
```
Which will create, say `Blafunc1` and `Blafunc2` which again call `myfunc1` and `myfunc2`.
Now, all of this works quite nicely. However I find it extremely unsatisfying that my macro clutters the current module’s namespace with new functions with ugly names. My solution would be to let `@mymacro` create a submodule (`Bla` in this case) that contains the generated functions which I can then call as e.g. `Bla.func1()`.
All of that works save for one issue which is that I can’t find a _general_ mechanism to make symbols from the surrounding module known within the newly generated module (`Bla`). I can add a `using`/`import` with the parent module name as argument to the generated module, however if the user-supplied functions haven’t been exported they will still not be visible.
So, my question is if there is any way to open the generated submodule’s scope to let it include all symbols in the parent module?
I can think of a few alternative solutions such as parsing the macro arguments for symbols and adding includes for all of them or requiring users to fully qualify all names, but none of them make me happy TBH.
---
### Author:  [@Mason](https://discourse.julialang.org/u/Mason)
#### Post date: [November 5, 2020, 9:09pm UTC](https://discourse.julialang.org/t/is-there-a-way-to-make-a-submodule-open/49641/2 "2020-11-05T21:09:14Z")
I have a little package called [StaticModules.jl](https://github.com/MasonProtter/StaticModules.jl) that _may_ be appropriate for this, depending on how many things you want to put in your module `Bla`. Here’s a little example of it in action with your problem:
```julia
julia> f(x) = x + 1
f (generic function with 1 method)
julia> g(x) = x - 1
g (generic function with 1 method)
julia> using StaticModules
julia> @staticmodule Bla begin
myfun1() = f(1) - g(1)
myfun2() = f(1) + g(1)
end
StaticModule Bla containing
myfun1 = myfun1
myfun2 = myfun2
julia> Bla.myfun1()
2
julia> @with Bla begin
myfun2() + 1
end
3
```
If you want the namespacing behaviour, but in a real module rather than a static module, then you might be able to repurpose the code in [https://github.com/MasonProtter/StaticModules.jl/blob/master/src/StaticModules.jl](https://github.com/MasonProtter/StaticModules.jl/blob/master/src/StaticModules.jl) to create a real module and automatically generate `import` statements.
---
### Author:  [@rfourquet](https://discourse.julialang.org/u/rfourquet)
#### Post date: [November 6, 2020, 7:01am UTC](https://discourse.julialang.org/t/is-there-a-way-to-make-a-submodule-open/49641/3 "2020-11-06T07:01:08Z")
What about using `gensym` for your function names? It means more ugly names but they won’t appear with tab completion for example. Or maybe having the funtions as fields of a named tuple, whose name can be gensymed (that might be similar to the StaticModule solution)
---
### Author:  [@mhinsch](https://discourse.julialang.org/u/mhinsch)
#### Post date: [November 6, 2020, 11:46am UTC](https://discourse.julialang.org/t/is-there-a-way-to-make-a-submodule-open/49641/4 "2020-11-06T11:46:36Z")
Thanks, that’s a really cool idea.
For my purposes the named tuple solution feels a bit too hacky, though, TBH (not to mention that the `@with` syntax is a bit inconvenient).
In terms of generating the import statements - I’m not sure I fully understand your code, but I take it `get_outers` et al parse the AST for non-local symbols, right? I’ve thought about something similar, but this seems to be rather heavy machinery to accomplish something ultimately quite simple.
In any case posting the question and reading the suggestions so far made me realise that maybe I’m on the wrong track anyway.
---
### Author:  [@mhinsch](https://discourse.julialang.org/u/mhinsch)
#### Post date: [November 6, 2020, 11:49am UTC](https://discourse.julialang.org/t/is-there-a-way-to-make-a-submodule-open/49641/5 "2020-11-06T11:49:32Z")
> [@rfourquet](#):
>
> What about using `gensym` for your function names? It means more ugly names but they won’t appear with tab completion for example. Or maybe having the funtions as fields of a named tuple, whose name can be gensymed (that might be similar to the StaticModule solution)
I’m not sure I understand you correctly but I think that doesn’t really solve my problem as the generated functions are part of a (generated) API.
---
### Author:  [@mhinsch](https://discourse.julialang.org/u/mhinsch)
#### Post date: [November 6, 2020, 12:07pm UTC](https://discourse.julialang.org/t/is-there-a-way-to-make-a-submodule-open/49641/6 "2020-11-06T12:07:18Z")
I have thought about this after reading the suggestions and while I still think it would be nice to have some sort of “open” submodule for things like this, a more idiomatically Julian solution might be to accept the overlap in function names and maybe use traits to tag the difference between different generated versions from repeated macro calls (which might not differ in argument types otherwise).
---
### Author:  [@ffevotte](https://discourse.julialang.org/u/ffevotte)
#### Post date: [November 7, 2020, 9:34am UTC](https://discourse.julialang.org/t/is-there-a-way-to-make-a-submodule-open/49641/7 "2020-11-07T09:34:52Z")
> [@rfourquet](#):
>
> Or maybe having the functions as fields of a named tuple,
> [@mhinsch](#):
>
> I’m not sure I understand you correctly but I think that doesn’t really solve my problem as the generated functions are part of a (generated) API.
If you drop the `gensym` part, I think the suggestion was to make
```julia
@mymacro Foo begin
f(1)
g(1)
end
```
expand to the creation of a NamedTuple like
```julia
Foo = (fun1=()->f(1), fun2=()->g(1))
```
This way, your current namespace is left uncluttered: only `Foo` is added to it. And users can call `Foo.fun1()` in the same way as if it had been defined in a submodule.
This is very similar to the `StaticModule` solution, without any of the complexity associated with finding non-local symbols because it’s not really needed in this case. The following MWE illustrates this:
```julia
macro mymacro(name, block)
nt = Expr(:tuple)
counter = 0
for e in block.args
e isa LineNumberNode && continue
counter += 1
funname = Symbol("fun", counter)
push!(nt.args, :($funname = ()->$e))
end
quote
$name = $nt
end |> esc
end
```
```julia
julia> @macroexpand @mymacro Foo begin
f(1)
g(2)
end
quote
Foo = (fun1 = (()->begin
f(1)
end),
fun2 = (()->begin
g(2)
end))
end
```
```julia
julia> f(x) = 2x; g(x) = 3x;
julia> @mymacro Foo begin
f(2)
g(3)
end
(fun1 = var"#5#7"(), fun2 = var"#6#8"())
julia> Foo.fun1()
4
julia> Foo.fun2()
9
```
---
### Author:  [@mhinsch](https://discourse.julialang.org/u/mhinsch)
#### Post date: [November 8, 2020, 10:50pm UTC](https://discourse.julialang.org/t/is-there-a-way-to-make-a-submodule-open/49641/8 "2020-11-08T22:50:01Z")
Yes, I guess that would work, but I still find it somewhat aesthetically displeasing to have a tuple effectively disguise itself as a module.
One idea I had, though - and this might be what @rfourquet had in mind when talking about gensym’d function names - would be to indeed gensym function names for the generated functions but then assign or call them from functions in a generated module. That way the functions themselves will live in the surrounding namespace without having to import anything while calling the clear-named version is still prefixed with a module name.
I am still debating with myself if I prefer that solution or if my discomfort with naming conflicts is just a remnant of my OOP days and I should just let multiple dispatch (plus possibly traits) sort things out.
---
### Author:  [@mhinsch](https://discourse.julialang.org/u/mhinsch)
#### Post date: [November 9, 2020, 5:29pm UTC](https://discourse.julialang.org/t/is-there-a-way-to-make-a-submodule-open/49641/9 "2020-11-09T17:29:36Z")
It turns out there’s a rather simple solution:
Generate a module containing a noop function for each function you want to have namespaced, then afterwards add methods to these functions (using fully qualified names).
So, as a mockup example:
```nohighlight
@genmodule AName
```
would produce:
```nohighlight
module AName
function fun1() end
function fun2() end
end
function AName.fun1(x)
# do something relevant
end
function AName.fun2(y)
# do something else
end
```
Where `fun1(x)` and `fun2(y)` have full access to the outer namespace but can only be called via AName.
---
### Author:  [@ffevotte](https://discourse.julialang.org/u/ffevotte)
#### Post date: [November 10, 2020, 8:15am UTC](https://discourse.julialang.org/t/is-there-a-way-to-make-a-submodule-open/49641/10 "2020-11-10T08:15:09Z")
That is a nice solution (although I have to admit that I’m a little puzzled why it would be more “aesthetically pleasing” than other solutions here: I would see no reason why modules should be preferred to named tuples in this case… But I guess I’m more of a pragmatist than an aesthete 🙂)
In any case, you might want to declare [empty generic functions](https://docs.julialang.org/en/v1/manual/methods/#Empty-generic-functions) though, in order to avoid creating useless 0-argument methods:
```julia
module AName
function fun1 end # Note the absence of parentheses:
function fun2 end # this declares functions with no method
end
AName.fun1(x) = user_supplied_function1(x) # AName.fun1 only has this method
```
---
### Author:  [@mhinsch](https://discourse.julialang.org/u/mhinsch)
#### Post date: [November 10, 2020, 8:36am UTC](https://discourse.julialang.org/t/is-there-a-way-to-make-a-submodule-open/49641/11 "2020-11-10T08:36:39Z")
Thanks, yes, that makes sense.
---
### Author:  [@mhinsch](https://discourse.julialang.org/u/mhinsch)
#### Post date: [November 10, 2020, 10:23am UTC](https://discourse.julialang.org/t/is-there-a-way-to-make-a-submodule-open/49641/12 "2020-11-10T10:23:58Z")
Interestingly it appears empty generic functions can not be created using escaped expressions:
```nohighlight
julia> macro Bla()
fname = :myfun
:(function $(esc(fname)) end)
end
@Bla (macro with 1 method)
julia> @Bla
ERROR: syntax: malformed expression
Stacktrace:
[1] top-level scope at REPL[8]:1
julia> macro Bla()
fname = :myfun
:(function $(esc(fname))() end)
end
@Bla (macro with 1 method)
julia> @Bla
myfun (generic function with 1 method)
```
Am I missing something?
---
### Author:  [@mcabbott](https://discourse.julialang.org/u/mcabbott)
#### Post date: [November 10, 2020, 10:48am UTC](https://discourse.julialang.org/t/is-there-a-way-to-make-a-submodule-open/49641/13 "2020-11-10T10:48:11Z")
You can do this:
```julia
julia> macro Bla1()
fname = :myfun1
:(function $fname end) |> esc
end;
julia> @Bla1
myfun1 (generic function with 0 methods)
```
---
### Author:  [@mhinsch](https://discourse.julialang.org/u/mhinsch)
#### Post date: [November 10, 2020, 1:55pm UTC](https://discourse.julialang.org/t/is-there-a-way-to-make-a-submodule-open/49641/14 "2020-11-10T13:55:09Z")
That works, thanks.
---
### Author:  [@c42f](https://discourse.julialang.org/u/c42f)
#### Post date: [November 10, 2020, 10:50pm UTC](https://discourse.julialang.org/t/is-there-a-way-to-make-a-submodule-open/49641/15 "2020-11-10T22:50:07Z")
So you want the methods to see variables from the local scope, while the functions they belong to aren’t visible as names in the current scope. What you’ve done seems like a workable and reasonable solution for that.
> [@mhinsch](#):
>
> still find it somewhat aesthetically displeasing to have a tuple effectively disguise itself as a module.
This is interesting. Can you pinpoint your discomfort with using a named tuple over a module?
Internally a `Module` is a dictionary-like object with
- Name, Parent module
- Dynamic variable bindings (affected by `eval()` and `using` etc).
- Hints for the compiler (optimization level, etc)
But in this case you’re basically not using all these things because you’re using the current module for evaluating the method bodies. From an internals perspective a named tuple seems to give the right thing (effectively a namespace) without providing the unused parts.
Perhaps the discomfort arises because a module is generally used as a container for code (even though code-is-data, and modules can contain non-code bindings). So seeing something of type `Module` brings certain expectations?
> [@mhinsch](#):
>
> it appears empty generic functions can not be created using escaped expressions
I think you’ve found a compiler bug, could you report it on github?
---
### Author:  [@mhinsch](https://discourse.julialang.org/u/mhinsch)
#### Post date: [November 10, 2020, 11:20pm UTC](https://discourse.julialang.org/t/is-there-a-way-to-make-a-submodule-open/49641/16 "2020-11-10T23:20:14Z")
> [@c42f](#):
>
> So you want the methods to see variables from the local scope, while the functions they belong to aren’t visible as names in the current scope. What you’ve done seems like a workable and reasonable solution for that.
>
> > [@mhinsch](#):
> >
> > still find it somewhat aesthetically displeasing to have a tuple effectively disguise itself as a module.
>
> This is interesting. Can you pinpoint your discomfort with using a named tuple over a module?
>
> Internally a `Module` is a dictionary-like object with
>
> - Name, Parent module
> - Dynamic variable bindings (affected by `eval()` and `using` etc).
> - Hints for the compiler (optimization level, etc)
>
> But in this case you’re basically not using all these things because you’re using the current module for evaluating the method bodies. From an internals perspective a named tuple seems to give the right thing (effectively a namespace) without providing the unused parts.
>
> Perhaps the discomfort arises because a module is generally used as a container for code (even though code-is-data, and modules can contain non-code bindings). So seeing something of type `Module` brings certain expectations?
Aesthetically, I think the main point for me is that the function of a module is specifically to contain code in a separate namespace, which is exactly the functionality I need. Semantically I _want_ a module. Of course I could fake all the functionality I need with a named tuple, but it just doesn’t seem right.
Besides, I don’t know how the compiler and the language are going to develop down the line, so it seems prudent to use a construct that matches the desired functionality as closely as possible.
The other thing is that I might need additional module functionality later on (init function, local variables, selective exports, etc.).
> [@](#):
>
> > [@mhinsch](#):
> >
> > it appears empty generic functions can not be created using escaped expressions
>
> I think you’ve found a compiler bug, could you report it on github?
Will do.
---
### Author:  [@Mason](https://discourse.julialang.org/u/Mason)
#### Post date: [November 11, 2020, 12:31am UTC](https://discourse.julialang.org/t/is-there-a-way-to-make-a-submodule-open/49641/17 "2020-11-11T00:31:25Z")
> [@mhinsch](#):
>
> Aesthetically, I think the main point for me is that the function of a module is specifically to contain code in a separate namespace, which is exactly the functionality I need. Semantically I _want_ a module. Of course I could fake all the functionality I need with a named tuple, but it just doesn’t seem right.
>
> Besides, I don’t know how the compiler and the language are going to develop down the line, so it seems prudent to use a construct that matches the desired functionality as closely as possible.
I’ll just chime in and agree with @c42f that there’s not really anything inappropriate about a named tuple here. Holding code is just as appropriate as holding data because code is data.
I also think that you may find that julia’s modules have their own warts and quirks that _could_ make them not so great for your use cases as well. E.g. you can’t create a module in a local scope.
---
### Author:  [@c42f](https://discourse.julialang.org/u/c42f)
#### Post date: [November 11, 2020, 1:32am UTC](https://discourse.julialang.org/t/is-there-a-way-to-make-a-submodule-open/49641/18 "2020-11-11T01:32:50Z")
> [@mhinsch](#):
>
> > [@](#):
> >
> > I think you’ve found a compiler bug, could you report it on github?
>
> Will do.
Thanks, much appreciated.
> [@mhinsch](#):
>
> I think the main point for me is that the function of a module is specifically to contain code in a separate namespace, which is exactly the functionality I need
But the funny thing is that you’ve subverted most of the namespacing functionality. To demonstrate:
```julia
julia> module Bla
function func1 end
end
Main.Bla
julia> function Bla.func1()
return "hi"
end
julia> parentmodule(Bla.func1)
Main.Bla
julia> methods(Bla.func1)
# 1 method for generic function "func1":
[1] func1() in Main at REPL[25]:1
julia> first(methods(Bla.func1)).module
Main
```
So while the _function_ `Bla.func1` lives in the `Bla` module, its _only implementation method_ lives in `Main`. And of course this is correct for your use case, because you want the implementation to see all the names in the scope of `@mymacro`.
So nearly all of the `Module` functionalities are being actively subverted here which is why I think this is quite an entertaining use of modules. (Not wrong mind you — I really don’t think it’s that harmful either. Just kind of amusing 🙂 )
---
### Author:  [@c42f](https://discourse.julialang.org/u/c42f)
#### Post date: [November 11, 2020, 1:36am UTC](https://discourse.julialang.org/t/is-there-a-way-to-make-a-submodule-open/49641/19 "2020-11-11T01:36:08Z")
> [@Mason](#):
>
> E.g. you can’t create a module in a local scope.
This is an interesting point in its own right. If you return a named tuple, it’d be possible for the whole thing to work fine in local scope. (Where `func1` and `Blafunc1` may well be closures rather than normal generic functions.) That might not be a use case you care about, however.
---
### Author:  [@mhinsch](https://discourse.julialang.org/u/mhinsch)
#### Post date: [November 11, 2020, 11:44am UTC](https://discourse.julialang.org/t/is-there-a-way-to-make-a-submodule-open/49641/20 "2020-11-11T11:44:34Z")
This discussion is getting slightly esoteric, but oh well…
I think everybody is making some valid points, but after a bit of thought I think I can say exactly why I don’t like the named tuple solution. There are two reasons:
From a practical point of view a module is preferable as I might need more module functionality (beyond namespacing) down the line. For context: I am writing a set of DSL-like macros for event-based simulations (you can see them in action [here](https://github.com/mhinsch/JuliaABMtemplate/blob/master/model.jl)). At this point the macros only create the functions that take care of condition testing and rescheduling, but I might want to keep track of some state as well and add a few service functions which again might require imports and exports, etc. At that point having a module would make things easier and tidier.
More fundamentally though, what the macros produce has the _syntax_ of a module. Therefore I expect users will treat it as such and expect module behaviour in all aspects. If I produce a named tuple the difference will leak at some point, confusing users.
The alternative would be to be upfront about it and say “look, this might look and feel like a module, but in fact it’s a named tuple containing functions, so treat it as such”. That would be a valid approach I guess.
However, I think macros increase the mental load on the user anyway as they effectively modify the language itself. So, for each set of macros one has to maintain an internal model of a) how it is used and b) how the code it produces behaves. Many macros are essentially just (very) glorified syntactic sugar, i.e. in principle you could write their results by hand. Part b) becomes much easier if the macro produces code that is similar to what you would end up with if you didn’t have the macros. In my case in absence of my macros I would find it a very reasonable design choice to wrap a simulation in its own module. I would not, however, store the infrastructure functions in a named tuple. This makes modules the lower-friction choice.
All of that said (rather long-windedly), I guess in the end it’s largely a matter of taste which solution you prefer.
[Next page](https://discourse.julialang.org/t/is-there-a-way-to-make-a-submodule-open/49641.md?page=2)