Having a submodule import a top-level module

jonathan-laurent · October 13, 2020, 12:15am

After experimenting a bit with the module system, I found that it was possible to write code as the following:

module TestPackage
  export x, y
  x = 0
  module Sub
    using TestPackage
    z = x
    # z = y # error!
  end
  y = 1
end

Here, when I do using TestPackage in Sub, it seems to be the case that all exported names that are defined before Sub in TestPackage are brought into scope. Therefore, in this case, x is accessible in Sub but y is not.

Is this behavior specified somewhere, or am I relying on some internal implementation details that might break someday here?

In fact, I suspect that the pattern I described is already broken. Although it seems to work well on small examples, I attempted to refactor my AlphaZero.jl package this way and then started to get weird segfaults.

Any thoughts?

A segfault example:

julia> using Revise; include("test/runtests.jl")
[ Info: Precompiling AlphaZero [8ed9eb0b-7496-408d-8c8b-2119aeea02cd]
[ Info: Using the Flux implementation of AlphaZero.NetLib.

signal (11): Segmentation fault
in expression starting at /home/jonathan/AlphaZero.jl/test/runtests.jl:1
jl_deserialize_value_array at /buildworker/worker/package_linux64/build/src/dump.c:1626
jl_deserialize_value at /buildworker/worker/package_linux64/build/src/dump.c:2139
...
...
...
jl_deserialize_value_array at /buildworker/worker/package_linux64/build/src/dump.c:1620
jl_deserialize_value at /buildworker/worker/package_linux64/build/src/dump.c:2139
_jl_restore_incremental at /buildworker/worker/package_linux64/build/src/dump.c:3231
jl_restore_incremental at /buildworker/worker/package_linux64/build/src/dump.c:3299
_include_from_serialized at ./loading.jl:681
_require_from_serialized at ./loading.jl:749
_require at ./loading.jl:1040
require at ./loading.jl:928
require at ./loading.jl:923
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2214 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2398
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1690 [inlined]
call_require at /buildworker/worker/package_linux64/build/src/toplevel.c:425 [inlined]
eval_import_path at /buildworker/worker/package_linux64/build/src/toplevel.c:462
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:682
jl_parse_eval_all at /buildworker/worker/package_linux64/build/src/ast.c:913
jl_load_rewrite at /buildworker/worker/package_linux64/build/src/toplevel.c:914
include at ./client.jl:457
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2231 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2398
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1690 [inlined]
do_call at /buildworker/worker/package_linux64/build/src/interpreter.c:117
eval_value at /buildworker/worker/package_linux64/build/src/interpreter.c:206
eval_stmt_value at /buildworker/worker/package_linux64/build/src/interpreter.c:157 [inlined]
eval_body at /buildworker/worker/package_linux64/build/src/interpreter.c:566
jl_interpret_toplevel_thunk at /buildworker/worker/package_linux64/build/src/interpreter.c:660
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:840
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:790
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:790
jl_toplevel_eval_in at /buildworker/worker/package_linux64/build/src/toplevel.c:883
eval at ./boot.jl:331
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2214 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2398
eval_user_input at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/REPL/src/REPL.jl:134
repl_backend_loop at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/REPL/src/REPL.jl:195
start_repl_backend at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/REPL/src/REPL.jl:180
#run_repl#37 at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/REPL/src/REPL.jl:292
run_repl at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/REPL/src/REPL.jl:288
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2231 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2398
#807 at ./client.jl:399
jfptr_YY.807_64838.clone_1 at /home/jonathan/Software/julia-1.5.2/lib/julia/sys.so (unknown line)
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2214 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2398
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1690 [inlined]
do_apply at /buildworker/worker/package_linux64/build/src/builtins.c:655
jl_f__apply_latest at /buildworker/worker/package_linux64/build/src/builtins.c:705
#invokelatest#1 at ./essentials.jl:710 [inlined]
invokelatest at ./essentials.jl:709 [inlined]
run_main_repl at ./client.jl:383
exec_options at ./client.jl:313
_start at ./client.jl:506
jfptr__start_60376.clone_1 at /home/jonathan/Software/julia-1.5.2/lib/julia/sys.so (unknown line)
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2214 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2398
unknown function (ip: 0x401931)
unknown function (ip: 0x401533)
__libc_start_main at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: 0x4015d4)
Allocations: 83940114 (Pool: 83915894; Big: 24220); GC: 54
fish: “julia --project” terminated by signal SIGSEGV (Address boundary error)

Tamas_Papp · October 13, 2020, 11:22am

I would recommend

module TestPackage
  export x, y
  x = 0
  y = 1
  module Sub
  using ..TestPackage
  z = x
  z = y
  end
end

Note the order, and the .. in using. This is documented here:

https://docs.julialang.org/en/v1/manual/modules/#Relative-and-absolute-module-paths

I am guessing that your code loads TestPackage from somewhere else using the code loading mechanism.

Regarding the segfault: if you can reproduce it on master, check existing issues and if this is a new one, please open an issue.

jonathan-laurent · October 13, 2020, 2:20pm

Replacing using TestPackage by using ..TestPackage does not seem to change anything on Julia 1.5.2.

However, I tested my code on Julia 1.6-DEV and I don’t have a segfault anymore.

I read this page and I did not see this behavior documented explicitly. Examples are given of using relative paths to import individual names from a parent module or all exported names from a sibling module.

But I have never seen the case where a sub-module imports a whole parent module and it is not clear to me what the semantics of such an import should be or whether it should be allowed. The behavior I am observing is that all exported names that are defined before Sub are brought into scope but I would feel better if this was mentioned explicitly in the documentation.

Edit: I just posted an issue to ask about this.

Tamas_Papp · November 2, 2020, 11:34am

This PR should fix that:

github.com/JuliaLang/julia

Reorganize the Modules chapter of the manual.

JuliaLang:master ← tpapp:tp/modules-usage

opened 11:16AM - 02 Nov 20 UTC

tpapp

+287 -126

# Motivation Many users, especially from languages which either don't have na…mespaces or do not use them widely, are confused by various aspects of Julia's namespace management. This is a reorganization (and partial rewrite) or the relevant chapter, with the intention that it should tide us over until (if) breaking changes like #8000 happen. I have searched the issues and the Discourse forum for related topics and tried to address every issue that comes up in a concise but occasionally introductory/pedagogical manner, going into details and corner cases when relevant after conveying the main idea. Each particular change/addition was motivated by people running into problems with some feature. I combined and modified examples from various contributors, and only reference the original when I use it unchanged. Since this is a substantial rewrite, thorough reviews would be appreciated. If, despite best effort, I missed some commonly used scenarios or made mistakes, please comment and I am happy to address them. # Main changes ## Namespaces, `export`, `import`, `using` - explain namespaces, qualified symbol names, and export lists - consolidate examples where applicable - fix examples which were outdated after #25306 - distribute the “Namespace miscellanea” section into parts in the appropriate sections - break up the summary table and explain each case in detail with examples - subsection on various strategies to handle symbol conflicts - use “bring into the namespace/scope” instead of “import” in the text, as readers confuse it with `import` - explicitly document what happens with multiple `using` / `import` statements - add relevant style / best practices suggestions which are now widely used in the Julia ecosystem ## Submodules - discuss usage of `using ParentModule.SubModule` etc (relies on code loading) - add submodule examples where order matters, fixes #38011 (also another example by @rdeits [from Discourse](https://discourse.julialang.org/t/problem-with-using-in-submodules/42321/2)) - mention that submodules do not “inherit” scope from parent # Misc incidental changes - in the Methods chapter, add a note about adding methods to functions in other modules - add a markdown id to the code-loading chapter The “Module initialization and precompilation” section was reviewed, but is left unchanged.