Will juliac solve the relocatabiliy issue?

nhz2 · August 23, 2025, 2:47pm

In the case of Oxygen.jl the use of @__DIR__ here is very suspect.

src/constants.jl

84eb8971d


      
          # Generate a reliable path to our package directory
          const PACKAGE_DIR :: String = @path @__DIR__
          
          # Generate a reliable path to our internal data folder that works when the 
          # package is used with PackageCompiler.jl
          const DATA_PATH :: String = @path joinpath(@__DIR__, "..", "data")

The other thing that makes this hard is that the builder paths could just be used to print debug messages, and not actually accessed. I think that is what is going on in GLMakie with this name ShaderSource field (but hard to tell for sure)

github.com/MakieOrg/Makie.jl

GLMakie/src/GLMakie.jl

687c4466c


      
                  @eval export $(name)
              end
          end
          
          import ShaderAbstractions: Sampler, Buffer
          export Sampler, Buffer
          
          struct ShaderSource
              typ::GLenum
              source::String
              name::String
          end
          
          function ShaderSource(path)
              typ = GLAbstraction.shadertype(splitext(path)[2])
              source = read(path, String)
              name = String(path)
              return ShaderSource(typ, source, name)
          end
          
          const SHADER_DIR = normpath(joinpath(@__DIR__, "..", "assets", "shader"))

I found these by searching for @__DIR__ and @path because these seem to be very easy macros to misuse.

ohmsweetohm1 · August 23, 2025, 3:19pm

Yes, there is an open issue

github.com/OxygenFramework/Oxygen.jl

Relocatability: Files in data directory not found by App

opened 03:15PM - 15 Apr 25 UTC

divbyzerofordummies

I have created an App using Oxygen (using `PackageCompiler`) and have put it int…o a Docker image (a multi-stage build where I only copied the contents of the App folder to the final image). The app per se works fine, I can call my get and post endpoints. However, when I go to `127.0.0.1:8980/docs`, the container throws an error ``` SystemError: opening file "/home/jl/.julia/packages/Oxygen/jV71k/data/swagger@5.7.2/swagger-ui-bundle.js": No such file or directory ``` Since the `.julia/packages` directory no longer exists, this is no big surprise. The same problem occurs if I copy the app and move it to another computer without a Julia installation. I don't quite understand why this is the case since you are using `RelocatableFolders`; somehow the scratch space is not generated? Julia: Version 1.11.3 Oxygen: Version 1.7.1 ``` [ Info: Listening on: 0.0.0.0:8980, thread id: 1 ┌ Error: handle_connection handler error. │ │ =========================== │ HTTP Error message: │ │ ERROR: TaskFailedException │ Stacktrace: │ [1] wait(?::Task) │ @ Base ./task.jl:370 │ [2] (::Oxygen.Core.var"#18#21"{Oxygen.Core.var"#16#17"{Oxygen.Core.var"#30#32"{Oxygen.Core.var"#35#38"{HTTP.Handlers.Router{typeof(HTTP.Handlers.default404), typeof(HTTP.Handlers.default405), Nothing}, Bool, Bool}, HTTP.Handlers.Router{typeof(HTTP.Handlers.default404), typeof(HTTP.Handlers.default405), Nothing}, String}}})(?::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}) │ @ Oxygen.Core ~/.julia/packages/Oxygen/jV71k/src/core.jl:289 │ [3] #invokelatest#2 │ @ ./essentials.jl:1055 [inlined] │ [4] invokelatest │ @ ./essentials.jl:1052 [inlined] │ [5] handle_connection(?::Function, ?::HTTP.Connections.Connection{Sockets.TCPSocket}, ?::HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, ?::Int64, ?::Function, ?::Bool) │ @ HTTP.Servers ~/.julia/packages/HTTP/MIUdD/src/Servers.jl:469 │ [6] (::HTTP.Servers.var"#16#17"{Oxygen.Core.var"#18#21"{Oxygen.Core.var"#16#17"{Oxygen.Core.var"#30#32"{Oxygen.Core.var"#35#38"{HTTP.Handlers.Router{typeof(HTTP.Handlers.default404), typeof(HTTP.Handlers.default405), Nothing}, Bool, Bool}, HTTP.Handlers.Router{typeof(HTTP.Handlers.default404), typeof(HTTP.Handlers.default405), Nothing}, String}}}, HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, Set{HTTP.Connections.Connection}, Int64, Oxygen.Core.var"#26#27", ReentrantLock, Bool, Base.Semaphore, HTTP.Connections.Connection{Sockets.TCPSocket}})() │ @ HTTP.Servers ~/.julia/packages/HTTP/MIUdD/src/Servers.jl:401 │ │ nested task error: TaskFailedException │ Stacktrace: │ [1] wait(?::Task) │ @ Base ./task.jl:370 │ [2] (::Oxygen.Core.var"#19#22"{HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, Oxygen.Core.var"#16#17"{Oxygen.Core.var"#30#32"{Oxygen.Core.var"#35#38"{HTTP.Handlers.Router{typeof(HTTP.Handlers.default404), typeof(HTTP.Handlers.default405), Nothing}, Bool, Bool}, HTTP.Handlers.Router{typeof(HTTP.Handlers.default404), typeof(HTTP.Handlers.default405), Nothing}, String}}})() │ @ Oxygen.Core ~/.julia/packages/Oxygen/jV71k/src/core.jl:287 │ │ nested task error: SystemError: opening file "/home/jl/.julia/packages/Oxygen/jV71k/data/swagger@5.7.2/swagger-ui-bundle.js": No such file or directory │ Stacktrace: │ [1] systemerror(?::String, ?::Int32; ?::Nothing) │ @ Base ./error.jl:176 │ [2] systemerror │ @ ./error.jl:175 [inlined] │ [3] open(?::String; ?::Bool, ?::Nothing, ?::Nothing, ?::Nothing, ?::Nothing, ?::Nothing) │ @ Base ./iostream.jl:295 │ [4] open │ @ ./iostream.jl:277 [inlined] │ [5] open(?::Base.var"#472#473"{String}, ?::String; ?::@Kwargs{}) │ @ Base ./io.jl:408 │ [6] swaggerhtml(?::String, ?::String) │ @ Oxygen.Core.AutoDoc ./<missing>:0 │ [7] (::Oxygen.Core.var"#65#67"{Oxygen.Core.var"#69#73"{String, String}, Oxygen.Core.Handlers.var"#26#28"{Oxygen.Core.Handlers.var"#26#27#29"{Oxygen.Core.Handlers.var"#4#16"}}})(?::HTTP.Messages.Request) │ @ Oxygen.Core ./<missing>:0 │ [8] (::HTTP.Handlers.Router{typeof(HTTP.Handlers.default404), typeof(HTTP.Handlers.default405), Nothing})(?::HTTP.Messages.Request) │ @ HTTP.Handlers ~/.julia/packages/HTTP/MIUdD/src/Handlers.jl:439 │ [9] (::Oxygen.Core.var"#30#32"{Oxygen.Core.var"#35#38"{HTTP.Handlers.Router{typeof(HTTP.Handlers.default404), typeof(HTTP.Handlers.default405), Nothing}, Bool, Bool}, HTTP.Handlers.Router{typeof(HTTP.Handlers.default404), typeof(HTTP.Handlers.default405), Nothing}, String})(?::HTTP.Messages.Request) │ @ Oxygen.Core ~/.julia/packages/Oxygen/jV71k/src/core.jl:397 │ [10] #13 │ @ ~/.julia/packages/Oxygen/jV71k/src/core.jl:255 [inlined] │ [11] (::HTTP.Handlers.var"#1#2"{Oxygen.Core.var"#13#15"{Oxygen.Core.var"#30#32"{Oxygen.Core.var"#35#38"{HTTP.Handlers.Router{typeof(HTTP.Handlers.default404), typeof(HTTP.Handlers.default405), Nothing}, Bool, Bool}, HTTP.Handlers.Router{typeof(HTTP.Handlers.default404), typeof(HTTP.Handlers.default405), Nothing}, String}, Sockets.IPv4, HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}}})(?::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}) │ @ HTTP.Handlers ~/.julia/packages/HTTP/MIUdD/src/Handlers.jl:58 │ [12] #16 │ @ ~/.julia/packages/Oxygen/jV71k/src/core.jl:272 [inlined] │ [13] (::Oxygen.Core.var"#20#23"{HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, Oxygen.Core.var"#16#17"{Oxygen.Core.var"#30#32"{Oxygen.Core.var"#35#38"{HTTP.Handlers.Router{typeof(HTTP.Handlers.default404), typeof(HTTP.Handlers.default405), Nothing}, Bool, Bool}, HTTP.Handlers.Router{typeof(HTTP.Handlers.default404), typeof(HTTP.Handlers.default405), Nothing}, String}}})() │ @ Oxygen.Core ~/.julia/packages/Oxygen/jV71k/src/core.jl:286 │ request = │ HTTP.Messages.Request: │ """ │ GET /docs HTTP/1.1 │ Host: 127.0.0.1:8980 │ User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:137.0) Gecko/20100101 Firefox/137.0 │ Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 │ Accept-Language: de,en-US;q=0.7,en;q=0.3 │ Accept-Encoding: gzip, deflate, br, zstd │ Connection: keep-alive │ Upgrade-Insecure-Requests: 1 │ Sec-Fetch-Dest: document │ Sec-Fetch-Mode: navigate │ Sec-Fetch-Site: none │ Sec-Fetch-User: ?1 │ Priority: u=0, i │ │ """ └ @ HTTP.Servers ~/.julia/packages/HTTP/MIUdD/src/Servers.jl:483 [ Info: 2025-04-15T15:01:40+0000 - 172.17.0.1:35380 - "GET /docs HTTP/1.1" 500 ```

nhz2 · August 23, 2025, 4:43pm

I think it would be cool to have the Julia equivalent of

Basically a very restrictive api that forces the code to explicitly answer what, how, and when user data or app data should be accessed, to make these issues really hard to run into.

One issue with @path from RelocatableFolders.jl is that it tries to hide things behind a macro that have to be explicitly decided on by the developer for things to work well.

nsajko · August 23, 2025, 6:07pm

Also pathof and pkgdir.

These being available as API in the first place is perhaps the root of the problem here. Probably someone should make a doc PR to discourage their use outside debugging and perhaps deprecate them.

EDIT: found relevant discussion, see PRs linked here:

github.com/JuliaLang/julia

Some uses of `@DIR` should be relocatable.

opened 02:01PM - 10 May 24 UTC

gbaraldi

~The proposed use I have for this is for when packages are looking for their Art…ifacts.toml file, they have to annoying things, and non relocatable ones even more annyoingly. (Maybe we need to add something like `@__PKGDIR` that does the relocatable thing)~ Currently `@__DIR__` bakes in the current path of the whenever it's called, but for this it's usually the path of the `src` directory of a package. Examples of this are the `Artifacts.toml` file or a `examples/`directory, The issue arises if one tries to move that package to another place, either via PkgCompiler or via copying the depot around. Those baked in absolute paths mean that it's not safe for this package to be relocatable. I don't have a clear proposal for this, but I imagine it would just be a expansion of the relocatable package image work done by @fatteneder, specially since if you get the package those files are already there, we just don't have a nice way of getting to them.

ohmsweetohm1 · August 23, 2025, 6:36pm

There is a Base.isrelocatable(pkg)

github.com/JuliaLang/julia

base/loading.jl

fc1fb4139


      
          
          """
              Base.isrelocatable(pkg::PkgId)
          
          Returns whether a given PkgId within the active project is precompiled and the
          associated cache is relocatable.
          
          !!! compat "Julia 1.11"
              This function requires at least Julia 1.11.
          """
          function isrelocatable(pkg::PkgId)
              path = compilecache_path(pkg)
              isnothing(path) && return false
              io = open(path, "r")
              try
                  iszero(isvalid_cache_header(io)) && throw(ArgumentError("Incompatible header in cache file $cachefile."))
                  _, (includes, includes_srcfiles, _), _... = _parse_cache_header(io, path)
                  for inc in includes
                      !startswith(inc.filename, "@depot") && return false
                      if inc ∉ includes_srcfiles
                          # its an include_dependency

nhz2 · August 23, 2025, 8:40pm

Maybe there is a flaw in this test, but pkgdir seems to relocate as expected.

$ rm -rf /tmp/testpkgdir
$ JULIA_DEPOT_PATH="/tmp/testpkgdir/foo:" julia -e 'using Pkg; Pkg.add("Example")'
  Installing known registries into `/tmp/testpkgdir/foo`
       Added `General` registry to /tmp/testpkgdir/foo/registries
    Updating registry at `/tmp/testpkgdir/foo/registries/General.toml`
   Resolving package versions...
   Installed Example ─ v0.5.5
    Updating `/tmp/testpkgdir/foo/environments/v1.11/Project.toml`
  [7876af07] + Example v0.5.5
    Updating `/tmp/testpkgdir/foo/environments/v1.11/Manifest.toml`
  [7876af07] + Example v0.5.5
Precompiling project...
  1 dependency successfully precompiled in 0 seconds
$ JULIA_DEPOT_PATH="/tmp/testpkgdir/foo:" julia -e 'using Example; println(pkgdir(Example))'
/tmp/testpkgdir/foo/packages/Example/SUIr0
$ mv /tmp/testpkgdir/foo /tmp/testpkgdir/bar
$ JULIA_DEPOT_PATH="/tmp/testpkgdir/bar:" julia -e 'using Example; println(pkgdir(Example))'
/tmp/testpkgdir/bar/packages/Example/SUIr0

Benny · August 23, 2025, 9:50pm

I think that’s just for making precompile caches relocatable at all, which could potentially let us install precompile caches and seems relevant to executables and libraries as well. It still can’t catch a program trying to read from a bad path, install missing dependencies, or do other undesirable behaviors at runtime.

Julia 1.11 Highlights - Precompile file relocatability

PSA: Cache relocation with Julia v1.11 - HackMD

Make precompile files relocatable/servable · Issue #47943 · JuliaLang/julia

There could be some static checks, but again, that only goes so far against bad practices. I have no idea how feasible it is, but let’s say we do have a (pre)compiler that identifies every instantiated file path tied to the host machine and warns if we ever use them via the end-product’s methods (we’d have to omit methods that only serve their purpose during precompilation). I release a product that passed that check, but you notice that it doesn’t work on your machine, or worse it fails sometimes for no clear reason. Turns out that while all the precompile-instantiated paths were relocatable, one of them had a file that stored other paths that would only be instantiated at runtime or be used to generate new paths at runtime, possibly only on some runs. I failed to make those relocatable, and a compiler can’t catch runtime paths it can’t be aware of. At some point, good tooling and instructions should be discoverable, and people need to opt in. Just last week, I ran into a prototype package that required end-users to separately install a shared library and store its path to an environment variable (which might be isolated to a shell session), then it searched that path in the global scope instead of __init__, which usually forces the use of dev instead of add; obviously not ideal, but making users separately install things and tweak settings isn’t that rare of a practice generally, and it needs to be deliberately improved or avoided.

It’s also worth keeping in mind that manually written paths are not the whole story for relocatability. The precompile cache relocatability improvements involved precompilation-generated paths, which can be statically checked but isn’t something we were involved in or can control. Memory addresses need to be handled at runtime to not get stuck at a state only suitable for the host machine at precompile-time. The package could be getting data from URLs that are blocked on some networks. There might be more things that I’m just not aware of.

So RelocatableFolders.@path falling back to a scratchspace at runtime isn’t the straightforward solution?

nhz2 · August 23, 2025, 10:39pm

I find baking in the data with the read function more straightforward:

github.com/ASML-Labs/PPTX.jl

src/constants.jl

acc2cf70f


      
          include_dependency(joinpath(TEMPLATE_DIR, "tableStyles.xml"))
          const DEFAULT_TABLE_STYLE_DATA = read(joinpath(TEMPLATE_DIR, "tableStyles.xml"))

@path if I understand it correctly also bakes in the file data, and then when converted to a String writes the files somewhere to get a String path. This seems like a worse version of Artifacts

nhz2 · August 24, 2025, 6:13pm

I created this script to check for relocation issues: Julia Package Relocatability Test Script · GitHub

When I run it on normal packages like julia reloc-helper.jl JSON everything works.

When run on julia reloc-helper.jl Oxygen or julia reloc-helper.jl GLMakie it shows a stacktrace with the relocation issue:

     Testing Running tests...
ERROR: LoadError: SystemError: opening file "/tmp/jl_29ZxGi/build/dev/Oxygen/src/methods.jl": No such file or directory
Stacktrace:
 [1] include(mod::Module, _path::String)
   @ Base ./Base.jl:562
 [2] include(x::String)
   @ Main.RunTests.TimeZoneTests /tmp/jl_29ZxGi/host/dev/Oxygen/test/extensions/timezonetests.jl:1
 [3] top-level scope
   @ /tmp/jl_29ZxGi/build/dev/Oxygen/src/Oxygen.jl:32
 [4] include(mod::Module, _path::String)
   @ Base ./Base.jl:562
 [5] include(x::String)
   @ Main.RunTests /tmp/jl_29ZxGi/host/dev/Oxygen/test/runtests.jl:1
 [6] top-level scope
   @ /tmp/jl_29ZxGi/host/dev/Oxygen/test/runtests.jl:8
 [7] include(fname::String)
   @ Main ./sysimg.jl:38
 [8] top-level scope
   @ none:6
in expression starting at /tmp/jl_29ZxGi/host/dev/Oxygen/test/extensions/timezonetests.jl:1
in expression starting at /tmp/jl_29ZxGi/host/dev/Oxygen/test/runtests.jl:1

shader cache: Error During Test at /tmp/jl_qTURTz/host/dev/Makie/GLMakie/test/unit_tests.jl:9
  Got exception outside of a @test
  SystemError: opening file "/tmp/jl_qTURTz/build/dev/Makie/GLMakie/assets/shader/postprocessing/fullscreen.vert": No such file or directory
  Stacktrace:
    [1] systemerror(p::String, errno::Int32; extrainfo::Nothing)
      @ Base ./error.jl:176
    [2] systemerror
      @ ./error.jl:175 [inlined]
    [3] open(fname::String; lock::Bool, read::Nothing, write::Nothing, create::Nothing, truncate::Nothing, append::Nothing)
      @ Base ./iostream.jl:295
    [4] open
      @ ./iostream.jl:277 [inlined]
    [5] #open#465
      @ ./io.jl:408
    [6] open
      @ ./io.jl:407 [inlined]
    [7] read
      @ ./io.jl:507 [inlined]
    [8] ShaderSource
      @ /tmp/jl_qTURTz/build/dev/Makie/GLMakie/src/GLMakie.jl:55 [inlined]
    [9] (::GLMakie.var"#1#2"{String})()
      @ GLMakie /tmp/jl_qTURTz/build/dev/Makie/GLMakie/src/GLMakie.jl:69
   [10] get!(default::GLMakie.var"#1#2"{String}, h::Dict{String, GLMakie.ShaderSource}, key::String)
      @ Base ./dict.jl:458
   [11] loadshader
      @ /tmp/jl_qTURTz/build/dev/Makie/GLMakie/src/GLMakie.jl:65 [inlined]
   [12] to_screen_postprocessor(framebuffer::GLMakie.GLFramebuffer, shader_cache::GLMakie.GLAbstraction.ShaderCache, screen_fb_id::Nothing)
      @ GLMakie /tmp/jl_qTURTz/build/dev/Makie/GLMakie/src/postprocessing.jl:285
   [13] to_screen_postprocessor
      @ /tmp/jl_qTURTz/build/dev/Makie/GLMakie/src/postprocessing.jl:285 [inlined]
   [14] empty_screen(debugging::Bool, reuse::Bool, window::Nothing)
      @ GLMakie /tmp/jl_qTURTz/build/dev/Makie/GLMakie/src/screen.jl:314
   [15] empty_screen(debugging::Bool; reuse::Bool, window::Nothing)
      @ GLMakie /tmp/jl_qTURTz/build/dev/Makie/GLMakie/src/screen.jl:258
   [16] screen_from_pool(debugging::Bool; window::Nothing)
      @ GLMakie /tmp/jl_qTURTz/build/dev/Makie/GLMakie/src/screen.jl:375
   [17] screen_from_pool
      @ /tmp/jl_qTURTz/build/dev/Makie/GLMakie/src/screen.jl:363 [inlined]
   [18] GLMakie.Screen(; resolution::Nothing, start_renderloop::Bool, window::Nothing, screen_config::@Kwargs{visible::Bool})
      @ GLMakie /tmp/jl_qTURTz/build/dev/Makie/GLMakie/src/screen.jl:462
   [19] macro expansion
      @ /tmp/jl_qTURTz/host/dev/Makie/GLMakie/test/unit_tests.jl:11 [inlined]
   [20] macro expansion
      @ ~/.julia/juliaup/julia-1.11.6+0.x64.linux.gnu/share/julia/stdlib/v1.11/Test/src/Test.jl:1709 [inlined]
   [21] top-level scope
      @ /tmp/jl_qTURTz/host/dev/Makie/GLMakie/test/unit_tests.jl:10
   [22] include(fname::String)
      @ Main ./sysimg.jl:38
   [23] macro expansion
      @ /tmp/jl_qTURTz/host/dev/Makie/GLMakie/test/runtests.jl:36 [inlined]
   [24] macro expansion
      @ ~/.julia/juliaup/julia-1.11.6+0.x64.linux.gnu/share/julia/stdlib/v1.11/Test/src/Test.jl:1709 [inlined]
   [25] top-level scope
      @ /tmp/jl_qTURTz/host/dev/Makie/GLMakie/test/runtests.jl:24
   [26] include(fname::String)
      @ Main ./sysimg.jl:38
   [27] top-level scope
      @ none:6

I used it to fix MeshCat.jl

nhz2 · August 24, 2025, 7:31pm

Make sure that the paths are Path objects by ValentinKaisermayer · Pull Request #267 · OxygenFramework/Oxygen.jl · GitHub fixes that Oxygen error. The next Oxygen error seems to be coming from Bonito.jl or WGLMakie.jl

Full error

WGLMakie Utils tests: Error During Test at /tmp/jl_PG0swv/host/dev/Oxygen/test/extensions/wglmakietests.jl:10
  Got exception outside of a @test
  File not found: /tmp/jl_PG0swv/build/packages/Bonito/Xtz2t/js_dependencies/Bonito.bundled.js
  Stacktrace:
    [1] error(s::String)
      @ Base ./error.jl:35
    [2] #to_data_url#98
      @ /tmp/jl_PG0swv/build/packages/Bonito/Xtz2t/src/asset-serving/asset.jl:212 [inlined]
    [3] to_data_url(file_path::String)
      @ Bonito /tmp/jl_PG0swv/build/packages/Bonito/Xtz2t/src/asset-serving/asset.jl:211
    [4] url(::Bonito.NoServer, asset::Bonito.Asset)
      @ Bonito /tmp/jl_PG0swv/build/packages/Bonito/Xtz2t/src/asset-serving/no-server.jl:48
    [5] url(session::Bonito.Session{Bonito.NoConnection}, asset::Bonito.Asset)
      @ Bonito /tmp/jl_PG0swv/build/packages/Bonito/Xtz2t/src/asset-serving/asset.jl:37
    [6] (::Bonito.var"#63#64"{Bool, Bool, String, Bonito.Session{Bonito.NoConnection}})()
      @ Bonito /tmp/jl_PG0swv/build/packages/Bonito/Xtz2t/src/session.jl:488
    [7] lock(f::Bonito.var"#63#64"{Bool, Bool, String, Bonito.Session{Bonito.NoConnection}}, l::ReentrantLock)
      @ Base ./lock.jl:232
    [8] #session_dom#62
      @ /tmp/jl_PG0swv/build/packages/Bonito/Xtz2t/src/session.jl:443 [inlined]
    [9] session_dom
      @ /tmp/jl_PG0swv/build/packages/Bonito/Xtz2t/src/session.jl:442 [inlined]
   [10] session_dom(session::Bonito.Session{Bonito.NoConnection}, app::Bonito.App; init::Bool, html_document::Bool)
      @ Bonito /tmp/jl_PG0swv/build/packages/Bonito/Xtz2t/src/session.jl:363
   [11] session_dom(session::Bonito.Session{Bonito.NoConnection}, app::Bonito.App)
      @ Bonito /tmp/jl_PG0swv/build/packages/Bonito/Xtz2t/src/session.jl:361
   [12] show_html(io::IOBuffer, app::Bonito.App; parent::Nothing)
      @ Bonito /tmp/jl_PG0swv/build/packages/Bonito/Xtz2t/src/display.jl:78
   [13] show_html(io::IOBuffer, app::Bonito.App)
      @ Bonito /tmp/jl_PG0swv/build/packages/Bonito/Xtz2t/src/display.jl:63
   [14] show
      @ /tmp/jl_PG0swv/build/packages/Bonito/Xtz2t/src/display.jl:97 [inlined]
   [15] backend_show(screen::WGLMakie.Screen, io::IOBuffer, m::MIME{Symbol("text/html")}, scene::Makie.Scene)
      @ WGLMakie /tmp/jl_PG0swv/build/packages/WGLMakie/LOBCu/src/display.jl:205
   [16] show(io::IOBuffer, m::MIME{Symbol("text/html")}, figlike::Makie.FigureAxisPlot; backend::Module, update::Bool)
      @ Makie /tmp/jl_PG0swv/build/packages/Makie/FUAHr/src/display.jl:258
   [17] show
      @ /tmp/jl_PG0swv/build/packages/Makie/FUAHr/src/display.jl:247 [inlined]
   [18] response(content::Makie.FigureAxisPlot, mime_type::MIME{Symbol("text/html")}, status::Int64, headers::Vector{Any})
      @ WGLMakieExt /tmp/jl_PG0swv/build/dev/Oxygen/ext/WGLMakieExt.jl:21
   [19] html
      @ /tmp/jl_PG0swv/build/dev/Oxygen/ext/WGLMakieExt.jl:37 [inlined]
   [20] html(fig::Makie.FigureAxisPlot)
      @ WGLMakieExt /tmp/jl_PG0swv/build/dev/Oxygen/ext/WGLMakieExt.jl:37
   [21] macro expansion
      @ /tmp/jl_PG0swv/host/dev/Oxygen/test/extensions/wglmakietests.jl:14 [inlined]
   [22] macro expansion
      @ ~/.julia/juliaup/julia-1.11.6+0.x64.linux.gnu/share/julia/stdlib/v1.11/Test/src/Test.jl:1709 [inlined]
   [23] top-level scope
      @ /tmp/jl_PG0swv/host/dev/Oxygen/test/extensions/wglmakietests.jl:12
   [24] include(mod::Module, _path::String)
      @ Base ./Base.jl:562
   [25] include(x::String)
      @ Main.RunTests /tmp/jl_PG0swv/host/dev/Oxygen/test/runtests.jl:1
   [26] top-level scope
      @ /tmp/jl_PG0swv/host/dev/Oxygen/test/runtests.jl:12
   [27] include(fname::String)
      @ Main ./sysimg.jl:38
   [28] top-level scope
      @ none:6
Test Summary:        | Error  Total  Time
WGLMakie Utils tests |     1      1  1.5s
ERROR: LoadError: Some tests did not pass: 0 passed, 0 failed, 1 errored, 0 broken.
in expression starting at /tmp/jl_PG0swv/host/dev/Oxygen/test/extensions/wglmakietests.jl:1
in expression starting at /tmp/jl_PG0swv/host/dev/Oxygen/test/runtests.jl:1

ohmsweetohm1 · August 25, 2025, 11:06am

Very cool work!
It would be nice if this was a package, and maybe a Github workflow such that a badge can be produced.

fatteneder · August 25, 2025, 11:18am

I want to clarify a few things about the current status of relocability.

As already mentioned in other posts, there are essentially two cases we distinguish:

Paths used with include(..)/include_dependency().
Hardcoded strings, including uses of @__DIR__ and friends.

The first one is taken care of as of 1.11.
As long as a user compiles a pkg into a writeable depot (first in DEPOT_PATH) it should work.
Use JULIA_DEBUG=loading to debug re-compilation and loading issues.

There are also two proposals to solve 2., but the discussion around this stalled.
These solutions introduce a new macro that pkg authors need to opt-in for relocatability.

It might be possible to do away with (many cases of) hardcoded strings (global or function level)
using a search-replace strategy. But I haven’t looked into this yet and it might come with load-time regressions.
However, as Benny mentioned, people are allowed to do weird things with strings.
So it seems reasonable to expect from pkg authors to do some thinking when writing pkg code that should be relocatable.
The current issue is that we lack a solution for 2, which could either be a @RELOCDIR macro or a potential Path type.

What I haven’t touched is relocation issues in third-party libraries, e.g. hardcoded paths in a C lib.
I hope these are more rare.

About Base.isrelocatable: Right now this only checks if a pkg is relocatable after compliation is done,
e.g. it checks if all include() paths could be resolved to a depot.
It was added as some pkgs like to tinker with DEPOT_PATH mid compilation, which you shouldn’t do …

PS: I like the relocatability test script.

nhz2 · August 25, 2025, 3:48pm

Thank you for all the work you’ve done in Julia to get precompilation relocatable.

Does calling pkgdir at runtime work as another alternative to @__DIR__? I don’t see how the two proposals would be better than that. For example, even though MeshCat.jl was using artifacts, I had to change:

const VIEWER_ROOT = joinpath(first(readdir(artifact"meshcat", join=true)), "dist")

to

VIEWER_ROOT() = joinpath(first(readdir(artifact"meshcat", join=true)), "dist")

If the data was in the repo, would something like:

VIEWER_ROOT() = joinpath(pkgdir(@__MODULE__), "dist")

have worked?

fatteneder · August 25, 2025, 4:20pm

The const case doesn’t work, because the RHS will be expanded to an (absolute) path and this will be baked-into the pkgimage. This is case 2 from above.

I think the two function-based versions rely on the assumption that VIEWER_ROOT() will not be const-proped and evaluated during compilation, and thus, also expand to an absolute path. Not sure how likely it is that this will ever be optimized. Maybe the presence of @__MODULE__ even guards it from evaluation and makes it safer, not sure. Otherwise, it might be fine.

The, IMO, correct implementation uses a Base specific type, e.g. RelocPath("/path/to/depot/sub/path/to/file"). When we construct that object we replace the first part to read @depot/sub/path/to/file and then bake this string into the pkg image. That’s also how include() works, which I think is good.
The current proposal comes with the overhead, that we have to patch the "@depot" every time we convert the type to a string. But it might be possible to only do this once during loading.

sdanisch · August 25, 2025, 4:37pm

I think to avoid downloading deps you need these options for create_app:

github.com/MakieOrg/Makie.jl

relocatability.jl

master


      
          
          open("MakieApp/src/MakieApp.jl", "w") do io
              print(io, module_src)
          end
          
          Pkg.activate(".")
          Pkg.add("PackageCompiler")
          
          using PackageCompiler
          
          create_app(joinpath(pwd(), "MakieApp"), "executable"; force = true, incremental = true, include_transitive_dependencies = false)
          exe = joinpath(pwd(), "executable", "bin", "MakieApp")
          # `run` allows to see potential informative printouts, `success` swallows those
          p = run(`$(exe)`)
          @test p.exitcode == 0
          
          julia_pkg_dir = joinpath(Base.DEPOT_PATH[1], "packages")
          @test isdir(julia_pkg_dir)
          mvd_julia_pkg_dir = julia_pkg_dir * ".old"
          new_makie_dir = makie_dir * ".old"
          mv(julia_pkg_dir, mvd_julia_pkg_dir; force = true)

nhz2 · August 25, 2025, 5:28pm

The artifact version of VIEWER_ROOT() can’t be const-proped because by calling the function artifacts_dirs julia/stdlib/Artifacts/src/Artifacts.jl at 364ecb3a114fd2638df60654807474e7f38cb04e · JuliaLang/julia · GitHub, its output depends on the runtime state of Base.DEPOT_PATH

Currently, it looks like pkgdir is safe from const-prop because it does a runtime lookup into Base.pkgorigins julia/base/loading.jl at 364ecb3a114fd2638df60654807474e7f38cb04e · JuliaLang/julia · GitHub, but it would be nice for this to be documented.

A proper path system would be nice instead of doing everything with strings, but I don’t think this is absolutely needed to get all the major packages relocatable, given all the existing tools we have.

lmiq · August 25, 2025, 7:09pm

Could this be added as a on optional test to Aqua.jl ?

Topic		Replies	Views
Relocatable app workflow with PackageCompiler General Usage package , package-compiler , application , pkgtemplates	5	983	July 22, 2021
Compiling and building binaries from your julia code General Usage compilation	49	24183	February 16, 2018
How to input files as command line arguments to compiled Julia General Usage question , package , dataframes , csv , package-compiler	46	2983	July 20, 2021
How to compile a portable binary (at least across macs) with `juliac.jl` Tooling interoperability , compilation	12	6433	March 23, 2018
[ANN] EmbeddedStaticFiles.jl - Store your static assets, both locally and relocatably Package Announcements package , announcement	12	250	September 28, 2024

Related topics