[ANN] EmbeddedStaticFiles.jl - Store your static assets, both locally and relocatably

Package Announcements
,
1

Edit) @nhz2’s answer is a way more sound solution, and I suggest doing the same when you need to embed something. I’ll leave this package only as an example of something you shouldn’t do.

Hi everyone! I’m happy to introduce you EmbeddedStaticFiles.jl, a package that lets you embed static files to your package, both locally and relocatably!

It is probably not the best approach this package has on the subject of local & relocatable file assets, but this was the only working solution I could come up with in the moment.
Wish it helps you too in times of need!

How it works

  • It basically encodes your static files to base64 format, turn them as julia strings, and save them as julia code files. They are retrieved in-memory when you load the wrapper julia module.

When to use

  • It can be used as a RELOCATABLE and LOCAL asset storage. Currently, there’s no other way of achieving them both; using @__DIR__ for local files will create a package that’s non-relocatable when compiled as a sysimage or an executable, and the Artifacts system doesn’t support local files. If your files are small, and serving them via separate servers seems to be an overkill, EmbeddedStaticFiles.jl may as well do the job.

Disclaimer

  • It is meant to be used for small files only, e.g. some files which are a few megabytes at tops. Beware that it is highly memory-inefficient! It stores some bloated version of your file from the start, as it is base64 encoded. And since you’re likely to use the decoded version, there would be least 2 copies of your file on the memory - the embedded base64-encoded file that’s stored, and the decoded file that you’ll use. This is a huge memory bloat, and definitely a footgun if you embed gigabytes of files.

  • Julia seem to have a maximum length of code it can parse, and embedding a very large file, e.g. more than 3 GB, will break the julia parser. (I’ve tried it and seen it happen)

  • It only supports static files. You won’t be able to create new files nor modify the existing ones in runtime.

Usage

Installation


using Pkg

# Use git for installation, as it is currently not on the julia package registry.

Pkg.add("https://github.com/JuliaServices/ConcurrentUtilities.jl.git")

Example scenario

Let’s assume there’s a simple julia package in ~/SimplePackage. It would be nice to have a few template html/js/css files as assets, which are located in ~/data as index.html, main.js, and main.css. You want the assets to be stored in ~/SimplePackage/src/static_files. You want the file handler(wrapper julia module)'s name to be “File”.


julia> using EmbeddedStaticFiles

julia> pwd()

"/home/some_user/SimplePackage"

# Create the FileHandleBuilder. Default values: handler_name="File", entries=Dict(), out_dir="./static_files"

julia> filehandle_builder=FileHandleBuilder(handler_name="File", entries=Dict(), out_dir="./src/static_files")

# push!(FileHandleBuilder, arbitrary_key_name, path_to_file)

julia> push!(filehandle_builder, "index_html", "../data/index.html")

julia> push!(filehandle_builder, "main_js", "../data/main.js")

julia> push!(filehandle_builder, "main_css", "../data/main.css")

# In case you pushed a wrong entry to the FileHandleBuilder

julia> push!(filehandle_builder, "maybe a cat", "walked over the keyboard")

julia> delete!(filehandle_builder, "maybe a cat")

# Create the encoded files and the wrapper module

julia> embed(filehandle_builder)

This will create the encoded files in ~/SimplePackage/src/static_files/data as index_html.jl, main_js.jl, and main_css.jl. The wrapper module to call them will be located as ~/SimplePackage/src/static/file_handle.jl.

Now, if you’re to use the files in SimplePackage, simply include the file_handle.jl


#~/SimplePackage/src/SimplePackage.jl

module SimplePackage

...

include("static_files/file_handle.jl")

#Now you can call the embedded files via MODULE_NAME.get(key_name)

File.get("index_html") # Give you the content of your file as an U8 array.
2 Likes
2

I have some gaps in my experience with this, but with what I have experienced, it is possible to use @__DIR__ at runtime and include artifact files next to the executable. It can be done by adding dir as a keyword argument to the entry point, like the main(; dir = @__DIR__) function, which can be passed at runtime. But if you want them embedded, I guess this is the way to go.

Note that there is also RelocatableFolders.jl package, which you could add to the comparison.

3

Unlike C you don’t need to create julia code files to embed data in a sysimage or pkgimage.
Just use the include_dependency function, and read into a global constant. For example:

2 Likes
4

Thank you for your suggestion of using dir=@__DIR__ to the entry point, but when tested,

module MyApp1

function julia_main(;dir=@__DIR__)::Cint
    println(dir);
    return 0
end

end

the dir is hardcoded, giving me the same absolute path whereever I change the executable’s location to.

As far as I know, problem with relocatability and @__DIR__ macro is that the macro is turned to an absolute path when compiled. So when you compile your project to a sysimage or an executable using PackageCompiler, the @__DIR__ will be replaced to the file dir when it was first compiled, and it will remain so even if you move the executable. Hence any version of code using @__DIR__ is unlikely to be relocatable. Some uses of @DIR should be relocatable. #54430 might help elaborate the issue.

If I’m mistaken and there’s a workaround for this, please let me know.

RelocatableFolders.jl uses Scratchspace when relocated, which is basically downloading the file from the url. This is also how julia’s current Artifact system works - downloading from the url, avoiding relocability issue by not using local files in the first place.

5

I would really want to embed it the proper way like this, but using it inside a package requires absolute path to the file, which enforces you to use @__DIR__ and cause the relocatability issue.

6

Because the reading is happening at precompile time (because the variable is a global constant), there won’t be any relocatability issues with using @__DIR__ in that specific context.

7

I used RelocatableFolders for QML source files which were needed for precompiling a workload using PrecompileTools. There I found that I could get rid of it if I pass dir as a compiled function argument.

It seems a proper solution with PackageCompiler, which compiles everything, is to imitate what C does. When a command is called in C, the first argument ARGV[0] is its path from which other file locations relative to the executable can be constructed. Perhaps Sys.BINDIR is set at runtime from which dir could be constructed.

8

Testing it actually seems to cause relocatability issue. It still requires the file to be in the exact location when it was compiled, and otherwise cause an error.

# Test package with sample.jpg
module MyPackage

export send_sample_file

function send_sample_file()
    include_dependency(joinpath(@__DIR__, "../sample.jpg"))
    read(joinpath(@__DIR__, "../sample.jpg"))
end

end

# Test app to be compiled
module MyApp
using MyPackage

function julia_main()::Cint
    println(send_sample_file());
    return 0
end

end

# Compiled test app using PackageCompiler
$ ./MyAppCompiled/bin/MyApp
UInt8[.....

# After moving MyPackage to another location
$ ./MyAppCompiled/bin/MyApp
SystemError: opening file "/some/file/path/src/../sample.jpg": No such file or directory
9

It is a valid and sound solution if you were to use it as a end-point product. But I’m afraid it’s not quite applicable to packages, as manually locating where the package would be is not quite viable in the moment.

10

Yes, this is not relocatable because the read happens inside a function. Try with the read happening at the top level into a global constant:

# Test package with sample.jpg
module MyPackage

export send_sample_file

include_dependency(joinpath(@__DIR__, "../sample.jpg"))
const SAMPLE_DATA = read(joinpath(@__DIR__, "../sample.jpg"))

function send_sample_file()
    SAMPLE_DATA
end

end
3 Likes
11

I haven’t put much though to it, but what happens to __init__ blocks in compilation. Isn’t it possible to set @__DIR__ as global constant there?

Ah, I see. The @__DIR__ seems is static when module is compiled. Perhaps in such situations a wrapper module could be used which could contain all the assets and is not compiled.

12

You’re right, it works!
I’ve been struggling with it for a long time, and I don’t know how to thank you enough.

Guess I should have asked about it first before trying build a package.

13

It seemed worth taking a look at, so I tried it, and it indeed is static when compiled.

Probably for the best, I guess. Spltting the assets part would be the general solution for assets with arbitrary size.
Thank you for your participation!

1 Like