# \[ANN\] EmbeddedStaticFiles.jl - Store your static assets, both locally and relocatably

**URL:** <https://discourse.julialang.org/t/ann-embeddedstaticfiles-jl-store-your-static-assets-both-locally-and-relocatably/119969>\
**Category:** Package Announcements\
**Tags:** package, announcement\
**Created:** [September 27, 2024, 3:20pm UTC](https://discourse.julialang.org/t/ann-embeddedstaticfiles-jl-store-your-static-assets-both-locally-and-relocatably/119969 "2024-09-27T15:20:00Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![littlelib](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/littlelib/32/209324_2.png) [@littlelib](https://discourse.julialang.org/u/littlelib)\
**Post date:** [September 27, 2024, 3:20pm UTC](https://discourse.julialang.org/t/ann-embeddedstaticfiles-jl-store-your-static-assets-both-locally-and-relocatably/119969/1 "2024-09-27T15:20:00Z")

</div>

Edit) @nhz2’s [answer](https://discourse.julialang.org/t/ann-embeddedstaticfiles-jl-store-your-static-assets-both-locally-and-relocatably/119969/10) is a way more sound solution, and I suggest doing the same when you need to embed something. I’ll leave this package only as an example of something you shouldn’t do.

* * *

Hi everyone! I’m happy to introduce you [EmbeddedStaticFiles.jl,](https://github.com/littlelib/EmbeddedStaticFiles.jl) a package that lets you embed static files to your package, both locally and relocatably!

It is probably not the best approach this package has on the subject of local & relocatable file assets, but this was the only working solution I could come up with in the moment.  
Wish it helps you too in times of need!

# How it works

- It basically encodes your static files to base64 format, turn them as julia strings, and save them as julia code files. They are retrieved in-memory when you load the wrapper julia module.

# When to use

- It can be used as a **RELOCATABLE** and **LOCAL** asset storage. Currently, there’s no other way of achieving them both; using @\_\_DIR\_\_ for local files will create a package that’s non-relocatable when compiled as a sysimage or an executable, and the Artifacts system doesn’t support local files. If your files are small, and serving them via separate servers seems to be an overkill, EmbeddedStaticFiles.jl may as well do the job.

# Disclaimer

- It is meant to be used for small files only, e.g. some files which are a few megabytes at tops. **Beware that it is highly memory-inefficient!** It stores some bloated version of your file from the start, as it is base64 encoded. And since you’re likely to use the decoded version, there would be least 2 copies of your file on the memory - the embedded base64-encoded file that’s stored, and the decoded file that you’ll use. This is a huge memory bloat, and definitely a footgun if you embed gigabytes of files.

- Julia seem to have a maximum length of code it can parse, and embedding a very large file, e.g. more than 3 GB, will break the julia parser. (I’ve tried it and seen it happen)

- It only supports static files. You won’t be able to create new files nor modify the existing ones in runtime.

# Usage

## Installation

```julia

using Pkg

# Use git for installation, as it is currently not on the julia package registry.

Pkg.add("https://github.com/JuliaServices/ConcurrentUtilities.jl.git")

```

## Example scenario

Let’s assume there’s a simple julia package in `~/SimplePackage`. It would be nice to have a few template html/js/css files as assets, which are located in `~/data` as `index.html`, `main.js`, and `main.css`. You want the assets to be stored in `~/SimplePackage/src/static_files`. You want the file handler(wrapper julia module)'s name to be “File”.

```julia

julia> using EmbeddedStaticFiles

julia> pwd()

"/home/some_user/SimplePackage"

# Create the FileHandleBuilder. Default values: handler_name="File", entries=Dict(), out_dir="./static_files"

julia> filehandle_builder=FileHandleBuilder(handler_name="File", entries=Dict(), out_dir="./src/static_files")

# push!(FileHandleBuilder, arbitrary_key_name, path_to_file)

julia> push!(filehandle_builder, "index_html", "../data/index.html")

julia> push!(filehandle_builder, "main_js", "../data/main.js")

julia> push!(filehandle_builder, "main_css", "../data/main.css")

# In case you pushed a wrong entry to the FileHandleBuilder

julia> push!(filehandle_builder, "maybe a cat", "walked over the keyboard")

julia> delete!(filehandle_builder, "maybe a cat")

# Create the encoded files and the wrapper module

julia> embed(filehandle_builder)

```

This will create the encoded files in `~/SimplePackage/src/static_files/data` as `index_html.jl`, `main_js.jl`, and `main_css.jl`. The wrapper module to call them will be located as `~/SimplePackage/src/static/file_handle.jl`.

Now, if you’re to use the files in SimplePackage, simply include the `file_handle.jl`

```julia

#~/SimplePackage/src/SimplePackage.jl

module SimplePackage

...

include("static_files/file_handle.jl")

#Now you can call the embedded files via MODULE_NAME.get(key_name)

File.get("index_html") # Give you the content of your file as an U8 array.

```

---

<div class="post-metadata">

**Author:** ![Janis\_Erdmanis](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/janis_erdmanis/32/10869_2.png) [@Janis\_Erdmanis](https://discourse.julialang.org/u/Janis_Erdmanis)\
**Post date:** [September 27, 2024, 4:17pm UTC](https://discourse.julialang.org/t/ann-embeddedstaticfiles-jl-store-your-static-assets-both-locally-and-relocatably/119969/2 "2024-09-27T16:17:37Z")

</div>

> [@littlelib](#):
>
> It can be used as a **RELOCATABLE** and **LOCAL** asset storage. Currently, there’s no other way of achieving them both; using @ **DIR** for local files will create a package that’s non-relocatable when compiled as a sysimage or an executable, and the Artifacts system doesn’t support local files.

I have some gaps in my experience with this, but with what I have experienced, it is possible to use `@ __DIR__ ` at runtime and include artifact files next to the executable. It can be done by adding `dir` as a keyword argument to the entry point, like the `main(; dir = @ __DIR__ )` function, which can be passed at runtime. But if you want them embedded, I guess this is the way to go.

Note that there is also `RelocatableFolders.jl` package, which you could add to the comparison.

---

<div class="post-metadata">

**Author:** ![nhz2](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nhz2/32/44428_2.png) [@nhz2](https://discourse.julialang.org/u/nhz2)\
**Post date:** [September 27, 2024, 6:42pm UTC](https://discourse.julialang.org/t/ann-embeddedstaticfiles-jl-store-your-static-assets-both-locally-and-relocatably/119969/3 "2024-09-27T18:42:13Z")

</div>

Unlike C you don’t need to create julia code files to embed data in a sysimage or pkgimage.  
Just use the `include_dependency` function, and `read` into a global constant. For example:

> <https://github.com/ASML-Labs/PPTX.jl/blob/cfd2d2fb03fb4e48ce4efab04362c4b9300827be/src/constants.jl#L15-L19>

---

<div class="post-metadata">

**Author:** ![littlelib](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/littlelib/32/209324_2.png) [@littlelib](https://discourse.julialang.org/u/littlelib)\
**Post date:** [September 27, 2024, 10:55pm UTC](https://discourse.julialang.org/t/ann-embeddedstaticfiles-jl-store-your-static-assets-both-locally-and-relocatably/119969/4 "2024-09-27T22:55:44Z")

</div>

> [@Janis\_Erdmanis](#):
>
> I have some gaps in my experience with this, but with what I have experienced, it is possible to use `@ __DIR__ ` at runtime and include artifact files next to the executable. It can be done by adding `dir` as a keyword argument to the entry point, like the `main(; dir = @ __DIR__ )` function, which can be passed at runtime. But if you want them embedded, I guess this is the way to go.

Thank you for your suggestion of using `dir=@ __DIR__ ` to the entry point, but when tested,

```julia
module MyApp1

function julia_main(;dir=@ __DIR__ )::Cint
    println(dir);
    return 0
end

end

```

the dir is hardcoded, giving me the same absolute path whereever I change the executable’s location to.

As far as I know, problem with relocatability and `@ __DIR__ ` macro is that the macro is turned to an absolute path when compiled. So when you compile your project to a sysimage or an executable using PackageCompiler, the `@ __DIR__ ` will be replaced to the file dir when it was first compiled, and it will remain so even if you move the executable. Hence any version of code using `@ __DIR__ ` is unlikely to be relocatable. [Some uses of @ **DIR** should be relocatable. #54430](https://github.com/JuliaLang/julia/issues/54430) might help elaborate the issue.

If I’m mistaken and there’s a workaround for this, please let me know.

> [@Janis\_Erdmanis](#):
>
> Note that there is also `RelocatableFolders.jl` package, which you could add to the comparison.

RelocatableFolders.jl uses Scratchspace when relocated, which is basically downloading the file from the url. This is also how julia’s current Artifact system works - downloading from the url, avoiding relocability issue by not using local files in the first place.

---

<div class="post-metadata">

**Author:** ![littlelib](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/littlelib/32/209324_2.png) [@littlelib](https://discourse.julialang.org/u/littlelib)\
**Post date:** [September 27, 2024, 10:58pm UTC](https://discourse.julialang.org/t/ann-embeddedstaticfiles-jl-store-your-static-assets-both-locally-and-relocatably/119969/5 "2024-09-27T22:58:06Z")

</div>

> [@nhz2](#):
>
> Unlike C you don’t need to create julia code files to embed data in a sysimage or pkgimage.  
> Just use the `include_dependency` function, and `read` into a global constant. For example:

I would really want to embed it the proper way like this, but using it inside a package requires absolute path to the file, which enforces you to use `@ __DIR__ ` and cause the relocatability issue.

---

<div class="post-metadata">

**Author:** ![nhz2](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nhz2/32/44428_2.png) [@nhz2](https://discourse.julialang.org/u/nhz2)\
**Post date:** [September 27, 2024, 11:14pm UTC](https://discourse.julialang.org/t/ann-embeddedstaticfiles-jl-store-your-static-assets-both-locally-and-relocatably/119969/6 "2024-09-27T23:14:00Z")

</div>

Because the reading is happening at precompile time (because the variable is a global constant), there won’t be any relocatability issues with using `@ __DIR__ ` in that specific context.

---

<div class="post-metadata">

**Author:** ![Janis\_Erdmanis](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/janis_erdmanis/32/10869_2.png) [@Janis\_Erdmanis](https://discourse.julialang.org/u/Janis_Erdmanis)\
**Post date:** [September 27, 2024, 11:20pm UTC](https://discourse.julialang.org/t/ann-embeddedstaticfiles-jl-store-your-static-assets-both-locally-and-relocatably/119969/7 "2024-09-27T23:20:29Z")

</div>

I used `RelocatableFolders` for QML source files which were needed for precompiling a workload using `PrecompileTools`. There I found that I could get rid of it if I pass `dir` as a compiled function argument.

It seems a proper solution with `PackageCompiler`, which compiles everything, is to imitate what `C` does. When a command is called in `C`, the first argument `ARGV[0]` is its path from which other file locations relative to the executable can be constructed. Perhaps `Sys.BINDIR` is set at runtime from which dir could be constructed.

---

<div class="post-metadata">

**Author:** ![littlelib](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/littlelib/32/209324_2.png) [@littlelib](https://discourse.julialang.org/u/littlelib)\
**Post date:** [September 27, 2024, 11:43pm UTC](https://discourse.julialang.org/t/ann-embeddedstaticfiles-jl-store-your-static-assets-both-locally-and-relocatably/119969/8 "2024-09-27T23:43:05Z")

</div>

Testing it actually seems to cause relocatability issue. It still requires the file to be in the exact location when it was compiled, and otherwise cause an error.

```julia
# Test package with sample.jpg
module MyPackage

export send_sample_file

function send_sample_file()
    include_dependency(joinpath(@ __DIR__ , "../sample.jpg"))
    read(joinpath(@ __DIR__ , "../sample.jpg"))
end

end

```

```julia
# Test app to be compiled
module MyApp
using MyPackage

function julia_main()::Cint
    println(send_sample_file());
    return 0
end

end

```

```julia
# Compiled test app using PackageCompiler
$ ./MyAppCompiled/bin/MyApp
UInt8[.....

# After moving MyPackage to another location
$ ./MyAppCompiled/bin/MyApp
SystemError: opening file "/some/file/path/src/../sample.jpg": No such file or directory

```

---

<div class="post-metadata">

**Author:** ![littlelib](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/littlelib/32/209324_2.png) [@littlelib](https://discourse.julialang.org/u/littlelib)\
**Post date:** [September 27, 2024, 11:55pm UTC](https://discourse.julialang.org/t/ann-embeddedstaticfiles-jl-store-your-static-assets-both-locally-and-relocatably/119969/9 "2024-09-27T23:55:29Z")

</div>

> [@Janis\_Erdmanis](#):
>
> It seems a proper solution with `PackageCompiler`, which compiles everything, is to imitate what `C` does. When a command is called in `C`, the first argument `ARGV[0]` is its path from which other file locations relative to the executable can be constructed. Perhaps `Sys.BINDIR` is set at runtime from which dir could be constructed.

It is a valid and sound solution if you were to use it as a end-point product. But I’m afraid it’s not quite applicable to packages, as manually locating where the package would be is not quite viable in the moment.

---

<div class="post-metadata">

**Author:** ![nhz2](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nhz2/32/44428_2.png) [@nhz2](https://discourse.julialang.org/u/nhz2)\
**Post date:** [September 28, 2024, 12:03am UTC](https://discourse.julialang.org/t/ann-embeddedstaticfiles-jl-store-your-static-assets-both-locally-and-relocatably/119969/10 "2024-09-28T00:03:45Z")

</div>

Yes, this is not relocatable because the read happens inside a function. Try with the read happening at the top level into a global constant:

```julia
# Test package with sample.jpg
module MyPackage

export send_sample_file

include_dependency(joinpath(@ __DIR__ , "../sample.jpg"))
const SAMPLE_DATA = read(joinpath(@ __DIR__ , "../sample.jpg"))

function send_sample_file()
    SAMPLE_DATA
end

end

```

---

<div class="post-metadata">

**Author:** ![Janis\_Erdmanis](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/janis_erdmanis/32/10869_2.png) [@Janis\_Erdmanis](https://discourse.julialang.org/u/Janis_Erdmanis)\
**Post date:** [September 28, 2024, 12:22am UTC](https://discourse.julialang.org/t/ann-embeddedstaticfiles-jl-store-your-static-assets-both-locally-and-relocatably/119969/11 "2024-09-28T00:22:10Z")

</div>

~~I haven’t put much though to it, but what happens to ` __init__ ` blocks in compilation. Isn’t it possible to set `@ __DIR__ ` as global constant there?~~

Ah, I see. The `@ __DIR__ ` seems is static when module is compiled. Perhaps in such situations a wrapper module could be used which could contain all the assets and is not compiled.

---

<div class="post-metadata">

**Author:** ![littlelib](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/littlelib/32/209324_2.png) [@littlelib](https://discourse.julialang.org/u/littlelib)\
**Post date:** [September 28, 2024, 12:23am UTC](https://discourse.julialang.org/t/ann-embeddedstaticfiles-jl-store-your-static-assets-both-locally-and-relocatably/119969/12 "2024-09-28T00:23:10Z")

</div>

You’re right, it works!  
I’ve been struggling with it for a long time, and I don’t know how to thank you enough.

Guess I should have asked about it first before trying build a package.

---

<div class="post-metadata">

**Author:** ![littlelib](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/littlelib/32/209324_2.png) [@littlelib](https://discourse.julialang.org/u/littlelib)\
**Post date:** [September 28, 2024, 12:46am UTC](https://discourse.julialang.org/t/ann-embeddedstaticfiles-jl-store-your-static-assets-both-locally-and-relocatably/119969/13 "2024-09-28T00:46:30Z")

</div>

> [@Janis\_Erdmanis](#):
>
> I haven’t put much though to it, but what happens to ` __init__ ` blocks in compilation. Isn’t it possible to set `@ __DIR__ ` as global constant there?

It seemed worth taking a look at, so I tried it, and it indeed is static when compiled.

> [@Janis\_Erdmanis](#):
>
> Perhaps in such situations a wrapper module could be used which could contain all the assets and is not compiled.

Probably for the best, I guess. Spltting the assets part would be the general solution for assets with arbitrary size.  
Thank you for your participation!