Register and specialize a function in a different module/package (Julia segfaults)

I am working on a DSL for creating web pages. Something like this (Julia like pseudo-code):

module Html

function table(f::Function; args...)
# actual implementation here
end

function tr(f::Function; args...)
# actual implementation here
end

function td(f::Function; args...)
# actual implementation here
end

end

A user would produce code as:

using Html

table(border=0) do 
  tr(style="border: 1px solid red") do 
    td()
    td()
  end
end

The Html module provides an API for all HTML5 elements, which is a well defined set (so function are created for each standard element). This allows other packages to extend the Html API, for instance to provide more advanced table APIs:

module HtmlX

using Html

function Html.table(rows::Vector, columns::Vector)
# actual implementation here
end

function Html.table(df::DataFrames.DataFrame)
# actual implementation here
end

end

So all goes well for APIs for working with the standard HTML5 elements. However, the goal is to use web components, which extend the HTML5 standard with random elements. Like say a paginated_table. The problem is that Html can not define a paginated_table function in advance.

So I tried something like this:

module Html

function register_element(elem)
  Core.eval(@__MODULE__, Meta.parse("function $elem end"))
end

end

But when running:

module HtmlX

Html.register_element("paginated_table")

end

Julia complains that I’m evaling in a closed module.

The objective is to have a rich ecosystem of 3rd party packages where users install and use a variety of such external elements. I don’t like the idea of having the elements defined in say 10 modules within a project, as this would be very confusing for the users. An alternative would be for the packages to export their functions but this would lead to an inconsistent API as Html does not export the functions.

Any ideas or best practices? Thanks!

You don’t want to use Core.eval, but <module>.eval instead. The following works for me:

julia> module M
       f(s) = @__MODULE__().eval(:(function $s end))
       end
Main.M

julia> M.f(:test)
test (generic function with 0 methods)

Although I’m not so sure, the lost transparency of this approach over having packages export their own modules/functions is really worth it.
Edit:
Or just call Html.eval(...) from outside the Html Module.

@simeonschaub Thank you! This is surprising, I expected that eval module code would have the same behaviour as module.eval code. I mean, hmmm, why different eval behaviours and why having restrictions on one if there’s another way of modifying a closed module.

I think that having all the HTML elements defined within Html would respect the principle of least surprise - users would just know where to find them. Otherwise, for consistency, I’d have to export over 100 functions defined within the Html module itself.


Related, I love the way functions in other modules can be specialised with Julia, it’s fantastic! But something that I’m missing and would make sense in this case (adding new functions) would be to allow reopening modules like Ruby does. Might be interesting for Julia v2:

# PaginatedTable.jl
opened module Html
function paginated_table end
end

module PaginatedTable

using Html

function Html.paginated_table()
# implementation
end

end

Are you talking about something similar to a header file? It seems to me like this is something that could probably be solved by tooling, maybe combined with some metaprogramming

FWIW, Core.eval(module) is better than module.eval()

It would produce a similar result, but less sucky and more powerful. In Ruby, one can simply reopen a module and operate in it as if it was yours (and add and remove methods for example).

Here, example from the Ruby docs:

Would you be so kind to elaborate in which way is it better and why? Always curious to learn about Julia’s internals. Thanks!

Because module.eval is not internal. It may or may not exist and may or may not do anything you want.

1 Like

For example:

julia> module BadlyBehaved
         function eval(x)
           println("nope")
         end
       end
Main.BadlyBehaved

julia> BadlyBehaved.eval(:(x = 1))
nope

julia> Core.eval(BadlyBehaved, :(x = 1))
1

julia> BadlyBehaved.x
1

1 Like

@yuyichao @rdeits Ah yes, thanks, makes total sense!

However, sadly, it does not work.

┌ Debug: 2020-03-06 14:38:53 Precompiling Stipple [c3245d6e-4e41-4672-accb-64620d5e5a71]
└ @ Base loading.jl:1273

WARNING: eval into closed module Html:
Expr(:function, :stylesheet)
  ** incremental compilation may be fatally broken for this module **

And then Julia segfaults:

signal (11): Segmentation fault: 11

This is the offending code:

Core.eval(Genie.Renderer.Html, :(function stylesheet end))

function Genie.Renderer.Html.stylesheet(href::String; args...) :: String
  Genie.Renderer.Html.link(href=href, rel="stylesheet", args...)
end

Running various tests seems to confirm that the first part is successful (defining the function) but the second (specializing the function) causes Julia to crash. This behaviour was consistent throughout all my tests (always segfaults).

Leaving just the eval and running the function specialization code from the REPL works fine.


Update 1:
Extracting away just the code, out of the package I’m developing into a minimal test file, works fine. So some combination of factors causes the segfault.

OK, pfff, so I managed to figure out when the segfault occurs. It seems to happen only when doing the eval and the specialization within a package.

Thus, if I place this code in a test.jl file:

module Stipple

using Genie

try
  Core.eval(Genie.Renderer.Html, :(function stylesheet end))
catch ex
  @error ex
end

function Genie.Renderer.Html.stylesheet(href::String; args...) :: String
  Genie.Renderer.Html.link(href=href, rel="stylesheet", args...)
end

end

and then run $ julia test.jl it works perfectly.

However, if I put the above code within a developed package called Stipple and then in test.jl I put using Stipple, Julia will segfault with:

$ julia test.jl                                                                                                                                                                                             
WARNING: eval into closed module Html:
Expr(:function, :stylesheet)
  ** incremental compilation may be fatally broken for this module **


signal (11): Segmentation fault: 11
in expression starting at /Users/adrian/test.jl:25
jl_unwrap_unionall at /Users/julia/buildbot/worker/package_macos64/build/src/jltypes.c:953
jl_deserialize_value_any at /Users/julia/buildbot/worker/package_macos64/build/src/dump.c:1986 [inlined]
jl_deserialize_value at /Users/julia/buildbot/worker/package_macos64/build/src/dump.c:2212
jl_deserialize_datatype at /Users/julia/buildbot/worker/package_macos64/build/src/dump.c:1371 [inlined]
jl_deserialize_value at /Users/julia/buildbot/worker/package_macos64/build/src/dump.c:2209
jl_deserialize_value_svec at /Users/julia/buildbot/worker/package_macos64/build/src/dump.c:1508 [inlined]
jl_deserialize_value at /Users/julia/buildbot/worker/package_macos64/build/src/dump.c:2057
jl_deserialize_datatype at /Users/julia/buildbot/worker/package_macos64/build/src/dump.c:1485 [inlined]
jl_deserialize_value at /Users/julia/buildbot/worker/package_macos64/build/src/dump.c:2209
jl_deserialize_struct at /Users/julia/buildbot/worker/package_macos64/build/src/dump.c:1901
jl_deserialize_typemap_entry at /Users/julia/buildbot/worker/package_macos64/build/src/dump.c:1941 [inlined]
jl_deserialize_value at /Users/julia/buildbot/worker/package_macos64/build/src/dump.c:2195
jl_deserialize_struct at /Users/julia/buildbot/worker/package_macos64/build/src/dump.c:1901
jl_deserialize_value_any at /Users/julia/buildbot/worker/package_macos64/build/src/dump.c:2013 [inlined]
jl_deserialize_value at /Users/julia/buildbot/worker/package_macos64/build/src/dump.c:2212
jl_deserialize_value_any at /Users/julia/buildbot/worker/package_macos64/build/src/dump.c:1981 [inlined]
jl_deserialize_value at /Users/julia/buildbot/worker/package_macos64/build/src/dump.c:2212
jl_deserialize_datatype at /Users/julia/buildbot/worker/package_macos64/build/src/dump.c:1481 [inlined]
jl_deserialize_value at /Users/julia/buildbot/worker/package_macos64/build/src/dump.c:2209
jl_deserialize_value_module at /Users/julia/buildbot/worker/package_macos64/build/src/dump.c:1815 [inlined]
jl_deserialize_value at /Users/julia/buildbot/worker/package_macos64/build/src/dump.c:2134
jl_deserialize_value_array at /Users/julia/buildbot/worker/package_macos64/build/src/dump.c:1573
_jl_restore_incremental at /Users/julia/buildbot/worker/package_macos64/build/src/dump.c:3213
jl_restore_incremental at /Users/julia/buildbot/worker/package_macos64/build/src/dump.c:3272
_include_from_serialized at ./loading.jl:676
_require_from_serialized at ./loading.jl:743
_require at ./loading.jl:1034
require at ./loading.jl:922
require at ./loading.jl:917
jl_apply at /Users/julia/buildbot/worker/package_macos64/build/src/./julia.h:1631 [inlined]
call_require at /Users/julia/buildbot/worker/package_macos64/build/src/toplevel.c:399 [inlined]
eval_import_path at /Users/julia/buildbot/worker/package_macos64/build/src/toplevel.c:436
jl_toplevel_eval_flex at /Users/julia/buildbot/worker/package_macos64/build/src/toplevel.c:656
jl_parse_eval_all at /Users/julia/buildbot/worker/package_macos64/build/src/ast.c:873
jl_load at /Users/julia/buildbot/worker/package_macos64/build/src/toplevel.c:878 [inlined]
jl_load_ at /Users/julia/buildbot/worker/package_macos64/build/src/toplevel.c:885
include at ./boot.jl:328 [inlined]
include_relative at ./loading.jl:1105
include at ./Base.jl:31
exec_options at ./client.jl:287
_start at ./client.jl:460
jfptr__start_2084.clone_1 at /Applications/Julia-1.3.app/Contents/Resources/julia/lib/julia/sys.dylib (unknown line)
true_main at /usr/local/bin/julia (unknown line)
main at /usr/local/bin/julia (unknown line)
Allocations: 5898322 (Pool: 5896950; Big: 1372); GC: 7
fish: 'julia test.jl' terminated by signal SIGSEGV (Address boundary error)

Any clues?

100% confirm (sorry for spamming but this might be important). The issue occurs when evaling and specializing a function within a package from another package. Evaling leads to a warning (eval into closed module) while specializing results in segfault.

This is the code:

module EvalPkgTest

using Revise

Core.eval(Revise, :(function foobar end))

function Revise.foobar() :: String
  "foo bar"
end

end # module

This is the package:


(just a regular package created with PkgTemplates).

I think this info should go as an issue in Julia? Or Pkg?