Best practice code generation

General Usage
1

I want to auto-generate some structs and functions of my package KiteUtils.jl. As input I have a .yaml file. I have a file build.jl that creates a few files that are then included in the package.

How can I achieve that this build script is called when the package is installed or used?

2

Wouldn’t it be a better idea to generate the structs and functions during development and store them along the ordinary code? Don’t you want to run test against the generated definitions?

If it’s too much of work to run the code generation manually, maybe a Git Hook or a CI workflow would help?

3

Well, this contradicts the idea that only source code should be in git, and auto-generated files should not be in git.

4

Yes, but before any unit tests are executed the package is loaded, and if on package load the code is auto-generated then also the tests would work.

5

My experience with things like that is, that it’s making things very hard to find and thus debugging a nightmare

6

Why generate files at all? Julia has built-in metaprogramming where you can just generate the AST and eval it. (Modulo the usual caveat to be wary of metaprogramming.)

And you don’t have to do this ahead of time. You can do all the generation in the package code itself, e.g.

module Foo

include("generator.jl")
generate_and_eval_stuff()

...

end

This will only happen once while the package is being precompiled, so in practice the overhead of code generation is unlikely to be a problem.

6 Likes
7

Is that compatible with Documenter?

Example of a file I currently autogenerate:

The point is, I would like to have easy-to-read and easy-to-understand code that is well documented.

1 Like
8

Sure, why wouldn’t it? You can generate docstrings in the AST too.

Documenter.jl just loads the module and looks for docstrings — it doesn’t do its own parsing of the files AFAIK.

A rule of thumb is that people shouldn’t be looking at the generated code at all. They should look at what it is generated from (e.g. your .yaml file), or at the generated documentation.

(This is a nice feature of generating and evaluating the AST directly — it doesn’t clutter your source tree with generated code that no one should be looking at. It’s also a lot less fragile than generating text and re-parsing.)

1 Like
9

Main disadvantage of your suggestion: I have to learn a new syntax.

10

What new syntax?

11

Writing code that creates AST trees…

12

That’s still Julia, just creating Expr objects that mirror Julia source syntax.

But yes, generating strings seems more obvious at first. As I argued in my 2019 JuliaCon talk, generating copy-paste code is a tool that it’s very tempting to reach for because it’s only one step up from the beginner technique of writing copy-paste code manually, but it’s also one I would generally recommend against (it’s fragile and inflexible) if there is any alternative.

3 Likes
13

Expr is totally tractable! Anything you want to make, you can always do this:

# how do I create the AST for defining a struct?
julia> dump(:(struct ExampleStruct{T}
    a::Int
    b::T
    c
end))
Expr
  head: Symbol struct
  args: Array{Any}((3,))
    1: Bool false
    2: Expr
      head: Symbol curly
      args: Array{Any}((2,))
        1: Symbol ExampleStruct
        2: Symbol T
    3: Expr
      head: Symbol block
      args: Array{Any}((6,))
        1: LineNumberNode
          line: Int64 2
          file: Symbol REPL[1]
        2: Expr
          head: Symbol ::
          args: Array{Any}((2,))
            1: Symbol a
            2: Symbol Int
        3: LineNumberNode
          line: Int64 3
          file: Symbol REPL[1]
        4: Expr
          head: Symbol ::
          args: Array{Any}((2,))
            1: Symbol b
            2: Symbol T
        5: LineNumberNode
          line: Int64 4
          file: Symbol REPL[1]
        6: Symbol c

Then, you can just reference it to make whatever you need.

Also, ExproniconLite.jl makes it even easier with a very lightweight dep.

2 Likes