Fastest way to define high-level functions in a text file and use them in Julia after processing?

General Usage
1

I’m trying to design a system where users write function definitions in a .txt file, using some higher-level or simplified syntax. Then, in my Julia code, I want to read that file, process/parse those definitions, and use the resulting functions in Julia.

What would be the fastest or most idiomatic way to do this in Julia?

More specifically, I’m wondering:

  • Should I parse the text and generate Julia expressions/functions dynamically?
  • Is there a recommended approach for safely evaluating user-defined function definitions?
  • Would macros, metaprogramming, or some kind of DSL be the right direction here?
  • If performance matters, how should I structure this?

My main goal is to let users define functions in a simple text-based format, then make those functions usable from Julia code with minimal overhead.

2

Welcome to the Julia forum.

This sounds like a domain-specific language (DSL) problem.

  1. Only if you really need a separate external syntax. Otherwise, I would strongly prefer an internal DSL in Julia, so users write a restricted high-level Julia-like syntax and you lower that to ordinary Julia code. In other words: if Julia can already parse the surface syntax, let Julia be the parser. You can move to an external DSL if you need a very non-Julia syntax or a fully separate language.

  2. I’m not sure about this one.

  3. Yes, this is basically a DSL problem. My default recommendation would be an internal DSL first, not a separate .txt language (external DSL). Macros/metaprogramming are useful if the user-facing syntax can stay close to valid Julia syntax.

  4. Parse and lower once, then reuse many times, while ensuring that the generated Julia functions remain type-stable.

2 Likes
3

If you’re trying to design a system where users can write text to define functions using a high-level syntax, you’re defining a programming language, but you already have one in front of you :slight_smile:.

Macros are extremely powerful. I would recommend using MacroTools.jl; see here for a recent example of some work, funnily enough, to define a DSL for parsing.

You could, in theory, define a macro to read a file, parse into Julia expressions (using PikaParser.jl or similar), and return that as a quote node, which would allow you to write in your desired syntax; for me, however, this would be a last resort.

What, exactly, are you trying to accomplish? More specifics would be helpful here, as advice can be more targeted.

2 Likes