Hypothetical visual programming in Julia - design?

I see the discussion of visual programming pop up in other discourse threads periodically and I wanted to have a single thread about what the design of a visual programming environment for Julia would/should look like. For the record, I’m glad this isn’t the focus of the core Julia team; there are much more important things than this. I’m not that interested in why or why not a visual programming for Julia is a good or a bad idea. I want this thread to focus on the hypothetical design. What are the best visual analogs for Julia’s design?

If enough ideas come together in this thread, I might spend some time to make a basic visual mock-up of what it could look like.

==============

Here’s a few ideas of my own. Take them or leave them. My experience with visual programming is limited to LabView, Blender nodes, and some Simulink.

This seems relatively basic, but I like basic node design of Blender shader nodes. The nodes are functions with input and output ports colors by the input/output types. If a port is unconnected, it’s value can be edited directly in the node with an appropriate user interface (color selector for color input, slider bar for bounded ranges, etc). You don’t really need a separate user interface because the nodes are convenient enough to edit directly. This only works if the types of inputs are known, but I’ll get to that below.
2021-04-03 14_46_16-Blender

I deeply despise LabView, but I like the edges colored by type. To support any possible type, it would be cool to hash types into an edge styles (color, thickness, dashes). I think some styles should be hard-coded: e.g. something with the type Any should probably black or gray. Also I like that the line thickness scales with the dimensionality in LabView. This might not be everybody’s preference; it could turn into a spaghetti rainbow. I think it would help with visually debugging.

By the same logic, ports on function nodes would be colored by the types of inputs. With multiple dispatch, many ports may be gray and/or hollow indicating they support many types until you start plugging in edges with known types. It would make sense to be able to set or lock the input type of a port. If you manually set the input type of a unconnected port, you should be able to edit the value in that port with an appropriate editor. For example, perhaps you want to lock one port of a + function to be an Integer you would be able to type in or scrub through different integers. If instead you set one port to be a DateTime type, then you could edit that value with a date-time selector.

The title bar of the node could help you figure out which method of that function you’re calling. For example if you haven’t specified enough inputs the title bar could tell you how many matching methods there are currently. Once you’ve connected enough inputs the title bar could indicate which method is being used.

Which side of a function node that a port is on could have a meaning. For example if you want the flow to generally go from left to right, port on the left could be inputs, ports on the right could be outputs. This leave the top and bottoms as free real estate. I think there should be dedicated output for the node or frame itself. As example use case, you could define your own function as set of blocks in a frame or by writing it directly in Julia. You should be able to plug that function into other functions (e.g. Optim.jl).

One big challenge I see is how to make a visual programming env. for Julia that is simultaneously generic and suitable for specific applications. The design for visual modeling of dynamical systems (eg. Simulink) is quite different than the design for data processing (ie, Luna). Possible solution: have a frame around some application-specific nodes that acts like a macro begin ... end block. This way all sorts of applications for visual programming can exist as visual DSLs within a more generic visual programming env. A “frame” around nodes may be the right visual analog for macros in Julia. So beyond having a frame for say… visually creating a reaction network à la Catalyst.jl, you could also drop a @time frame around a set of nodes.

Some other ideas and comments:

  • Variable could be implemented as small blocks which essentially name an edge. I’m interested in hearing alternatives. Edges wouldn’t need to be named to work; basically the default would be a bunch of piping operators. Named edges should also allow an edge to be referenced at some other location in the env. without having to route the edge all the way across the screen.
  • Creating a script from the visual env. should be really direct and easy. It seems more difficult to me to turn a script into nodes and edges but hopefully still possible. It should also be easy for blocks of code to coexist in the visual env.
  • It should be simple to write a function and have it turn into a custom function node.
  • It should be simple to group together a set of function nodes and create a new function node (ie, making a function by grouping together selected nodes)
  • There should probably be a different visual design for mutating and non-mutating functions.
  • A function that can take many arguments could give you an extra port as you plug in inputs. Example: + might give you one port when you first place it. If you plug something in, a second port would appear, plugging a second thing in would make a third port appear, etc. Alternatively, a circular port could get stretched out into a pill shape.
  • There should be a visual splat operator to go with the comment above.
  • There are a few ways to represent control flow visually. I’m interested to hear what you guys think is the best way might be. As frames? As loops and branches of edges? Something else?
  • There’s a lot of cool tricks that could be done to help debug and profile code. E.g. the title bars of functions could become bar graphs for how much time they take after you’ve profiled you code.

I know this is a lot of random stuff and it would be a lot of work to implement. I’m interested to hear your ideas. Perhaps this discussion will turn into a roadmap for one or more ambitious contributors.

Final note: Is there a way to hack together a Julia node editor in Blender as a proof of concept or that more work than it’s worth? I’m an engineer not a computer scientist and have little sense of how hard that might be. I generally like Blender’s UI with workspace tabs flexibly divided into areas for different editors.

9 Likes

There’s a long history of these. E.g. The Whole Code Catalog

2 Likes

Nice overview! Are there any particular visual programming details or paradigms that you think fit in well with Julia’s design?

Just as another possible source for ideas, the Unreal Engine has a pretty nice visual scripting language.

The future of coding group Slack might have some thoughts.

I made a really quick script which creates a line style from a variable’s type.

Line color: I wrote a function which hashes an input into a color. Wires are colored by the type of the variable. Wires carrying arrays are colored by the type of their elements. The only “preset” color I have in this demo is for functions. I set all functions to be the Julia logo purple. I randomly decided for a Dict’s line color to be the color of its keys.

Line style: The line style to be dashed for functions and dashdot for Dicts. The line is solid for everything else.

Thickness: The line thickness is set to the dimensions of the array.

I tested this on a few variables with custom types (pasted from the ThinkJulia examples). I like that a user could write their own type and have wires carrying that type be easy to identify.

This was thrown together to demonstrate the idea. There’s a lot of arbitrary decisions made and room for improvement.

function colorhash(input)
    h = hash(input) >> 40 #remove 40 bits to 24 bit number

    # Bitmask and shift to convert three numbers btwn 0 and 255
    r = (h & 16711680) >> 16
    g = (h & 65280) >> 8
    b = (h & 255)
    rgb = [r,g,b] ./ 255
    RGB(rgb...)
end
logocolors = Colors.JULIA_LOGO_COLORS
jlblue, jlred, jlgreen, jlpurple = logocolors.blue, logocolors.red, logocolors.green, logocolors.purple

function type2linestyle(input)
    # Defaults
    lw = 1
    color = colorhash(typeof(input))
    ls = :solid

    if input isa AbstractArray
        lw = ndims(input)+1
        color = colorhash(eltype(input))
        ls = :solid
    end

    if input isa Dict
        color = colorhash(eltype(keys(input)))
        ls = :dashdot
    end

    if input isa Function
        color = jlpurple
        ls = :dash
    end

    lw, color, ls
end
2 Likes