I see the discussion of visual programming pop up in other discourse threads periodically and I wanted to have a single thread about what the design of a visual programming environment for Julia would/should look like. For the record, I’m glad this isn’t the focus of the core Julia team; there are much more important things than this. I’m not that interested in why or why not a visual programming for Julia is a good or a bad idea. I want this thread to focus on the hypothetical design. What are the best visual analogs for Julia’s design?
If enough ideas come together in this thread, I might spend some time to make a basic visual mock-up of what it could look like.
==============
Here’s a few ideas of my own. Take them or leave them. My experience with visual programming is limited to LabView, Blender nodes, and some Simulink.
This seems relatively basic, but I like basic node design of Blender shader nodes. The nodes are functions with input and output ports colors by the input/output types. If a port is unconnected, it’s value can be edited directly in the node with an appropriate user interface (color selector for color input, slider bar for bounded ranges, etc). You don’t really need a separate user interface because the nodes are convenient enough to edit directly. This only works if the types of inputs are known, but I’ll get to that below.
I deeply despise LabView, but I like the edges colored by type. To support any possible type, it would be cool to hash types into an edge styles (color, thickness, dashes). I think some styles should be hard-coded: e.g. something with the type Any should probably black or gray. Also I like that the line thickness scales with the dimensionality in LabView. This might not be everybody’s preference; it could turn into a spaghetti rainbow. I think it would help with visually debugging.
By the same logic, ports on function nodes would be colored by the types of inputs. With multiple dispatch, many ports may be gray and/or hollow indicating they support many types until you start plugging in edges with known types. It would make sense to be able to set or lock the input type of a port. If you manually set the input type of a unconnected port, you should be able to edit the value in that port with an appropriate editor. For example, perhaps you want to lock one port of a + function to be an Integer you would be able to type in or scrub through different integers. If instead you set one port to be a DateTime type, then you could edit that value with a date-time selector.
The title bar of the node could help you figure out which method of that function you’re calling. For example if you haven’t specified enough inputs the title bar could tell you how many matching methods there are currently. Once you’ve connected enough inputs the title bar could indicate which method is being used.
Which side of a function node that a port is on could have a meaning. For example if you want the flow to generally go from left to right, port on the left could be inputs, ports on the right could be outputs. This leave the top and bottoms as free real estate. I think there should be dedicated output for the node or frame itself. As example use case, you could define your own function as set of blocks in a frame or by writing it directly in Julia. You should be able to plug that function into other functions (e.g. Optim.jl).
One big challenge I see is how to make a visual programming env. for Julia that is simultaneously generic and suitable for specific applications. The design for visual modeling of dynamical systems (eg. Simulink) is quite different than the design for data processing (ie, Luna). Possible solution: have a frame around some application-specific nodes that acts like a macro begin ... end block. This way all sorts of applications for visual programming can exist as visual DSLs within a more generic visual programming env. A “frame” around nodes may be the right visual analog for macros in Julia. So beyond having a frame for say… visually creating a reaction network à la Catalyst.jl, you could also drop a @time frame around a set of nodes.
Some other ideas and comments:
- Variable could be implemented as small blocks which essentially name an edge. I’m interested in hearing alternatives. Edges wouldn’t need to be named to work; basically the default would be a bunch of piping operators. Named edges should also allow an edge to be referenced at some other location in the env. without having to route the edge all the way across the screen.
- Creating a script from the visual env. should be really direct and easy. It seems more difficult to me to turn a script into nodes and edges but hopefully still possible. It should also be easy for blocks of code to coexist in the visual env.
- It should be simple to write a function and have it turn into a custom function node.
- It should be simple to group together a set of function nodes and create a new function node (ie, making a function by grouping together selected nodes)
- There should probably be a different visual design for mutating and non-mutating functions.
- A function that can take many arguments could give you an extra port as you plug in inputs. Example:
+might give you one port when you first place it. If you plug something in, a second port would appear, plugging a second thing in would make a third port appear, etc. Alternatively, a circular port could get stretched out into a pill shape. - There should be a visual splat operator to go with the comment above.
- There are a few ways to represent control flow visually. I’m interested to hear what you guys think is the best way might be. As frames? As loops and branches of edges? Something else?
- There’s a lot of cool tricks that could be done to help debug and profile code. E.g. the title bars of functions could become bar graphs for how much time they take after you’ve profiled you code.
I know this is a lot of random stuff and it would be a lot of work to implement. I’m interested to hear your ideas. Perhaps this discussion will turn into a roadmap for one or more ambitious contributors.
Final note: Is there a way to hack together a Julia node editor in Blender as a proof of concept or that more work than it’s worth? I’m an engineer not a computer scientist and have little sense of how hard that might be. I generally like Blender’s UI with workspace tabs flexibly divided into areas for different editors.
