Thoughts on visual scripting, and why Bevy UI should have a design goal of being able to clone VSCode

[SamPruden]

Disclaimer: This is a collection of thoughts about visual scripting. Nothing here is either tested or robustly designed. I am not an experienced user of visual scripting systems.

I make an argument for

• Why the eventual Bevy Editor should include visual scripting
• Roughly what that should look like and how it can be better that other visual scripting offerings
• A code first design that requires re-implementing something akin to VS Code's editor, which I propose is a good stress test for upcoming UI work.

Why visual scripting?

Visual scripting as its best can make programming non-intimidating to people who don't consider themselves programmers. The idea of having to learn to code deters many indies with promising design careers but without a technical background, and UE Blueprints specifically have shown that there is significant industry utility in having the feature available in an engine. It may be especially valuable for Bevy, as Rust can be an unusually intimidating language even for experienced programmers. I believe that the dense technical syntax of code is often what puts people off starting, and what VS replaces with a more accessible alternative.

Ideal features of visual scripting

This is an opinionated list of ideal visual scripting features, some common is existing implementations, some not.

• Easy and intuitive for a person without programming experience
• Skills learned in the visual scripting environment transfer into classic code based programming - learn to program "by accident"
  • Code best practices are VS best practices - break out reusable functions etc
  • A smooth ramp between "everything node based" and "everything code based" as users get progressively more comfortable with code
  • Seamless two way interoperability between nodes and code
  • Cleanly use code snippets in nodes when appropriate
• An experienced programmer should be able to use the visual scripting environment without being frustrated by how clumsy it feels - every VS I've used has failed on this point spectacularly
  • Low development overhead of managing messy node layouts
  • None of that junk where relatively simple mathematical expressions become a sea of node spaghetti
• Zero cost abstraction - performance should never be a factor
• Compiles into neat ergonomic code in the target language

High level design proposal - Code-first visual scripting

I propose that a good way to achieve this may be a code-first node based visual scripting environment that takes advantage of Rust's expression everything approach.

There would be very few fundamental nodes, the main one of which is the Expression Node. The expression node simply provides a rich text editor for entering arbitrary Rust expressions. This editor implements VSCode style rich editing support, with highlighting, autocompletion, Rust Analyzer, documentation popups, live typechecking, semantic symbol renaming, etc. This is an IDE in a node. By default this is a single line text box, but expandable to arbitrarily large multiline editing. Arbitrary code goes here.

Vaguely like this programmer art

Node grouping is done similarly to other visual scripting solutions, with a node representing a sub graph of nodes with well defined inputs and outputs. These are explicitly called functions, which is what they compile into.

There is a large library of inbuilt functions built out of the fundamental expression node, including trivialities like a multiply node that simply multiplies two floats. People can still take the everything nodes approach that they're used to from Blueprints etc, but will be gently guided towards replacing more complex node setups with equivalent expression nodes as they get comfortable doing so. As nodes simply compile into ergonomic Rust, we can have a "Convert to expression" feature to automate this for subgraphs. You can easily look inside any function to see how it works.

The idea here is to give people a node based workflow to be comfortable in, but show them how the nodes they're putting together are equivalent to pure code, and to provide them with a path to slowly convert more of their nodes into pure expressions as they grow comfortable doing so. This would be highly encouraged with nontrivial mathematical expressions, which are much neater in code than in nodes.

I hope that this makes the engine accessible to people for whom the thought of coding is intimidating, backdoor teaches them to write Rust code without ever feeling like they've sat down to study, and provides a simple and more concise way to represent mathematical expressions which often become a disaster in node based setups.

This sounds hard to build!

Yes, but I'm spinning that as a positive.

Including a VSCode style text editor in the nodes sounds like a pretty big task. It would probably require implementing a text editor in Bevy UI with full support for the Language Server Protocol. This is a significant amount of work. If we look at the DOM behind VSCode, there's a huge number of elements used; nearly one per symbol because of the syntax highlighting being done with separate elements. There's a lot of rich interactivity and live data binding going on, and all of this has to run smoothly in real time. This would probably be the most complex widget implemented in Bevy UI by a significant margin.

If the upcoming UI work were to set the goal of being able to support this, that should ensure that it can do anything. This is a hardcore dogfooding stress test. Supporting this could force Bevy UI into a very high quality implementation.

Conclusion

These are just some high level thoughts that I've wanted to get out for a while. There's a lot more detail to go into in terms of how you handle things like branching, ECS specific features, the output format of the node graph etc but this is just about broad ideas. Obviously this is a fairly far future feature, but I'm justifying posting this now because of the tie in to the UI work.

I'm interested to hear what people generally think of this as a potential direction!

[ywmaa]

I really like your ideas, I am thinking in trying to do this myself.
I really love to have visual scripting in game programming (yeah it isn't the best for programming but it does get prototyping really fast)
I liked blueprints in unreal engine but I wanted a visual script were it converts the script to actual code.

and I think it would be awesome to have the visual script to analyse Rust written code to visual script .

even though a lot of programmers don't like visual script but visual script is appealing to designers

so if a designer in the team wants to look at something in the code or modify it he could just use visual script as a tool.