Create a high-level saving/loading API based on the high-level multiplayer API

[nlupugla]

Godot does not do much in the way of helping users save and load their games. On the other hand, it has a great high-level multiplayer API by way of SceneMultiplayer, MultiplayerSpawner, and MultiplayerSynchronizer, which make it possible to set up multiplayer through the editor. Multiplayer and saving/loading are similar tasks in that they both involve serializing/deserializing game data. I propose creating a similar high-level save/load API similar to the multiplayer API to make it easier for users to save and load their games.

The high-level save/load system would consist of three main components:

1. A global SceneSaver (analogous to SceneMultiplayer)
2. A SaveGameSpawner node (analogous to MultiplayerSpawner)
3. A SaveGameSynchronizer node (analogous to MultiplayerSynchronizer)

Here's how it would work. In any scene with nodes whose properties need to be saved/loaded, the user places a SaveGameSynchronizer. They configure the node to point to those properties. In any scene that will instantiate other packed scenes at runtime, the user places a SaveGameSpawner and configures it to point to those packed scenes. Finally, SceneSaver will expose global methods like save_game and load_game that can be called from anywhere.

Here's how the scene tree for a simple game involving a Player scene and some enemies that spawn on a timer (like the Dodge The Creeps tutorial game https://docs.godotengine.org/en/stable/getting_started/first_2d_game/index.html).

Main

• Player
• Timer
• SaveGameSynchronizer (pointing to Timer time)
• SaveGameSpawner (pointing to Enemy.tscn)

Player

• CharacterBody2D
• • Sprite2D
• • CollisionShape2D
• SaveGameSynchronizer (pointing to CharacterBody2D position)

Enemy

• KinematicBody2D
• • Sprite2D
• • CollisionShape2D
• SaveGameSynchronizer (pointing to KinematicBody2D position)

What do you think?

Credit: @SlugFiller for coming up with the idea and looking over this proposal, as well as @YuriSizov for participating in the discussion that spawned this idea.

[SlugFiller]

Besides saving and loading to file, this can also be used to produce game snapshots for e.g. Prince of Persia Sands of Time style time-rewind. It can also be helpful for implementing rollback netcode, although that would require a few extra pieces.

The biggest challenge, IMO, in implementing this is order of operations: You can't set properties on a node before it was spawned. So the correct order of spawning and synchronizing needs to be maintained, while making sure not to leak any memory when anything is despawned.

[AThousandShips]

To be truly useful I feel this also needs to allow you to extract the serialized data as a dictionary or array, and provide custom data directly (virtual save/load methods for example) to avoid restricting users to a specific save format and save method, for example allowing you to save in some custom way, send it over network, compress it, etc.

There's always a risk of a feature only being useful for certain situations, or certain levels of complexity, a built-in save system can easily become restrictive to users with complex setups, though providing ease of saving to relatively simple projects, or relatively new users, has its own merit

The tools already available for users with groups and similar are powerful, but can be hard to implement as a beginner

[nlupugla]

Definitely agree about being able to serialize and deserialize directly.

About the complexity, I think you could say something similar for multiplayer. If you want to make a multiplayer game at scale, you're going to need finer control than what the high level API provides. On the other hand, the high level stuff provides a good starting point for beginners and prototyping.

[SlugFiller]

What is the use case of getting the data as a Dictionary or Array instead of a PackedByteArray? Considering at least some of the data would be internal stuff to ensure the properties go into the right objects, how useful would such a breakdown be to a user?

Going over the examples you did list:

• Sending it over the network is still possible with a PackedByteArray
• Lossless compression is doable with PackedByteArray. Lossy, or data-aware compression, can be achieved the same way that is done here, near the end of the section (It's too bad this is only present in the blog post, and not the tutorial in the docs). Instead of directly setting synchronization to the object's active properties, you create a new PackedByteArray getter/setter property, that compresses the properties on get, and decompresses them on set, and synchronize this property instead.
• "Saving in a custom way" requires more details, and also a use case. Taking, for example, saving as JSON instead of a binary format, the advantages are unclear, you'd have issues due to certain lossiness of serializing to string, and you'd have issues due to certain internal ids being needed to ensure properties are set on the correct objects, being no more readable than binary. So it's all cons and no pros.

[AThousandShips]

"Saving in a custom way" requires more details

"save in some custom way, send it over network, compress it, etc." was the description I gave

I was saying that getting the data rather than being forced to save it, which is the case for some data situations in the engine, where the only option is to serialise it, I only brought it up because the flexibility is important

I was only giving feedback and minor pointers, I am aware of all these details, that's why I said having direct access was to me an important point

[mrjustaguy]

Saving a Game is as simple as keeping all the variables you want to keep in using the @export before variable declaration, and setting every child you want to save to have the owner be the root node, and simply packing the scene using PackedScene.pack(Rootnodepath) and calling ResourceSaver.save(PackedSceneVar, FilePath)

It's quick and easy to implement, performs much like editor saving performs and loading is simply a matter of changing to the scene you saved, and as a bonus you can open Saves in the Editor if you wish to Debug something or just tweak the values to Mod the Save

[nlupugla]

Or using the method described in the documentation

While that's certainly an option, it definitely has room for improvement. As you can see from reading the tutorial, the editor doesn't do much to help you save/load and you end up writing a lot of boilerplate.

In comparison, the high-level multiplayer API turns the boilerplate you would need to write into configuration data.

[SlugFiller]

Incidentally, the method used by the documentation, while not as bad, is still pretty bad around this line:

var new_object = load(node_data["filename"]).instantiate()

That's an arbitrary string. And while Godot's filesystem doesn't have something similar to data-urls, there's still a lot of funny business you can do this way.

Also, that solution is not as generic as it looks, as you can see around the setting of the position property. You would need to add such code for every property that you want to save that doesn't have a matching JSON primitive type.

[SlugFiller]

The advantage of the multiplayer API is that it was designed with security concerns in mind. After all, if anyone could connect to a game server over the internet, and send a packet that would infect it with a trojan, that would be pretty bad.

Passing a saved game might not be as quick as opening a connection over the internet, but it's still a concern to be kept in mind.

[mrjustaguy]

Performance wise it's really not nearly as bad as it is stated, I mean yeah it isn't the most efficient as there is redundancy, however it also makes sure that if the game changes a level, you still get the OG level, which preventing saves from becoming unusable, so it does have some benefits in that redundancy.

The security issue I haven't considered as I've only used Local saves and never downloaded someone's saves so security isn't an issue for such people, and those downloading this type of stuff should be well aware of the risks, though yea having this security risk mitigated from the Game side would be a Big +

Opening in another game wouldn't work unless they share the same resources, as the references wouldn't match, and even then with unique IDs for resources...

[SlugFiller]

Loading a game from a previous version of the level, and loading a game from a completely different game, aren't that different. If the resources and scripts are embedded into the scene, the entire thing would load and play.

Incidentally, if a level has changed so much that loading a save from a previous version is not an option, it's likely you've changed it for a reason, and don't want players suddenly playing a broken old version of it. By contrast, if the only changes in the level are to static elements (walls, etc), then a save that only contains entities that have moved will automatically update to the latest version. Unless the place the player saved in is now inside a wall.

Downloading saves has been common ever since fighting games decided that it's a good idea for characters selectable in VS to be unlockable via completing single player/story mode. Actually, it's been common since before saves, as strategy guides would contain passwords for starting the game in various levels. Also, it would be kind of silly if the game needs to add a disclaimer "Do not load saved games downloaded from the internet, they may contain viruses".

Loading a saved game from the internet should pose no more risk than loading an MP3 file from the internet in media player, or a PSD file from the internet in Photoshop. Microsoft has already demonstrated how bad failing to hold this basic assumption can get, by allowing viruses in Word documents.

[cridenour]

For what it's worth, the MultiplayerSpawner node is already pretty good to use for saving/loading as-is. We encode scenes as we would to send over the spawner and save the PackedByteArray as part of the save file. On load, we just send them back through the spawner, which handles any players joining later.

I can definitely see the value of the SaveSynchronizer and SaveSpawner for single player projects.

[AThousandShips]

The custom data sent to spawner has to be provided manually or by replication, there's no functionality as far as I can see to extract that manually, except as the properties in replication config

But that's really not much more than just adding properties to track and storing that, so I'm not sure how much the spawner helps with implementing this

[SlugFiller]

The input to MultiplayerSpawner isn't a PackedByteArray. It's a root node, and a list of allowed subscenes. Its output is a list of calls to SceneMultiplayer.object_configuration_add. In the opposite direction, it receives a bunch of MultiplayerSpawner.instantiate_custom and MultiplayerSpawner.instantiate_scene. Again, none of the methods along the path are exposed for calling from GDScript.

The conversion to/from byte arrays is done in methods like SceneReplicationInterface::_make_spawn_packet or SceneReplicationInterface::on_spawn_receive. But those bytes are never exposed, and cannot be used for anything other than being sent to/received from the multiplayer peer.

The spawner also uses MultiplayerSynchronizers to gather properties to be sent. But, again, none of these are present as PackedByteArray in any area of the code that's exposed to GDScript. More importantly, the gathering of all of the above from across the scene tree, into a single SceneMultiplayer is also never exposed. So, again, I'm not seeing where you get a PackedByteArray out of this...

[cridenour]

When using spawn_function you are in most cases using byte-packing (and honestly I assumed it was a requirement, but I see now it takes in any Variant and not only PackedByteArray).

By using spawn_function, you are creating a data representation of how to spawn that scene, which is what we store in our save files and use to recreate the world. It works great in single or multiplayer mode.

And I never said we could get those bytes out of the node currently, just that we would need to use them if this SaveSpawner node could be configured with autospawn like the current multiplayer variant.

[SlugFiller]

Oh. spawn_function is only used for nodes spawned using MultiplayerSpawner.spawn. The normal way to use MultiplayerSpawner is to set spawn_path, and then add nodes as children of that path from any script using Node.add_child. Any node added that way is automatically tracked, and has matching data sent across the network. However, spawn_function is not called for such a node, only to nodes added via MultiplayerSpawner.spawn. And yes, it can receive any Variant as a parameter.

From the get-go, rewriting the entire game to use spawn instead of add_child already defeats the purpose. Games written normally, without multiplayer or saving in mind, would use the latter method. Enabling spawn tracking without refactoring any of the existing game scripts is a big factor.

In other words, even if you set a custom spawn_function that serializes the data before using it, it won't be called for ordinary spawns, i.e. most spawns. It would only be called for spawns made specifically by calling the spawn function, at which point, hooking spawn_function is redundant. The only reason it's originally necessary is because it's called on both server and client with the same data. Incidentally, the marshaling used for the data is the same as FileAccess.store_var with full_objects set to false.

[cridenour]

To be honest, I don't feel like we're having the same conversation. Likely my fault for short fly by comments, but you seem to be countering points I'm not making / or at least attempting to make. What you described above is what I've been referencing by "autospawn" in previous comments. And I agree autospawn is the preferred way of using these nodes.

I'm just here to voice support for the proposal and wanted to mention that the current spawner can be used in this way now with some legwork, but this proposal sounds better. I hope to see these in 4.x someday.