Full server snapshot/reload support

[Roudenn]

Mega-PR that fixes #2406
Requires #6189

Overview

The main idea behind this PR is that if your content codebase will support full game saves, you can run a command that will save everything in Content.Server/data/Saves/[insert date and time here]/ folder, so you can shutdown the server, then turn server back on, load the save file, and have everything running like shutdown never happened.

Also it can be used for server rollbacks. For example, when someone griefs the whole game for everyone, an admin can just load an autosave to revert things as they were before the raid.

Technical Details

This PR adds GameSavesSystem EntitySystem, savegame and loadgame commands, and a CVar to enable/disable autosaving before server shutdown or just disable game saves on the server for everyone except the host.

GameSavesSystem just serializes all entities into 1 YAML file, and then compresses it using ZSTD to reduce the size quickly, or decompresses and loads the file back if loading.
So basically. it's just a wrapper for things that are already implemented in the engine.

Since RobustToolbox has ECS structure, if we save all entities as they were at some moment of time, and then spawn the back the same way, it should also continue working the same way. So instead of doing the path that Replays took by just writing every NetMessage into all files, we save only the state of the simulation itself.

That means all EntitySystems shouldn't have any data saved inside them (except for caches), since they are disposed on shutdown and will not get saved into the file without proper ECS implementation. So, some (all) codebases will have to fix their code before they can turn game saves on.

Task List

• Basic functionality for saving/loading all entities at once Already done in Add support for including map/game saves in replays. #6189
• Basic GameSavesSystem
• Server-side CVar to control game saves functionality
• ZSTD compression and decompression

PRs to do next

Even after full game saves are supported, there's still a lot of work to do on the engine side, such as:

• EntitySerializer optimizations, it's too slow and because of that it makes implementing some features like dynamic autosaves impossible.
• Code analyzers that will throw warnings (or even errors) if you make your code 100% unsavable (example: using Timer.Spawn() in EntitySystem) also will be nice, but i think they need a separate -1 difficulty category tbh, they are indeed VERY hard to implement.
• Save migration support: Right now SS14 has some basic functionality for migrating EntProtoIDs, but that will definitely be not enough. Since the game is constantly updating, systems get refactored all the time, and game saves can become broken quickly. Some codebases may require some tools to migrate these saves, at least as C# scripts that are automatically executed before all saves are loaded to fix them. We can't implement migrations until we will have actual problem examples, that's why it's something we should worry about later and not now.

Impact on SS14

For saves to be actually supported at least in SS14, it will require a lot of content changes, since game saves will have no mercy on any code that violates ECS principles (and this has some potential to nuke downstreams).

When i successfully saved a Bagel station with Nukeops gamerule in SS14 using tools from this PR and content fixes, the resulting file weighted 16 megabytes and had tons of bloat. We have to improve custom serializers to have less bloat to reduce the file size and make it faster.

[VerinSenpai]

not the hero we deserved, but the hero we needed

[ElectroJr]

I know its still a draft, but IMO its probably better to split this into multiple smaller PRs and IMO the maploader changes are can just be merged on their own, so I reviewed it anyways.

There are a couple of things that I think should be changed and either #6189 or this needs to be updated to prevent conflicts (sorry again), but otherwise I think its fine to merge.

[ElectroJr]

Generally new server-side commands should be toolshed commands, and they should be a lot easier to create / have less boilerplate. Though in this specific case, I know toolshed still needs to implement completion option generation for file paths, so its probably fine to just keep it as is.

[ElectroJr]

I think the flushing behaviour should be determined based on a command argument. Because I've needed a similar non-flushing command before. And to avoid people accidentally wiping the server when they load files, the argument should either be optional and default to non-flushing, or be a mandatory argument.

It might seem weird to "load a game" without flushing existing ents, but I'd want to support scenarios like an admin loading up a save from a previous nukie round to use the old station as a kind of ruin/salvage area.

[ElectroJr]

There should be a variant that has an out LoadResult, so that content can do something with the loaded maps/grids.

[ElectroJr]

AFAICT this is the only method that actually overlaps with #6189, and other than my handling of the IsSerializable checks being rather messy, I prefer my implementation for a couple of reasons

• This finds all root / nullspace entities and uses SerializeEntitiesRecursive(), That method uses a lot of LINQ and wasn't really meant to be used with a large set of root entities (and there probably are a large number of nullspace ents)
• My implementation has a variant that just returns a MappingDataNode instead of writing a file, and something like that is needed if you want to save to a non-standard location (e.g., replays), or allow content to add data to the main save file.