Content Storage
What started as an effort to bring editing capabilities for Train-Paths to the Toolbox, looks like ending up in a larger refactoring of some fundamentals of FreeTrainSimulator and underlying content file formats and internal storage options.
MSTS and OpenRails are using a proprietary content format (Kuju), and beyond the original editor tools from MSTS, there is limit support editing content files to create activities, train paths, consists etc.
OpenRails has implemented a parser engine to load existing content, but no support to write or update content.
With no/limited tool support available, content builders have often started to manually updating content files, leading to inconsistensies or variations in these content files, which are difficult parse correctly.
Further OpenRails-specific storage formats, such as binary SaveGame states, and the message format used for multiplayer messaging, are all different formats, require manual parsing (not using standard tools/libraries), and have been grown over time in a mixture of content and format, making it difficult to maintain parser backend over time.
As FreeTrainSimulator started an effort earlier to simplify the SaveGame states, removing the annoying "Incompatible save state"-errors, and also testing an improved multiplayer-messaging protocol, these efforts led to the decision to leverage this same approach as core storage format for the Free Train Simulator.
As we have been looking to utilize other and open formats, including such as human-readable json to store edited content, but also internal items like settings or the GameSave state, a decision was made to use a raw .NET format at core, which could be transformed and exported to i.e. json, and/or import from other custom formats, including Kuju and the interim formats from OpenRails, as neccessary.
Using raw .NET format allows to read objects pretty quickly in memory, avoiding extensive parsing (and content error handling), once data is available in this format. Storing (writing/updating) content data is almost as simple and fast as a memory dump of the specific objects, where also reading back avoids a lot of the surrounding syntax and schema from "human readable" formats including Kuju, Json or Xml, while using an standardized approach also allows to focus on content and data itself, not on parser and error handling.
Providing new storage format has multiple goals in focus:
- Minimize and remove usage of custom parsers and their related code, from common usage paths. Currently, there are different implementations to read content files, spread across the code base. Ideally with new standardized format, all handling and code to read or write, is separated, and ideally not even managed and maintained as part of FreeTrainSimulator, but using standard libraries.
- Instead of parsing the same content (and content errors) again and again, read and convert them once into the new format.
- Unifying data persistance and implementations, also for messaging protocals like multiplayer gaming, and SaveGame state.
- Enabling editor tools like Toolbox or other potential Activity Editors, to save (write/update) content changes, through standardized set of interfaces/library, instead of implementing custom approaches for every content type, or even manually editing files.
- Providing backward compatibility through one-way import capabilitities for existing content.
- Backward compatibility for new features. Using a new internal format, will not overcome the existing limitations to create/update MSTS-style content.
- Backward compatibility with older platforms, both OS or .NET framework.
The typical approach for existing route and car content is to use existing MSTS/OR format parser to read content, and load into FreeTrainSimulator contentn format, which than will be stored separately. Existing content will not be changed or updated in any way, and is read only.
For other storage types, such as settings or gamesave states, only limit upgrade capabilities will be implemented, in most cases old structures will no longer be used while new formats are introduced.
One of the first implementations leveraging the new storage format, are the new Configuration and Settings Profiles, which allow to have multiple parallel profiles each with independent configuration and setting options. Each profile and related settings are stored in individual files in their respective folders, where they could also be exported or moved to other environments if desired.
Barely visible at first glance, with first batch of changes and the introduction of user setting profiles, all content which is needed for the Menu application to allow users selecting routes and activities to play with, are now based on the new storage format.
When starting the FreeTrainSimulator Menu application first time (from version 2.0.0-dev and newer), users will get a message to confirm the existing content. When confirming through "OK" on the Options-form, all existing content (routes, paths, activities, train cars etc) will be read, and corresponding data files cached in the user's application data folder (typically C:\Users\USERNAME\AppData\Roaming\Free Train Simulator). This process may take a little moment (indicated by a status progress bar), and is only neccessary once, or when a newer version of the FreeTrainSimulator is released.
Afterwards, the Menu application will no longer need to read any of the existing MSTS/OR content to show all avaiable content for selection, but read from FTS content store only.
In future releases, not only the menu application, but also the game itself will be able to read and use the pre-parsed content.
The new data storage format is balancing between performance and flexiblity. While new fields could be added to existing entities, in code these must be added after existing fields/properties, otherwise it will break reading existing content.
For more details, please read about MemoryPack.