Data architecture

[QuintillusCFC]

Separating out from the discussion at #234

Tagging @WildWeazel , @pcen, @maxpetul , @JimOfLeisure as parties who have expressed interest or been tagged, but open to everyone else, too.

One of the points of contention has been the data architecture. Currently, everything that is written out to the save file lives in C7GameData. I like this separation. It's clean architecturally. It makes it clear what is part of the save format. You can still have object-oriented methods on the data objects, such as TurnsUntilGrowth on City.cs.

As a side benefit, it keeps the coupling between the engine and the data loose. Someone else could use the game data without having to use the whole engine. Maybe we put our map generator in a separate project, and then someone can create their own map finder, like Moonsinger did for Civ3.

However... and this is where I'm not the best person to explain... not everything is perfect. We've been adding extension methods in the engine, such as MapUnitExtensions.cs, that add methods to e.g. the MapUnit that are not in the C7GameData, like the aforementioned TurnsUntilGrowth. I believe this is because many of these methods call out to animation methods. E.g. MapUnitExtensions's Fortify method does this in a straightforward way.

This makes sense; animation does not belong in the data, and should not be stored out to the save file, so it does not logically belong in C7GameData. The extension methods also help make sure that the animations are consistently played; when the AI calls unit.fortify, the animation plays, for example, and it's nicely integrated into the unit's effective API.

---

So, what's the direction to be? Personally, the balance feels largely right to me. My understanding of extension methods is based on Kotlin, which states:

Kotlin provides the ability to extend a class with new functionality without having to inherit from the class or use design patterns such as Decorator. This is done via special declarations called extensions.
For example, you can write new functions for a class from a third-party library that you can't modify. Such functions can be called in the usual way, as if they were methods of the original class. This mechanism is called an extension function. There are also extension properties that let you define new properties for existing classes.

C7GameData is essentially our library, where and has core functions, just like the "string" class in C#. Obviously we can modify it, but that doesn't mean there isn't value to separation.

But there are tradeoffs. I believe @maxpetul has mentioned something about extension methods not being overrideable in C# mods, for example, and "should this belong in the extension or the base class?" is a bit of an art, not a science.

I'm also still thinking through the AI applications as well; while I like that the separation makes it clear that the AI data is written to the JSON save file, and the AI processing classes take instances of that data (and other data) to act on, there will need to be some more flexible linkage that says (for example) the DefenderAI uses this specific data class. This also factors into how to make AI files moddable, e.g. not in the core engine at all, but loadable via DLL or equivalent, and still able to put long-term AI state data into the save file.

I'm particularly curious what @WildWeazel thinks as the original architect. It seems to be an area where Flintlock and I had different ideas, both with reasons behind them, and have created something bifurcated, with newer developers inheriting that and rightfully having questions about why things are the way they are.

[JimOfLeisure]

Not a direct answer to your post, but some thoughts: I'm multiple months behind the codebase now, FWIW.

One challenge with the 'just serialize C7GameData' is that it completely breaks backwards/forwards compatibility with save files whenever the data structure changes. But I have no easy ideas for getting around that; keeping "upgrade readers" around and maintained is extra boring work we probably don't need in the near term, and that goes double for a non-code-structure tied save format. I figured something would eventually emerge as a good idea once we have a better idea of what a completed game will look kind of like, data-wise.

I did put some thought into how a Lua mod for example might put non-pre-structured data in the save. There are a couple of ways; I can't recall them well enough to describe them offhand, but I think it involved declaring a var type and then ad-hoc serializing/deserializing some on-the-fly data. It would be a little extra work but not as much as a generic decoupled save format.

My very vague general idea was hooks in each class; hooks for some extra save file data storage and hooks into method queues of sorts so that a C# or Lua mod can stick its code in there to be run in some form of handler queue method. But again, it's a very vague idea, and I haven't begun to mentally prototype it much less make some example code. The most obvious hooks would be in serialization and deserialization for injecting/reading save data for mods or other non-core functionality.

[WildWeazel]

Mostly responding first to @maxpetul from the original issue. The whole unit creation thing is a great starting point to thinking about this stuff.

My first instinct for object creation is to go with the GetInstance method. That's basically a unit factory, where we can do whatever logic we need to build an appropriate object. The builder pattern also comes to mind, because there are a lot of optional things you might want to specify/change before getting the object back (eg. UnitBuilder("Warrior").SetExperience("Elite").SetName("Conan").Build()), and don't want to couple that too closely to the Unit API. I'm not sure which is better here, or if it makes sense to combine them. On second thought: once we get rid of these hard-coded units and use real data, the only times a unit would be created would be when reading a scenario/save or newly spawned, right? The former is just deserializing raw data which can be handled by the Civ3/C7 data class itself (see below), and the latter has a small fixed set of parameters perfect for a factory method. So the builder pattern seems like overkill.

I guess I don't have a strong opinion on the empty object pattern vs null checks in general, but in C# you have null-conditional (.?) and null-coalescing (??) operators to make the latter simpler. I don't really see the usefulness of empty objects for what I'm going to call "standalone" objects though. Experience levels, sure- that's a property that you'd expect every Unit object to have. But in a fleshed out game where would you see an empty Unit or City object? Also FWIW I'm reminded that if we so choose we could have extension methods that work on null references and check internally, because they're actually static class methods passed an implicit reference. But that takes a very consistent API to rely on as a solution.

What's the current relationship between the engine and game data module and why couldn't the game data create objects? (I'll go look.) I think UnitPrototype and the method to create one should be IN the game data. I guess I didn't comment in the original thread but I agree with Quintillus' reasoning for separating logic and data. (A mod editor would be another external project to use the data classes.) To that end, refer to my original architecture sketch: the Core (game engine/logic) depends on the data layer, which contains all of the structures that are populated by mod data. These are the equivalent of the business and data access layers respectively in a traditional application. The core doesn't define anything itself, and the data layer doesn't "do" anything except CRUD. The Civ3 and C7 I/O layers use the same API to load/save/convert data between their respective native file formats and the types defined in the data layer, probably by subclassing the data types with their own (de)serialization logic.

Regarding game logic methods, per my design they'd be not static methods but Component (I know.... getting to it) instance methods which would themselves belong in a static container. That may seem like splitting hairs but it avoids many of the problems with having everything be static/singleton.

[QuintillusCFC]

My thinking has evolved a little bit since the last posts (also, late April was about when my involvement went down). Mainly as I start to add more AI classes, thinking about how they can be extensible as well (so new ones could be added by modders without recompiling the whole program).

The central problem that has me thinking about things is how do we allow custom data to be saved, beyond what's in our code in the data classes?

One of the challenges is serialization. Right now we're using System.Text.Json, but it doesn't really support polymorphic serialization and deserialization. Kind-of the former but really not the latter unless you write custom de-serializers for every class. Which is a decent amount of work to ask for mod writers, especially as they may not be as technical as we are. References:

https://docs.microsoft.com/en-us/dotnet/standard/serialization/system-text-json-polymorphism
https://docs.microsoft.com/en-us/dotnet/standard/serialization/system-text-json-converters-how-to?pivots=dotnet-6-0#support-polymorphic-deserialization

However, there is an alternative. According to Microsoft (https://docs.microsoft.com/en-us/dotnet/standard/serialization/system-text-json-migrate-from-newtonsoft-how-to?WT.mc_id=DOP-MVP-4039670&pivots=dotnet-6-0), if we use Newtonsoft JSON, it does support polymorphic serialization and deserialization.

The tradeoff seems to be that System.Text.Json is faster, and doesn't require an additional dependency. My inclination is that loading one JSON file per save, we probably don't need top-notch performance, and being able to deserialize polymorphic classes easily brings significantly more benefits.

There's also another alternative, which is what I've been doing with StrategicPriority so far - store the data, at least in the data layer, as a Dictionary (Map) of properties. See https://github.com/C7-Game/Prototype/blob/54081a035a92695339dbb11a3f76312e51f160fb/C7GameData/AIData/StrategicPriorityData.cs for an example of this. So far, for AI data, this is working. But looking at the UnitAI code, I realized that there are good reasons that it wants to store things like TilePath's, and Tile's. It kind-of works sometimes to store maps where the values are guids, but it doesn't mean it's always the intuitive answer. In theory the UnitAI example could re-inflate flattened-to-a-map data to the objects after load, but that's essentially custom deserialization without it having to be in a JSON converter.

So far I plan to finish up the StrategicPriority work with the Dictionary/Map alternative, but assuming we don't run into any integration problems with it, I think Newtonsoft is the way to go longer term.

(Creating a separate post to reply to other comments)

[QuintillusCFC]

Jim is right that a disadvantage of automatic deserialization is that it may cause compatibility issues across versions. In my observations in this era of games that are supported over a period of years (e.g. Factorio, Europa Universalis), the most common method seems to be keeping support within a major update, but requiring using old updates to load saves thereafter, and making it relatively easy to load old versions. E.g. I can't load a Factorio 0.15 save in Factorio 1.1, but having bought the license I can go download Factorio 0.15 and fire up my old saves that way. This seems like a sensible balance to me. All the more so at this point where I doubt we have players starting games that they'll still be playing in 2029. But if they are, they can always play it in Carthage Preview 1.

---

Indeed, once things are more built out, units will only be created on game/save load, and on spawn. Off the top of my head, I don't see any reason why the data layer couldn't create objects. But I haven't been focusing on that lately.

Flintlock wrote in the thread this spawned from:

The issue is that almost all of the game logic will need to be written as static methods, extensions or not, inside the engine module so they can have access to the game data in EngineStorage. For example, anything that relies on units' effective strengths (including defensive bonuses) will need to be in the engine since defensive bonus values are specified in the scenario data. It works but it's awkward. The more natural thing would be to put all that code in class methods.

It's interesting and I'm not entirely sure if it's a problem. The way I see it, the mechanics should generally be in the engine (or, as we become more modular, in various pluggable components that effectively become extensions of the engine). The data layer should define the basic structure of what gets written to disk, and for built-in classes such as MapUnit, some specifics as well.

Thinking about things being pluggable/extensible is causing my thinking to shift a bit though. For pluggable StrategicPriority implementations, my thinking is it makes sense for any implementations to be able to include both behavior and custom data in one place - if only because it isn't feasible for a third party to directly add it to the data classes (which would require re-compiling the whole program). Thus my current thinking is:

Data layer: Any built-in data, base classes for extensible data
Engine: The game mechanics, e.g. turn resolution, combat, AI
Pluggable components: Custom logic that can be plugged in to the engine via APIs, e.g. I've written a couple AI modules of this style, integrated via an API (though not yet the logic to make them actual separate DLL-style components, so for now they still live in the Engine)

Pluggable components may have both additional data that gets serialized with the data layer (this is where I propose Newtonsoft as a solution), and logic that is invoked by the engine, calling APIs those plugins implement.

[WildWeazel]

While we're on the topic I just want to mention (not endorse, yet) Odin serializer. It's the backend for a popular Unity plugin so I'm not sure how helpful it would be on its own, but I've had it in my pocket for reference.

[QuintillusCFC]

Interesting. At first I thought you meant the Odin that lets you run Win32 programs on OS/2. Hadn't heard that the Norse god had a new project. It claims to be about twice as fast as Newtonsoft (JSON.NET), although with larger file sizes. I think the bigger question will be how easy is it to use outside of Unity. I didn't find an easy answer in 10 minutes of exploring.