Asset cooking

Asset cooking is a process that can yield great benefits but it's an advanced technique so to use it correctly (and to avoid making the situation worse) you must understand the theory behind it. Note that that article was written before a workable way to cook mod assets was found.

IMPORTANT: Do not skip or skim the link above. You will not understand what this article talks about

Also be aware that if you didn't fix the TLE issue with your SDK, the cooker will simply crash without any output. You can find the instructions on the wiki

Use cases for cooking

Cooking mod assets works well in 2 cases:

• When all the assets are custom (made by you)
• When you reference game's assets which are not part of a seekfree standalone package

In the first case you can expect ~10x package size reduction on average. In the second case, inlining the SF standalone missing assets into your mod is likely to take less space than including the original .upk from SDK (the editor).

In case your assets reference objects that reside within packages that were already shipped as SF standalone with the game, the only benefit would be a slight loading time boost for players with a HDD, but the price you pay is bloating your mod by duplicating game's assets.

Cooking your assets

Heads up: Cooking has been tested only with the full_content version of the SDK. Cooking will likely fail otherwise, even if you plan to cook exclusively custom assets

As mentioned before, not all assets are good candidates for cooking. As such, the build script expects a new folder in your mod - ContentForCook. The maps and packages there will be run through the cooker and only the SF versions will be shipped with your mod - not the originals.

Important: assets in the ContentForCook directory cannot reference objects from your packages inside the Content folder - the cooker will simply not find the latter. This limitation is on purpose - it prevents you from accidentally duplicating your own assets

After you prepared your assets for cooking, you need to tell the build script what you want to do with them. To do that, you must first setup ContentOptions.json and then set the cooking options (setting any of them enables asset cooking, otherwise it's skipped).

• sfStandalone - each element of the array is the name of a package (sans the extension, .upk) which will be made SeekFree Standalone
• sfMaps - each element of the array is the name of a map (sans the extension, .umap) which will be made SeekFree
• sfCollectionMaps - extremely advanced, see section below

Remember - not every package needs to be seekfree. It's perfectly fine (and intended) to have packages which are meant to be only referenced by other packages (or levels) and never loaded directly. Since the cooker will "pack" everything together anyway, you can separate your packages however you wish - by content type (e.g. specific packages for animations, materials, meshes, etc.), by feature/purpose/segment/part, by max size, etc.

Collection maps

DANGER: If you misuse this feature, your assets will simply not load at all

Sometimes you want to be able to load arbitrary assets from several packages at the same time. A simple example would be several custom UI elements which you expect the player to see while in strategy, with a package per each UI element. The simplest solution would be to make each of those packages SF standalone and add them to the MapContent (see Textures loaded by UI). However, there are 2 slight issues with this approach:

• Players using an HDD will experience longer loading time (due to the HDD jumping around to read the different packages - the exact scenario SeekFree approach was designed to avoid)
• (Different use case) If more than 1 of these packages reference the same shared asset, said asset will be duplicated - inlined into each SF package

A "simple" approach would be to create a "collection package" that will have nothing but an ObjectReferencer (or a set of them) to cause the cooker to inline the objects from the individual "source" packages. The degree to which this approach is laborious and error-prone cannot be understated as you have to list each and every object individually. Instead, we can use UE3's PackagesToForceCookPerMap to cause our packages to be made SF and packaged together into a single file - the "map". In this case, the assets will always be available as long as the map is loaded.

Here's an example from the Covert Infiltration mod:

"sfCollectionMaps": [
	{
		"name": "StrategyCollection_CI",
		"packages": [
			"GeoscapeMesh_CI",
			"UILibrary_CI_ChainPreview",
			"UILibrary_CovertInfiltration"
		]
	}
]

This will create a single SF map - StrategyCollection_CI - which will include everything from the 3 packages.

You don't need to provide a CollectionMap.umap (e.g. StrategyCollection_CI.umap in the example above) - the build script will provide it for you, since in most cases it will be empty. If you do want something in the map itself, just create the file yourself inside the ContentForCook directory and it will be used instead

Important: the assets inside the collection will be available only while the map is loaded - otherwise the game will simply have no idea where to look for these assets

Loading collection maps

MapContent

Continuing the above example, we want our StrategyCollection_CI to be always loaded while, well, in strategy. We can use our trusty Content.MapContent config while abusing the fact that maps are also unreal packages:

[Content.MapContent]
.Map=Avenger_Root
.Package=StrategyCollection_CI

Other approaches

Any way to load a map will work for this scenario. For example, you can manually control the loading state using

`MAPS.AddStreamingMap();
`MAPS.RemoveStreamingMapByName();

NOTE: While there is no reason to believe that the above will not work, it has not been tested

Cooking output

The cooking process will add a new folder to the final version of the built mod - CookedPCConsole. It will contain the seekfree packages (all extensions will be changed to .upk, even for maps) and the .tfcs - if you cooked any streaming textures (e.g. UI textures are not streamed and hence not moved to .tfcs).

However, the actual cooking process doesn't place files there - instead the cooker outputs to SDK\XComGame\Published\CookedPCConsole. If you pay close attention, you will notice a couple of differences between that folder and your mod's CookedPCConsole:

• Standalone SeekFree packages have a _SF suffix added (e.g. MyPackage.upk -> MyPackage_SF.upk). This is the expected behavior (and its how the game's own packages are shipped) but for some reason the game fails to find the suffixed packages when they reside in a mod, so X2MBC renames them back to the unsuffixed/original file name
• There is a GuidCache_ModName.upk file. It is not shipped because it's only useful for unreal networking (which X2 doesn't use) and FXS's changes to the engine cause the game to hard crash if it attempts to load a DLC/mod's GuidCache
• There are also the cooking output files for other mods (including CHL, cooking which leaves a lot of files) - SDK\XComGame\Published\CookedPCConsole is shared by all uses of the cooker

Generally, you do not need to manually touch the Published\CookedPCConsole folder and its contents. The files there are cached/kept in order to avoid recooking packages whenever possible (e.g. if nothing changed since previous cook). Depending on how many assets you cook, this could be a significant time save during the build process.

If you want to do a full recook of a specific mod, use X2MBC's clean functionality and it will remove files that are specific to that particular mod. If you do want to do a manual full reset of cooking, it's a good idea to delete the entire folder since some files might reference other files (e.g. upk -> tfc) and incorrect manual cleaning can lead to incorrect behaviour or crashes in game.

Pitfalls and peculiarities of cooking

First of all, cooking assets is an extremely new practice and you should make sure to test

• your assets in-game to make sure they work as expected
• the file sizes of cooked packages to make sure that any inlining of base game assets (e.g. by mistake) isn't negating the compression advantages

It's also a good idea to defer cooking until your asset(s) are ready and have been tested in-game - this makes it easier to figure out whether the problem is with the asset itself or with the (usage of) cooker.

Opening cooked packages

Recall that as part of cooking process the editor-only data is removed from the packages - this makes opening them with the editor impossible (just like it's impossible to open a ModShaderCache despite it using a .upk extension). It's a good idea to provide a link to the original packages, e.g. a GitHub repository

Textures loaded by UI

When the UI (scaleform/flash) attempts to load from disk a texture (either as a visual for your custom UI or an image/icon for research, item, ability, etc.) it will fail if the texture resides in a SF standalone package and is inside a grouping (e.g. SomePackage.SomeGrouping.SomeTexture instead of SomePackage.SomeTexture). There are two ways to avoid this problem

1. Move the texture to the root of the package
2. Make sure the texture is already loaded in memory when the UI attempts to load it

FXS always uses either of these options, if not both at the same time (which is why they probably were not aware of this issue).

Using the first option is the simplest, albeit likely tedious as you have to go and change all the usages. Using the 2nd option will depend on how/when your textures are used.

For example, if the texture is a part of a custom UI that the player will interact with while in strategy you can force your package to be always loaded while in strategy, by adding the following code to XComContent.ini:

[Content.MapContent]
.Map=Avenger_Root
.Package=YourPackageWithTexture

IMPORTANT: If you use +Map, your config will be ignored if the player doesn't have DLC2

Note: this is the approach that FXS uses for their UI

TFC incremental self-bloat

There is no DLC-specific GPCD (every cooker invocation starts from the shipped one) so the cooker doesn't know what textures are already included in your TFCs - editing a package with textures, even if the textures were not changed, will cause the textures to be dumped again into your TFCs.

X2MBC will warn you if your TFCs grew (and by how much). While your mod will continue to properly function, a full recook is recommended before distributing your mod (e.g. by publishing to workshop). Example:

Name                                                   OriginalSize CurrentSize Increase
----                                                   ------------ ----------- --------
CharTextures-a_InfiltratorsDisguises_DLCTFC_XPACK_.tfc 8.22 MB      16.43 MB    2x      



WARNING: TFC files grew since initial creation. This could indicate data duplication.
Your mod will still function normally, but the file size might be larger than needed
(i.e. useless data present). See above for details.
You should consider doing a full rebuild before distributing your mod (e.g. via the workshop).

IMPORTANT: do not attempt to manually fix bloat by selectively deleting packages or .tfc files. A full recook is the only correct solution.

Texture file cache waste exceeds 127 MByte

Warning, Texture file cache waste exceeds 127 MByte. Please consider a full recook!

This warning is coming from base game TFC files. You should ignore it - it's not relevant to your assets and there is nothing you can do about it.

If there is potential waste in your TFC files, X2MBC will warn you (see above).

Map up-to-dateness checking

The map/level up-to-dateness check currently doesn't fully consider dependencies (#47). If you changed something and the map isn't being recooked, either force a full recook or delete the specific map from Published\CookedPCConsole.

Cooking using vanilla SDK

The cooking pipeline was developed with WOTC in mind. It technically works with vanilla SDK, but there are caveats:

• If your packages contain or reference streamed textures, they will most likely not work for neither vanilla game nor WOTC
• Voicepacks are confirmed to work in both vanilla game and WOTC
• Animations, UI textures, etc. (everything that doesn't involve TFCs/streamed textures) will likely work in both vanilla and WOTC, but was not tested
• RequiresXPACK=true is set in the .XComMod file. See #87

Same package name as one from SDK/game

While it's possible to override a package at runtime by providing one with the the same name, it's not possible to cook such package right now due to X2MBC bug (#76).

If you attempt to, the cooked package will be exactly the same as the game one.

Shipping texture updated for DLC; this is not allowed (Texture2D TextureLibrary_Asphault.Textures.AsphaltSmoothA_DIF)

If the cooker crashes with this error, make sure your WOTC SDK's CookedPCConsole Junction points to the WOTC game's CookedPCConsole.

You can check it by opening a Command Prompt, navigating to your SDK\XComGame, run dir and check the path contained in the square brackets:

SDK\XComGame> dir

>> 2021-04-06  18:06    <JUNCTION>     CookedPCConsole ['**\XCOM 2\XCom2-WarOfTheChosen\XComGame\CookedPCConsole']

If it points to the vanilla game's **\XCOM 2\XComGame\CookedPCConsole instead of WOTC's **\XCOM 2\XCom2-WarOfTheChosen\XComGame\CookedPCConsole you need to remake it.