Question about "terraindata.bin" files

[Grenadier928]

Hello,
I've been looking into how the Alan Wake engine (the original game) handles terrain. In the extracted game folders, most level-related folders have several files associated with what I assume is terrain data, such as HD048_029.bin. I believe that these follow a naming convention similar to [HD/LD][Cell-X]_[Cell-Z].bin. The unbin.py tool works on these files, and produces a few output .bins, including one called "cid_terraindata.bin", which I believe holds data such as heightmap, texture, and vegetation info. However, I can't figure out what to do from here. The bin2xml tool does not work on these files, and they are mostly just raw binary data (besides for some path references to textures near the top of the file). Any tips on how I could proceed if I wanted to extract heightmap information? Thanks, and great job on the toolset!

[Nostritius]

Hi,
You are right with your assumption, that the cid_terraindata.bin file contains data for the terrain. These files use a special format which is not like the other cid files. The terrain in both Alan Wake games doesnt use heightmaps (at least not a global) but rather defines a set of mesh quads or in some cases triangles which are simply parts of the terrain. Every quad can have up to 3 textures blended together. If you want vegetation info, you should take a look into the cid_foliagedata.bin files which contains a list of foliages contained in the scene.
This repository contains a the tool terraindata2obj.py which at least extracts the geometry of the terrain and some blending maps, but it shouldnt be considered as up to date, since I found out more stuff in the meantime which changes some assumptions made in this tool. I have another implementation here, which is more up to date and covers more details. Finally I have a kaitai file where I experiment with the format and which contains even more information not published in neither of the two tools before. If you want I would share it with you, since it is not yet published.
If you have any more questions feel free to ask

[Grenadier928]

Thanks for the reply, I can't believe I missed that! I noticed that the tool works fine when converting terrain data that originated from a bin file with the LD prefix, but fails when converting terrain data from a bin with the HD prefix (which I'm assuming are low detail / high detail). The python error is "Line 105: unpack requires a buffer of two bytes".

I had a few questions about the terrain system in general. There are several folders that group many terrain files together based on region, such as se_cauldronsouth or brightfalls_part_2. I'm guessing that these are loaded based on the player's position (rather then based on the current level / episode), since freecamera allows the player to travel to any part of the world from any level. Is this correct?

Many of the terrain pieces seem to only have an "LD" file, particularly those grouped on region, such as in the folders prefixed "nw" and "se" (which I believe the player would always be pretty far away from). I also noticed that only terrain files with the HD prefix ever have a .collisions file. This seems reasonable, as the only cells the player would be near would have the HD version loaded. However, is it possible to have an LD cell still use a .collisions file?

Finally, unrelated to terrain, I wanted to ask how the GIDRegistry.txt files worked. They appear in many folders, but only seem to have data in the folders the relate to levels / scenes, such as "level_01". I'm guessing they reference every asset used in the level / scene. Would modifying these files change anything in-game, or are they just a dump?

Thanks again!

[Nostritius]

The terraindata2obj.py tool does only work on the low detail files because they contain the more fundamental data, for example the geometry, and the high detail files just extend this with for example a displacement map which would enable tessellation if you approach that quad (as far as I understand it). The tools here are more of a rough proof, that it is possible to extract this data, then a fully fledged modding toolset, so I didn't bothered too much with catching all possible errors.
I don't know, if you have noticed, but there is the file gameworld.xml (or dlc1_world.xml and dlc2_world.xml for the dlcs) in the folder globaldb, which specify these scenes which need to be loaded for a specific episode. It is a mystery for me too, how they load the stuff in the freecam mode without permanently testing if this whole thing is available. Perhaps this mode is actually checking apart from the main game.
You are probably right about that these parts with only LD files are only rim regions, not of interest for the player directly. The collisions files define the terrain phjysics collisitions . If you will never visit this cell as the player directly it doesn't make much sense to add a terrain collision at that point.
About the GIDRegistry.txt file, every object in a scene that interact with the player or are in some way dynamic, have a so called GlobalID (GID). It consists of a type number and a hash. It is used to reference a specific object for example in the scripts. Now these GIDRegistry.txt files maps the GIDs of these scenes and levels to a human readable name of the object. I doubt that the fame would care, if they are removed, since the whole system works pretty well without them. MAybe they had some debugging purposes in the beginning.

[Grenadier928]

Thanks for the information! I've been playing around with unpacking level information, and was wondering about cid_spawnpoint.bin and cid_spawnposition.bin. I'm not sure if I'm unpacking the right files, but from my tests all levels / scenes seem to have the same values for spawnpoint and spawnposition. Taking the Ferry ride for example, in the folder level02, BrightFalls.bin, root.bin, and S01_Ferry.bin all have identical values in these files, and from what I can tell it's the same for every level bin file. Am I not looking in the right place?

[Nostritius]

Thats an interesting finding, I always thought these additional bin archives contain additional data. In my OpenAWE project I only load the root.bin and get a pretty complete level. I will take a look into that myself.

[Nostritius]

I have today investigated the files you mentioned and as I see it, the explanation is pretty simple, the files are empty. The cid files contain a list of serialized objects but they can also be empty.