Enums/Flags

[Marlamin]

We've been talking about enums/flags support in WoWDBDefs for years now and haven't actually gone forward with it and it's probably time to change that.

Shipping it as part of a 'V2' of the DBD format might be a bit much to take on or wait for, so a different strategy was discussed in the datamining Discord that I want to bring over here because having important discussions like that on Discord sucks.

Stuff that was previously discussed in the V2 thread on here as well as more recently on Discord:

File structure

• Using separate files to describe flags/enums as well as link them to fields as to not require any changes in the DBD files themselves.
• These files could live in an extra folder.
• A mapping file could be used to 'map' columns inside of DBD files to their respective enums/flag definitions.
• Files named after their flag/enum would live next to the mapping file.

Format

• Similar to DBD? Different? Formats were previously suggested in the v2 thread and in schlumpf's demo definition format.
• Having flags/enums described in hex was preferable, is that still the case?
• Do we want to include longer descriptions of enums/flags? Would they be comments or part of the actual format? I'd love to have this for user-facing tool/diff reasons.

Versioning

• How do we tackle this?
• Do we need to? Are there tangible examples of Blizzard changing flags/enums over time to the point where one would need to redefine them similar to how we redefine version definitions in DBDs?

Would love to get some discussion going and maybe even work towards an actual implementation. As we're doing it separately from the fabled 'V2' format this can be done without disturbing the current definitions.

[QartemisT]

My main comment on this would be being able to provide a version range on some of them, as they change over time. :3

[Marlamin]

Do we only need versioning for the enums/flag files themselves and not the mapping file? Have enums/flags every switched meanings for certain fields entirely?

[bloerwald]

I sure agree on "no complete copies of the list on every minor addition". It might make sense to allow the entire things to change so much to warrant an entirely new list though.

A few independent thoughts:

[C] As a person adding information, I don't want to be forced to define a version, especially not if it has to be there and has to be exact. I don't have the energy to bisect client decompilation to figure out when code was added to handle it.

[A] What about a hybrid format where sub ranges can be version-specific, and overwritten?

enum "M2 Attachement" {
  0 Shield
  0 MountMain
  0 ItemVisual0             // this is a side-comment: allow multiple names for same value, please
  1 HandRight
  1 ItemVisual1
  …
  38 SpellHandDirected
  
  version ≥ wotlk {
    39 VehicleSeat1
    40 VehicleSeat2
  }

  version ≥ Mists {
    55 HeadTop
  }

  version ≥ TwilightFalls {
    38 SpellFeetPorn
  }
}

[B] If stuff changed that much, is it really still the same field/type/enum? We already had this pain with the SoundEntries → SoundKit change everywhere, and I think we just plain ignored it, keeping field name, and changing referenced table to the new one, breaking all old definitions. The cleaner solution would have been adding a new field with separate name and fk definition and using that instead. A more painful solution, but a cleaner one. This essentially is an argument for "decouple enum definition and mapping, and allow versioning it". It even expands to "fk mapping should be outside as well".

[Marlamin]

C: Agreed.
A: Inline versioning, while far more readable for humans, worries me in terms of format parsing complexity. I don't have a better solution, but I know this has been a pain point (for the in my opinion rather simple DBD format) in the past, so something to take note of at least.
B: There's probably cases where it could be either the same field/type/enum but changed significantly or a different one entirely. And yes, SoundEntries -> SoundKit still pains me to this day. Adding FK mappings to the mapping/extra file is perfectly fine by me as a term of "overriding" FKs in DBDs. It would get rid of one the things we'd need to "fix" for V2 as well, so that's nice.

[bloerwald]

A: if there is a agreed-upon need to avoid a custom format and only have a custom schema, I’m fine with transitioning to YAML. I had expected the custom format to be easier, but I‘m not a user.

[Marlamin]

Its more nuanced than just "no custom format", just not too complex of one. Again, I don't have a solution, an easily already parseable format is one, an easier custom one without too many shenanigans is another.

[Dorovon]

A few special cases that I think we should try to support in the format include

• Conditional Enums/Flags/FKs: One option here would be to have a section of the mapping file specifically for these fields where instead of listing a field and its enum name, it instead lists the field, the conditional field, and the required value of that field before the enum name.
• Empty Flags: This would be for fields that we know are bitfields, but where we don't know what any of the flags are so that tools can still display them as flags. These could be defined alongside other fields, but simply without a reference to a flags file.
• FK Flags: For example, SkillLineAbility::RaceMask appears to use ChrRaces::PlayableRaceBit as flags.

Long Descriptions

• Do we want to include longer descriptions of enums/flags? Would they be comments or part of the actual format? I'd love to have this for user-facing tool/diff reasons.

I think this is a good idea, but we'll have to decide how we want to format the short and long strings together. One of the big use cases for having these files is for tools like DB2 viewers where being able to read more about what an enum value does would be very helpful. There is also the question of just how detailed/long we want to let these be. As an example for spell effect types, we could have a brief descriptions to give a simple idea of what something does or long descriptions that fully explain what the effect does and how the spell data is used.

98 "Knockback" "Knocks the target(s) in a direction."

98 "Knockback" "This overrides the current velocity of the target(s). Vertical velocity is set to EffectBasePointsF/10 yards/sec up and horizontal velocity is set to EffectMiscValue[0]/10 yards/sec away from the origin of the Knockback."

The former is short enough to show in most text diffs but isn't very useful in this case, while the latter is long enough that it wouldn't display very well in some cases but provides most of the useful details someone reading about a spell may want to know.

Types?

While we're at it, I think it could be helpful to also use the mapping file to map fields to generic types that show up a lot where we would want our tools to handle all of them in the same way. The biggest example that comes to mind here is the integer fields that represent colors used in many DB2s, but we could also use this for Unix timestamps and anything else sensible that shows up.

[Marlamin]

Yeah, for the mapping we could do like...

FLAGS Achievement::Flags AchievementFlags
FLAGS SpellMisc::Flags[0] SpellAttributes0
FLAGS SpellMisc::Flags[15]
ENUM Map::ExpansionID ExpansionLevels
COLOR LightData::SkySmogColor

Where SpellMisc::Flags[15] is marked as a flags field but doesn't have an accompanying file yet.

Conditionals have a third argument to specify their condition, can also make them CENUM/CFLAGS or CONDENUM/CONDFLAGS for additional clarity.

ENUM ModifierTree::Asset ItemQuality ModifierTree::Type=14 
FLAGS ChrCustomizationReq::ReqValue ClassMask ChrCustomizationReq::ReqType=1

For descriptions for enums themselves we could have the really long ones/explanations be a comment which is still accessible to parsers.

98 KnockBack "Knocks the target(s) into a direction" // This overrides the current velocity of the target(s). Vertical velocity is set to EffectBasePointsF/10 yards/sec up and horizontal velocity is set to EffectMiscValue[0]/10 yards/sec away from the origin of the Knockback.

FK flags... hmmm... maybe just another :: field instead of a flag name? Unsure.

[Dorovon]

Thinking about it more, maybe FK Flags aren't such a huge deal unless I'm forgetting other places where they get used. For cases like RaceMasks most tools would probably just want to show the name anyway, so we could just do those manually.

[bloerwald]

I absolutely loved the wow.tools content dependent fkeys etc. I do support this. It is another reason for me to pivot to more complex FK definitions and making those conditional and versioned.

Empty Flags

Fine with me. Just as fine as adding placeholder flag definitions though. Both with "seen values" added (I had done that in comments in the past, even without knowing semantics) and with a completely empty list.

FK as Flags

Not a huge deal, but why not. It is a trivial annotation on FKeys. int<ChrRaces::PlayableRaceBit$Mask>?

Long Descriptions

I think that's why I added COMMENT everywhere back then. There is a need for developer-readable text which isn't displayed on first sight in UIs.