gltfpack: EXT_mesh_gpu_instancing - support instance colors

[andredsm]

I have a glb model that uses EXT_mesh_gpu_instancing extension. Apart from TRANSLATION, ROTATION, and SCALE attributes, each instance has a single color for its entire surface (so, a color per instance) and this info is in _COLOR_0 attribute. When I run optimize the model with gltfpack, the _COLOR_0 attribute is stripped from the output model. I am aware that gltfpack removes unknown/custom extensions, but since custom attributes are supported by EXT_mesh_gpu_instancing extension, I think these attributes should not be removed.

As with vertex attributes, custom instance attributes may be prefixed with an underscore (e.g. _ID, _COLOR, etc.) and used for application-specific effects.

As result, the ThreeJS loads and renders my model as expected, but the colors are missing when using the optimized model:

models.zip

Command:
gltfpack -i input-model.glb -o output-model.glb

Input model rendered in gltf-viewer:

Output model rendered in gltf-viewer:

[zeux]

For custom vertex attributes, gltfpack currently prefers only supporting attributes for which it understands the desired semantics. This is helpful because this means gltfpack knows how to compress / store data. Similarly, instance attributes right now are only supported when the semantics is known (so T/R/S).

I didn't realize three.js supports _COLOR_0 out of the box; looks like that is special-cased in GLTFLoader. Are you interested in color support, specifically? Supporting just instance colors would be much easier than fully generic per-instance attribute support, and it would potentially allow more support for automatic instancing in the future (right now -mi will not merge meshes into instances if they use different materials, but maybe some other command line flag like -mic could be used to auto-instance objects with different colors by moving color from the material to the instance attribute).

[andredsm]

At this moment, supporting instance color would be very helpful. About ThreeJS support, I sneak peeked the GLTFLoader code and noticed that they check whether _COLOR_0 instance attribute exists, so I exported the instance colors using this attribute name. Lastly, I didn't know gltfpack has the same behaviour for custom vertex. For example, I am merging meshes, and in the near future, I will need to differentiate each individual mesh. I plan to achieve this by using an ID as a custom vertex attribute, something similar to EXT_mesh_features.

[zeux]

gltfpack supports ID vertex attributes (_ID, _BATCHID, _FEATURE_ID_n) - this is similar to the color support discussed above, in that while the glTF specification doesn't define their meaning, the ecosystem converged on specific semantics so gltfpack knows what to do with these. I'll add this, and the _COLOR_0 support once implemented, to gltfpack documentation.

[andredsm]

Speaking as a user, I would prefer to retain my custom attributes in the output file, even if it isn't possible to efficiently compress them, as the library doesn't know the semantics of such attributes. Anyway, the support for _COLOR_0 instance attribute will be very helpful ☺️. Thanks.

[zeux]

There are other considerations beyond efficiency - for example, if an attribute represents direction (e.g. velocity), it needs to be transformed by a matrix if any scene graph reorganization is performed. If an attribute represents vertex id, it needs to be updated when two meshes are merged. If an instance attribute represents a material property, some of them need to be scaled with the mesh scale which changes under transformation; some need to be analyzed to understand if, for example, the mesh needs to be rendered with transparency.

So generally speaking, any scene modification needs to be aware of the semantics of individual attributes. gltfpack takes an opinionated approach to scene processing and needs to be aware of the behavior of every scene component; it is not extensible by design (compared to something like glTF-Transform that can be scripted to add extra functionality).