best practices for maintaining color with AR - USDZExporter not matching three.js

[rawwerks]

i'm really struggling to maintain the aesthetic of a very simple PBR GTLF material when going from web-viewer to AR. the results are significantly worse than the code we wrote to do this directly in three.js - so i suspect there may be something going on. at the same time, i'm not really sure if this is a bug, so i figured i'd start with a discussion thread rather than create an issue.

here's the summary, in pictures. (you can try for yourself with the link at bottom). the model looks like this in webviewer

looks like a bright lemon yellow on android:

looks brown/dark - even green under certain lighting conditions on iOS:

for clarification, the above three images are all from AR of the same file on model-viewer. for comparison - when we wrote our own three.js exporter using USDZexporter.js, we saw a much more favorable result in USDZ:

(i don't have an android comparison because of the whole sceneviewer blob debacle (a different thread.))

since github won't let me upload .GLB, see here for the file i'm testing with: https://ar-polyspectra-com.pages.dev/your-stl-here-PBR-simple.glb and here for a very simple test site using model-viewer: https://ar-polyspectra-com.pages.dev/

is this color discrepancy considered normal? i understand that we are trying to get lots of different things to play nice together with Apple/Google and WebGL/WebXR/Scene Viewer/Quick Look and that there will be tradeoffs - but for such a simple GLTF 2.0 compliant pbrMetallicRoughness material - i am pretty surprised by the inconsistency.

i do have some breadcrumbs towards a possible solution, but i don't know enough about the guts of model-viewer to say for sure if it is relevant here. when we were getting really crappy-looking USDZ exports from three.js - "Tape" on the three.js discord suggested it was an issue with surface normals, and @donmccurdy suggested the following approach, which completely fixed the problem:

geometry = geometry.toNonIndexed();
geometry.computeVertexNormals();

the full workflow looks like:

      let tmp = scene.clone()
        
     //a bunch of other stuff here

        tmp.traverse(child => {
            if (child.material) {
                child.geometry = child.geometry.toNonIndexed();
                child.geometry.computeVertexNormals();
                KUtils.assignUVs(child.geometry);
            }

then later:

const exporter = new USDZExporter();
exporter.parse(tmp).then(raw => {
//blob stuff here...you get the point

Any thoughts?

[rawwerks]

@elalish - could you point me to the correct way to access the material geometry in modelviewer? i'm running into issues that are likely some combination of the way in which three.js is used and that i don't understand typescript.

in ar.ts, here is what i am trying to do, after removing the shadow and right before the USDZ export:

      scene.traverse(child => {
        if (child.material) {
            child.geometry = child.geometry.toNonIndexed();
            child.geometry.computeVertexNormals();
            KUtils.assignUVs(child.geometry);
        }
      });

which gives the build error: Property 'geometry' does not exist on type 'Object3D'.

i've been trying to figure out the correct type from looking at how three.js is imported, but seem to be flailing a bit:

if ((child as THREE.ThreeMaterial).isMaterial) {

/Users/mylocalpathto/model-viewer/packages/model-viewer/node_modules/@types/three/index"' has no exported member 'ThreeMaterial'

[elalish]

I think the surface normals issue is the key. Notice beyond the coloring that the Android example is faceted? It sounds like USDZ doesn't support faceted, so you're getting some weird behavior. I'd recommend instead of digging around in the guts of three.js, just export your GLB with vertex normals, which anything like Blender should be perfectly capable of. Then you'll get a smooth and fairly consistent look when auto-converting to USDZ.

Of course the lighting is different between MV and Quick Look, so the colors will always be a bit off, but agreed they shouldn't be this bad.

[donmccurdy]

Property 'geometry' does not exist on type 'Object3D'.

This part is a TypeScript compiler error; it doesn't know that object.material implies the object is a Mesh. Instead you can cast or typecheck:

// (a)
tmp.traverse(child => {
  if (child instanceof Mesh) {
    // ...
  }
})

// (b)
tmp.traverse(child => {
  if (child.material) {
    (child as Mesh).geometry = ...
  }
})

This could also be baked in advance on the glTF model, using https://gltf.report – removing normals is equivalent to computing flat normals here.

import { unweld } from '@gltf-transform/functions';

await document.transform(unweld());

for (const mesh of document.getRoot().listMeshes()) {
  for (const prim of mesh.listPrimitives()) {
	  prim.setAttribute('NORMAL', null);
  }
}