Azure Devops Tests succeeding while fail expected

[Mittchel]

Hi,

We've been using swift-snapshot for a while now and locally everything seems to work correctly. Now we host our source on Azure Devops and manage our CI/CD there. We've checked in our Snapshots (isRecording = true) in main. I've modified the source code (changed some texts) so tests are failing.

However, when running Xcodebuild in Azure Devops, all the tests are succeeding when they fail locally and I expect them to fail on the pipeline as well. When extracting the .xcresult with XCParse to get the screenshots I've noticed there are no screenshots in there.

I am wondering if someone knows whats causing this or is able to point me into the right direction to debug this.

Azure Devops task (removed some personal info):

- task: Xcode@5
  inputs:
    actions: 'clean build test archive'
    configuration: '${{ parameters.configuration }}'
    sdk: '$(sdk)'
    scheme: '${{ parameters.scheme }}'
    xcodeVersion: '14'
    packageApp: true
    signingOption: 'manual'
    signingIdentity: '__signingidentity__
    provisioningProfileUuid: ${{ parameters.provisioningUuid }}
    destinationPlatformOption: 'iOS'
    destinationSimulators: 'iPhone 14 Pro,OS=16.2'
    args: '-verbose -derivedDataPath $(Build.BinariesDirectory)/derived'
    publishJUnitResults: true
    useXcpretty: false

Produces the following xcodebuild command (removed placeholders):
xcodebuild -sdk iphoneos -configuration TST -workspace ../project.xcworkspace -scheme Release -destination platform=iOS Simulator,name=iPhone 14 Pro,OS=16.2 clean build test archive -verbose -derivedDataPath /Users/runner/work/1/b/derived CODE_SIGN_STYLE=Manual CODE_SIGN_IDENTITY=__signingIdentity__ PROVISIONING_PROFILE=__guid__ PROVISIONING_PROFILE_SPECIFIER=

If more information is needed please let me know. Thank you in advance!

[mbrandonw]

Hi, as this is not an issue with the library and most likely has to do with Azure, I'm going to move this to a discussion.

[nshelleyacorns]

@Mittchel Are you using perceptual comparison? If so, there is a bug that has been fixed in this PR that gives false positives when metal is not available. The PR is yet to be merged, unfortunately.

[Mittchel]

Thank you for your response. Actually I believe we do, since we do the following:

   func testLoadingViewOniPhoneXrPortraitLightMode() throws {
      assertSnapshot(matching: viewController, as: .image(drawHierarchyInKeyWindow: true, precision: precision, perceptualPrecision: perceptualPrecision, size: iPhoneXrPortraitSize, traits: traitLightMode))
   }

However, I haven't noticed the warning yet in our log files, but I'll investigate that further. Is there any workaround for this @nshelleyacorns?

Also, @mbrandonw do you see any chance of approving that PR?

Thanks everyone!

[nshelleyacorns]

When I looked through the logs (I think I had to disable xcpretty maybe to see them, but I can't remember for sure), I saw some messaging around metal not being available, which tipped me off to search this repo for issues related to metal.