What causes failures between iOS versions and existing strategies for regression testing?

[pgobrien]

I'm aware of the disclaimer:

⚠️ Warning: By default, Xcode will try to add the SnapshotTesting package to your project's main application/framework target. Please ensure that SnapshotTesting is added to a test target instead, as documented in the last step, below.

My team is using this delightful library. We have some concerns about reference images taken on iOS versions and then having to retake new images every iOS update since we will have 1000s of images.

1. What causes image discrepancies between snapshots from different iOS versions?
2. Will this be possible to include in a future release?
3. What are some current strategies that the community has been using to work around this?

It seems unpredictable which iOS version snapshots will pass and how the diffing image will look. 13.1, 13.2, 13.3 all pass but below is a diffing image for a minor release of 13.3 -> 13.4. Another diffing image from 13.4 -> 14 has a different diffing failure. I suspect it's due to unpredictable changes in the APIs that Apple renders the views between versions?

Thank You!
Pat OBrien

[0rtm]

I am having the same issue with reference screenshots taken on iOS 13 and testing on iOS 14. What I found out is that only view controllers were affected. Tests for views by themselves are still passing. This makes me think there is something they changed in iOS 14

[pgobrien]

I actually have done some extensive testing to try to find a pattern or a reason. I understand how the images are gathered and compared from the source code. I wanted to dig a bit deeper. I have had viewcontrollers and uiviews pass and fail unpredictably. For example a reference image for a tableviewcontroller at 13.1 will pass at 13.2, 13.3, and fail at 13.4+. But a UIView such as the cell could still fail at 13.4+, but could pass if I remove a shifted textview from inside of it. The hierarchy is slightly shifted in different ways. You could remove views to compensate, but it's just not worth it.

The original hope was to find unintended UI changes in our sdk between versions. I don't see a way to distinguish what could be an issue with our sdk or if it's introduced by the iOS version change itself. Also the reviewing of new references images as they are needed for new versions might be too time consuming.

[0rtm]

I actually have done some extensive testing to try to find a pattern or a reason. I understand how the images are gathered and compared from the source code. I wanted to dig a bit deeper. I have had viewcontrollers and uiviews pass and fail unpredictably. For example a reference image for a tableviewcontroller at 13.1 will pass at 13.2, 13.3, and fail at 13.4+. But a UIView such as the cell could still fail at 13.4+, but could pass if I remove a shifted textview from inside of it. The hierarchy is slightly shifted in different ways. You could remove views to compensate, but it's just not worth it.

The original hope was to find unintended UI changes in our sdk between versions. I don't see a way to distinguish what could be an issue with our sdk or if it's introduced by the iOS version change itself. Also the reviewing of new references images as they are needed for new versions might be too time consuming.

As a workaround you can maybe try to use XCTSkip to not run tests with old reference images on newer OS versions

[frankacy-d]

FWIW, this is also a massive headache for me. I'm trying to convince my team to get onboard with snapshot testing but this is making things difficult 😓

[stephencelis]

Converting this to a discussion. Ideally Apple would offer a consistent way of screen shotting across devices, but alas they do not. In the meantime, maybe we can discuss our various ways of working around the issue, and maybe some ideas can be contributed back!