[Screenshots] expectation failure in `JPScreenshotTest` when running on locale other than `en-US`

[AliSoftware]

Parent: #17128

When running the JPScreenshotTest in a locale other than en-US, the test fails because it is waiting on an expectation for a TextView with text "Choose Site", but that text is translated and thus this English copy never appears when testing on a non-en-US locale.

To Test

Run bundle exec fastlane screenshots app:jetpack locale:fr-FR

Stacktrace

Example when testing with fr-FR:

[10:13:06]: $ /Users/olivierhalligon/Library/Android/sdk/platform-tools/adb -s emulator-5554 shell am instrument --no-window-animation -w \
-e testLocale fr-FR \
--no-hidden-api-checks \
-e appendTimestamp false \
-e class org.wordpress.android.ui.screenshots.JPScreenshotTest \
com.jetpack.android.test/org.wordpress.android.WordPressTestRunner
[10:16:25]: ▸ org.wordpress.android.ui.screenshots.JPScreenshotTest:
[10:16:25]: ▸ Error in jPScreenshotTest(org.wordpress.android.ui.screenshots.JPScreenshotTest):
[10:16:25]: ▸ androidx.test.espresso.base.DefaultFailureHandler$AssertionFailedWithCauseError: 'an instance of android.widget.TextView and view.getText() with or without transformation to match: is "Choose site"' doesn't match the selected view.
[10:16:25]: ▸ Expected: an instance of android.widget.TextView and view.getText() with or without transformation to match: is "Choose site"
[10:16:25]: ▸      Got: view.getText() was "Choisir un site" transformed text was "Choisir un site"
[10:16:25]: ▸ View Details: AppCompatTextView{id=-1, visibility=VISIBLE, width=368, height=74, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=false, is-layout-requested=false, is-selected=false, layout-params=androidx.appcompat.widget.Toolbar$LayoutParams@1aefc9c, tag=null, root-is-layout-requested=false, has-input-connection=false, x=198.0, y=40.0, text=Choisir un site, input-type=0, ime-target=false, has-links=false}
[10:16:25]: ▸ 	at dalvik.system.VMStack.getThreadStackTrace(Native Method)
[10:16:25]: ▸ 	at java.lang.Thread.getStackTrace(Thread.java:1538)
[10:16:25]: ▸ 	at androidx.test.espresso.base.DefaultFailureHandler.getUserFriendlyError(DefaultFailureHandler.java:12)
[10:16:25]: ▸ 	at androidx.test.espresso.base.DefaultFailureHandler.handle(DefaultFailureHandler.java:7)
[10:16:25]: ▸ 	at androidx.test.espresso.ViewInteraction.waitForAndHandleInteractionResults(ViewInteraction.java:5)
[10:16:25]: ▸ 	at androidx.test.espresso.ViewInteraction.check(ViewInteraction.java:12)
[10:16:25]: ▸ 	at org.wordpress.android.e2e.pages.MySitesPage.switchSite(MySitesPage.java:62)
[10:16:25]: ▸ 	at org.wordpress.android.e2e.pages.MySitesPage.startNewSite(MySitesPage.java:85)
[10:16:25]: ▸ 	at org.wordpress.android.ui.screenshots.JPScreenshotTest.generateSiteTopic(JPScreenshotTest.java:214)
[10:16:25]: ▸ 	at org.wordpress.android.ui.screenshots.JPScreenshotTest.jPScreenshotTest(JPScreenshotTest.java:132)
[10:16:25]: ▸ 	... 41 trimmed
[10:16:25]: ▸ Caused by: junit.framework.AssertionFailedError: 'an instance of android.widget.TextView and view.getText() with or without transformation to match: is "Choose site"' doesn't match the selected view.
[10:16:25]: ▸ Expected: an instance of android.widget.TextView and view.getText() with or without transformation to match: is "Choose site"
[10:16:25]: ▸      Got: view.getText() was "Choisir un site" transformed text was "Choisir un site"
[10:16:25]: ▸ View Details: AppCompatTextView{id=-1, visibility=VISIBLE, width=368, height=74, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=false, is-layout-requested=false, is-selected=false, layout-params=androidx.appcompat.widget.Toolbar$LayoutParams@1aefc9c, tag=null, root-is-layout-requested=false, has-input-connection=false, x=198.0, y=40.0, text=Choisir un site, input-type=0, ime-target=false, has-links=false}
[10:16:25]: ▸ 	at androidx.test.espresso.matcher.ViewMatchers.assertThat(ViewMatchers.java:16)
[10:16:25]: ▸ 	at androidx.test.espresso.assertion.ViewAssertions$MatchesViewAssertion.check(ViewAssertions.java:11)
[10:16:25]: ▸ 	at androidx.test.espresso.ViewInteraction$SingleExecutionViewAssertion.check(ViewInteraction.java:2)
[10:16:25]: ▸ 	at androidx.test.espresso.ViewInteraction$2.call(ViewInteraction.java:12)
[10:16:25]: ▸ 	at androidx.test.espresso.ViewInteraction$2.call(ViewInteraction.java:1)
[10:16:25]: ▸ 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[10:16:25]: ▸ 	at android.os.Handler.handleCallback(Handler.java:873)
[10:16:25]: ▸ 	at android.os.Handler.dispatchMessage(Handler.java:99)
[10:16:25]: ▸ 	at android.os.Looper.loop(Looper.java:193)
[10:16:25]: ▸ 	at android.app.ActivityThread.main(ActivityThread.java:6669)
[10:16:25]: ▸ 	at java.lang.reflect.Method.invoke(Native Method)
[10:16:25]: ▸ 	at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:493)
[10:16:25]: ▸ 	at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:858)
[10:16:31]: ▸ Time: 196.47
[10:16:31]: ▸ There was 1 failure:
[10:16:31]: ▸ 1) jPScreenshotTest(org.wordpress.android.ui.screenshots.JPScreenshotTest)
[10:16:31]: ▸ androidx.test.espresso.base.DefaultFailureHandler$AssertionFailedWithCauseError: 'an instance of android.widget.TextView and view.getText() with or without transformation to match: is "Choose site"' doesn't match the selected view.
[10:16:31]: ▸ Expected: an instance of android.widget.TextView and view.getText() with or without transformation to match: is "Choose site"
[10:16:31]: ▸      Got: view.getText() was "Choisir un site" transformed text was "Choisir un site"
[10:16:31]: ▸ View Details: AppCompatTextView{id=-1, visibility=VISIBLE, width=368, height=74, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=false, is-layout-requested=false, is-selected=false, layout-params=androidx.appcompat.widget.Toolbar$LayoutParams@1aefc9c, tag=null, root-is-layout-requested=false, has-input-connection=false, x=198.0, y=40.0, text=Choisir un site, input-type=0, ime-target=false, has-links=false}
[10:16:31]: ▸ 	at dalvik.system.VMStack.getThreadStackTrace(Native Method)
[10:16:31]: ▸ 	at java.lang.Thread.getStackTrace(Thread.java:1538)
[10:16:31]: ▸ 	at androidx.test.espresso.base.DefaultFailureHandler.getUserFriendlyError(DefaultFailureHandler.java:12)
[10:16:31]: ▸ 	at androidx.test.espresso.base.DefaultFailureHandler.handle(DefaultFailureHandler.java:7)
[10:16:31]: ▸ 	at androidx.test.espresso.ViewInteraction.waitForAndHandleInteractionResults(ViewInteraction.java:5)
[10:16:31]: ▸ 	at androidx.test.espresso.ViewInteraction.check(ViewInteraction.java:12)
[10:16:31]: ▸ 	at org.wordpress.android.e2e.pages.MySitesPage.switchSite(MySitesPage.java:62)
[10:16:31]: ▸ 	at org.wordpress.android.e2e.pages.MySitesPage.startNewSite(MySitesPage.java:85)
[10:16:31]: ▸ 	at org.wordpress.android.ui.screenshots.JPScreenshotTest.generateSiteTopic(JPScreenshotTest.java:214)
[10:16:31]: ▸ 	at org.wordpress.android.ui.screenshots.JPScreenshotTest.jPScreenshotTest(JPScreenshotTest.java:132)
[10:16:31]: ▸ 	... 41 trimmed
[10:16:31]: ▸ Caused by: junit.framework.AssertionFailedError: 'an instance of android.widget.TextView and view.getText() with or without transformation to match: is "Choose site"' doesn't match the selected view.
[10:16:31]: ▸ Expected: an instance of android.widget.TextView and view.getText() with or without transformation to match: is "Choose site"
[10:16:31]: ▸      Got: view.getText() was "Choisir un site" transformed text was "Choisir un site"
[10:16:31]: ▸ View Details: AppCompatTextView{id=-1, visibility=VISIBLE, width=368, height=74, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=false, is-layout-requested=false, is-selected=false, layout-params=androidx.appcompat.widget.Toolbar$LayoutParams@1aefc9c, tag=null, root-is-layout-requested=false, has-input-connection=false, x=198.0, y=40.0, text=Choisir un site, input-type=0, ime-target=false, has-links=false}
[10:16:31]: ▸ 	at androidx.test.espresso.matcher.ViewMatchers.assertThat(ViewMatchers.java:16)
[10:16:31]: ▸ 	at androidx.test.espresso.assertion.ViewAssertions$MatchesViewAssertion.check(ViewAssertions.java:11)
[10:16:31]: ▸ 	at androidx.test.espresso.ViewInteraction$SingleExecutionViewAssertion.check(ViewInteraction.java:2)
[10:16:31]: ▸ 	at androidx.test.espresso.ViewInteraction$2.call(ViewInteraction.java:12)
[10:16:31]: ▸ 	at androidx.test.espresso.ViewInteraction$2.call(ViewInteraction.java:1)
[10:16:31]: ▸ 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[10:16:31]: ▸ 	at android.os.Handler.handleCallback(Handler.java:873)
[10:16:31]: ▸ 	at android.os.Handler.dispatchMessage(Handler.java:99)
[10:16:31]: ▸ 	at android.os.Looper.loop(Looper.java:193)
[10:16:31]: ▸ 	at android.app.ActivityThread.main(ActivityThread.java:6669)
[10:16:31]: ▸ 	at java.lang.reflect.Method.invoke(Native Method)
[10:16:31]: ▸ 	at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:493)
[10:16:31]: ▸ 	at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:858)
[10:16:31]: ▸ FAILURES!!!
[10:16:31]: ▸ Tests run: 1,  Failures: 1

Note: I'd be curious if you manage to find a proper way to run the test in a locale other than en-US from AndroidStudio.

• Running the test from the CLI (adb shell im instrument -e testLocale fr-FR … — see log above) works and use the test locale provided to change the emulator and in-app locale appropriately
• But personally I haven't managed to pass the -e testLocale fr-FR value to the test runner from Android Studio — despite setting it in the "Instrumentation arguments" window (see: p1661527010684449-slack-C02QANACA).
• The only workaround I found to run it from AS was to put breakpoints in the constructors of both LocaleTestRule.class and WPLocaleTestRule.java, and change the value of the variable after it has (failed to) read it from the call to LocaleUtil.getTestLocale(). 🤷