Post deployment test runbook
This document is here to help you figure out if the tests have failed because of a problem with the tests or a problem with the website.
If there's a problem with the website, you'll want to reproduce the issue and decide if we can rollback or if we need to fix forward.
If there's a problem with the tests, you can take a breath and find out if the issue is permanent or transient, and if it happened with the automated run, the build or the actual tests.
You might have gotten a message on your PR or noticed the message on google chat. Reply to the automated message on google chat to let others know you're looking into it!
The E2E Github Action runs on every pull_request to main that also has the Seen-on-prod label attached to it (from prout). The action runs a suite of Playwright tests against the live site in Browserstack via the npm script pw-browserstack-test.
If there's been a failure, the first place to look is as the output of this command in the failing build's corresponding workflow run. To see the failing build, Open the playwright workflow. Click into the details and check the build log to see what failed. You might need to scroll down to the bottom.
You have the option to re-run the failed job.
For some reason we have 4 test runs appearing in browserstack for every deployment, and these run on browserstack on Windows 10/Google Chrome.
Open browser stack. It's not usually clear which of the runs was the failing one, as from browserstack's point of view it usually did everything successfully. Sometimes it's a case of looking at which one spent the longest running, or just going through all of them.
The clue if the test has timed out waiting for an element, is to find a really long period of yellow lines
no such element: Unable to locate element: {"method":"css selector","selector":".thank\-you\-stage"}
The department has multiple BrowserStack accounts, and chances are you've logged in to the wrong one! This account is only used to run these tests. Asks the leads (or anyone in the Digital/Reader Revenue chat room really) in either subscriptions or contributions to send you an invite.