test: Add Remix e2e tests with test utils. #649

onurtemizkan · 2024-08-20T14:34:34Z

Adds utilities for E2E testing wizard CLI

Non-testing updates

This PR adds new CLI arguments to provide pre-selected project data. This is required to run tests with dummy Auth Tokens / DSNs without opening browser / signing in. Looking at the comments, I got the impression that, this feature was already planned. Please let me know if that's not the case, and I'll try to figure out another way.

Testing utilities

The flow of the initial e2e test structure is:

Create a new git repo in the local test application
Run the wizard on it with pre-provided auth token / dsn.
- Optionally check for the stdout for expected prompts / messages, provide conditional stdin
Check for the file contents / file existences in the application after the wizard is finished.
Check if the project builds successfully
Check if the project runs on production mode successfully
Check if the project runs in development mode successfully

In the future, we can also add new utilities to check if the events/transaction from the created example page are sent successfully to Sentry

e2e-tests/utils/index.ts

lforst · 2024-09-02T11:51:36Z

package.json

    "xcode": "3.0.1",
    "xml-js": "^1.6.11",
-    "yargs": "^16.2.0"
+    "yargs": "^17.7.2"


Yargs 17 dropped support for Node 14 in yargs/yargs#2384 which is why we didn't bump it earlier. I'd say though that bumping it is probably fine, I would just do it in a separate PR and cut a major release for the wizard (no big deal imo). Does that make sense?

I had to update it when I was experimenting but it seems we can stay at the current version with what I ended up. So just reverted the version update 👍

lforst

Just a first quick pass. Generally love it. I think we should address the two comments I left above and then we can iterate more (although I don't think there is much more necessary).

lforst · 2024-09-02T11:55:21Z

e2e-tests/utils/index.ts

+ * @param integration
+ * @param projectDir
+ */
+export async function runWizard(integration: Integration, projectDir: string) {


What if, instead of having all of this functionality in this function, we call this function something like startWizardInstance and it returns the wizardTestEnv so you can call stuff on it inside the jest tests? That way, we're a little more dynamic in what we can test and the various permutations we can test for more complicated wizards.

Updated and moved to Jest runner 👍

onurtemizkan · 2024-09-23T13:11:06Z

src/run.ts

+  preSelectedProject?: {
+    authToken: string;
+    selfHosted: boolean;
+    dsn: string;
+    projectId: string;
+    projectSlug: string;
+    projectName: string;
+    orgId: string;
+    orgName: string;
+    orgSlug: string;
+  };


Note: This overlaps with #671. So if preSelectedProject is provided, the newly added --orgSlug and --projectSlug will not have any effect on the wizard behaviour.

lforst · 2024-10-03T12:05:53Z

e2e-tests/tests/remix.test.ts

+      wizardInstance.sendStdin(KEYS.ENTER);
+    }
+
+    const examplePagePrompted = await wizardInstance.waitForOutput(


I don't fully understand why we need the optional configuration... Would you mind explaining it to me? I feel like I'm missing something. 😄

It's primarily for local development where the test flows can be fragile, for example when the package manager is detected somehow and it's not prompted, or the leftovers from the example page exist and it's not prompted.

I think we can improve the utils in follow-up PRs to make them more debuggable, but this makes writing / debugging tests slightly easier in the meantime.

Seems good! 👍

lforst

The tests run suspicously fast 😂 but otherwise lgtm. Would you mind going over the changes I made one more time and make sure that I didn't do anything dumb? Then we can ship this! Sorry for the insane delay - totally on me.

onurtemizkan · 2024-10-03T12:55:56Z

@lforst, the updates look good to me, I just increased the timeout after yarn install. That was timing out for me locally. Thanks for the review!

onurtemizkan force-pushed the onur/e2e-tests branch 10 times, most recently from 83d2cc5 to f2bf4f4 Compare August 22, 2024 08:13

github-advanced-security bot found potential problems Aug 27, 2024

View reviewed changes

e2e-tests/utils/index.ts Dismissed Show dismissed Hide dismissed

onurtemizkan force-pushed the onur/e2e-tests branch 3 times, most recently from 8ad89ee to 074ec99 Compare August 27, 2024 21:13

onurtemizkan requested review from Lms24 and lforst August 28, 2024 10:55

onurtemizkan marked this pull request as ready for review August 28, 2024 10:55

lforst self-assigned this Aug 29, 2024

lforst reviewed Sep 2, 2024

View reviewed changes

onurtemizkan force-pushed the onur/e2e-tests branch from 04d8d03 to d107969 Compare September 9, 2024 14:46

onurtemizkan added 9 commits September 23, 2024 15:00

Add Remix test app

701645d

Add utils and Remix tests.

bb5b0f3

Use chai and tsx versions that support Node 14.

e75bc40

Run e2e tests on Node 20

43272f5

Remove eslint from test app.

12af21e

Add git author info to avoid commit error.

6b20e7a

Cleanup

fa33692

Add non-empty username / email.

8ed4756

Update eslint and jest configurations

38f421f

onurtemizkan added 10 commits September 23, 2024 15:01

Clean up

60894bc

Add optional stdout support.

2f63320

Add debug mode

d52b881

Switch back to yargs 16.2.0

ab6b2f8

Switch to jest runner with refactors.

32c80d3

Fix linter

20dba51

Unref the child process and destroy all pipes.

adc8bc2

Remove success/info logs.

ae95e15

Update tests to support feature selection.

9d9e8b4

Switch from .sentryclirc to .env-sentry-build-plugin

475ca82

onurtemizkan force-pushed the onur/e2e-tests branch from d107969 to 475ca82 Compare September 23, 2024 13:08

onurtemizkan commented Sep 23, 2024

View reviewed changes

Luca Forstner added 2 commits October 3, 2024 13:45

Merge remote-tracking branch 'origin/master' into onur/e2e-tests

fe10d2d

order

7df6ac7

lforst reviewed Oct 3, 2024

View reviewed changes

Luca Forstner added 2 commits October 3, 2024 14:06

Mild refactor

982d6d0

Remove unnecessary readme

6df8f79

lforst approved these changes Oct 3, 2024

View reviewed changes

Increase timeout, waiting for yarn install

57990bb

lforst merged commit 6b580f5 into master Oct 3, 2024
14 checks passed

lforst deleted the onur/e2e-tests branch October 3, 2024 13:23

Uh oh!

test: Add Remix e2e tests with test utils. #649

test: Add Remix e2e tests with test utils. #649

Uh oh!

Conversation

onurtemizkan commented Aug 20, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Non-testing updates

Testing utilities

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lforst left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lforst left a comment

Choose a reason for hiding this comment

Uh oh!

onurtemizkan commented Oct 3, 2024

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

onurtemizkan commented Aug 20, 2024 •

edited

Loading