Is it okay that a lot of tests are failing on the latest main?

[ansavchenco]

I'm considering contributing to the this project and so I forked the repo and ran the tests with yarn jest command as described in the CONTRIBUTING.md and a lot of them failed. Here is the output.

Snapshot Summary
 › 1 snapshot failed from 1 test suite. Inspect your code changes or run `yarn run jest -u` to update them.

Test Suites: 23 failed, 2 skipped, 140 passed, 163 of 165 total
Tests:       532 failed, 55 skipped, 3318 passed, 3905 total
Snapshots:   1 failed, 1 total
Time:        112.172 s

Is this okay? How would I know that my code does not break anything? Should I only run the tests that are related to the part of the codebase I have made changes to? Or am I just doing something wrong?

Thanks!

[LFDanLu]

Odd, mind sharing maybe a couple of the failures your are seeing? The tests should be passing on main. What version of node are you running with?

[snowystinger]

What machine do you use? is it an M1 Mac? if so, I use node 16.13.2 currently, you could try that

[snowystinger]

Hmmm I tried running it too, everything passes for me in a brand new clone of the repo.
I know you said you haven't changed anything.

Could you triple check that you have no local changes in your fork, committed or otherwise? Or even better, clone the repo anew, and run the tests from the new clone?

Apologies if you've been through this, I just really want to make sure.

[LFDanLu]

Just an update, @snowystinger and I just finished running the tests on a variety of setups (on main and a fork of the repro with a variety of node versions) and they all pass. The tests are passing just fine on CircleCi as well, so we believe there might be something different in your environment. I'm not entirely sure what could be causing these issues, could you also try running the failing tests one by one in case this might be a cascading test failure issue?

[ansavchenco]

I'm on Intel Mac and recloning the repo helped. On node v14.17.6 I still have 2 failing tests but on v16.11.0 it's all green!

Here are the tests that failed on v14.17.6.

 FAIL  packages/@react-spectrum/numberfield/test/NumberField.test.js (56.794 s)
  ● NumberField › nb-NO can have arabic numerals entered

    expect(element).toHaveAttribute("value", "US$21.00") // element.getAttribute("value") === "US$21.00"

    Expected the element to have attribute:
      value="US$21.00"
    Received:
      value=""

      1920 |
      1921 |     let formatter = new Intl.NumberFormat(locale + '-u-nu-arab', {style: 'currency', currency: 'USD'});
    > 1922 |     expect(textField).toHaveAttribute('value', formatter.format(21));
           |                       ^
      1923 |   });
      1924 |
      1925 |   it.each(locales)('%s can have hanidec numerals entered', (locale) => {

      at packages/@react-spectrum/numberfield/test/NumberField.test.js:1922:23

  ● NumberField › nb-NO can have hanidec numerals entered

    expect(element).toHaveAttribute("value", "US$21.00") // element.getAttribute("value") === "US$21.00"

    Expected the element to have attribute:
      value="US$21.00"
    Received:
      value=""

      1931 |
      1932 |     let formatter = new Intl.NumberFormat(locale + '-u-nu-hanidec', {style: 'currency', currency: 'USD'});
    > 1933 |     expect(textField).toHaveAttribute('value', formatter.format(21));
           |                       ^
      1934 |   });
      1935 |
      1936 |   describe('beforeinput', () => {

      at packages/@react-spectrum/numberfield/test/NumberField.test.js:1933:23

Probably the problem was that I actually did make changes to code and then reverted them before running the tests.
Anyway, thank you all for your help!

[snowystinger]

Glad it's working now. The two you're seeing fail are known, some node 14 versions have a bug in Intl unfortunately and it's not worth fixing because they work everywhere else.

[ansavchenco]

As @snowystinger suggested here I recloned the repo and switched to node@v16 and it solved the problem.

[piecyk]

Hmm same for me, lot's of failing tests, was trying node v16.14.0, v18.15.0 yarn 1.22.19 on M1 Mac

running for example yarn test -- react-aria-components/test/TagGroup.test.js

Error: Uncaught [Error: Invalid slot "description". Valid slot names are "description" i "errorMessage".]

build also is not working 🤷‍♂️

🚨 Build failed.

unknown: Could not find entry: /Users/dpieczynski/work/my/react-spectrum/packages/@react-spectrum/alert
make: *** [build] Error 1
error Command failed with exit code 2.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

[LFDanLu]

Definitely odd, worked for me on a fresh clone of the repo. @snowystinger did you need to do anything special for you M1 Mac?

[snowystinger]

Nope, I use zsh and just pulled a fresh clone as well. Both commands worked.
Could you try switching to node 18, pulling a new copy of the repo down, and running the commands again. I just want to rule out anything being left behind in the switch from node 16 -> 18

[piecyk]

Ok, also tried fresh clone, latest node 18, setup with nvm,

• build worked, there where some type errors
• after running, yarn run test

Test Suites: 24 failed, 188 passed, 212 total
Tests:       1444 failed, 7 skipped, 3877 passed, 5328 total
Snapshots:   0 total
Time:        46.541 s

still feels not right

[snowystinger]

I think it has to do with order of operations. You're running build, which is creating dist folders in all of the packages. When node does resolutions during testing, it's finding those dist folders which aren't what the tests expect.
If you remove all the dist folders, it should work. You can do a make clean_dist to clear them out.

[snowystinger]

just checking in, were you able to get it working?