Spec: Automated QA Strategy

[dan-menlo]

Objective

• Be able to move to weekly release of Jan and Cotex w/o increase in headcount, with 0 bug reports

Scope

• Installation and Uninstallation testing
• Upgrade testing
• Functionality testing

[Van-QA]

Hi @dan-homebrew,
Notion spec for the discussion: https://www.notion.so/jan-ai/Automated-QA-Strategy-a53e1e72445343398d1d519a5ffe7084?pvs=4

Linking related tickets, TBU to cover more: https://github.com/orgs/janhq/projects/5/views/53?sliceBy%5Bvalue%5D=Automated+Testing

Unit test:

• t‌‌est: Jan unit tests & code coverage #3495
• test: Unit test for UI component & code coverage #3507
• epic: unitest for cortex cpp cortex.cpp#1069

API test:

• test: API Collection Run for Jan #3547
• https://github.com/janhq/cortex/issues/1067 => To be discussed, to cover more hardware

E2E config:

• test: Integrate Trace Viewer with Playwright for Jan Electron App #2047

• test: Capture console log error with Playwright for Jan Electron App #3510

• epic: Monkey testing for Jan app

• feat: run e2e test in multiple screensize

E2E test

• test: pre-download model and run basic test on different hardware #3556
• feat: Sidebar configuration for threads settings reflect in API request

To be updated for more tickets