Skip to content

Commit decc7f1

Browse files
authored
Mute/Unmute tool (#11124)
1 parent 64d58b3 commit decc7f1

File tree

4 files changed

+945
-226
lines changed

4 files changed

+945
-226
lines changed

.github/config/mute_rules.md

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -89,3 +89,60 @@ Open the [Flaky](https://datalens.yandex/4un3zdm0zcnyr) dashboard.
8989
- If the `summary:` column shows `mute <= 3` and `success rate >= 98%` - **it's time to enable the test**.
9090
- Perform steps from [How to Unmute](#how-to-unmute)
9191
- You are awesome!
92+
93+
### Unmute stable and flaky tests automaticaly
94+
95+
96+
**setup**
97+
1) ```pip install PyGithub```
98+
2) request git token
99+
```
100+
# Github api (personal access token (classic)) token shoud have permitions to
101+
# repo
102+
# - repo:status
103+
# - repo_deployment
104+
# - public_repo
105+
# admin:org
106+
# project
107+
```
108+
3) save it to env `export GITHUB_TOKEN=<token>
109+
4) save to env `export CI_YDB_SERVICE_ACCOUNT_KEY_FILE_CREDENTIALS=<iam_cloud_file>
110+
111+
**How to use**
112+
113+
0) *update your branch* - you shoud have last version of muted_ya localy
114+
1) Run instance https://github.com/ydb-platform/ydb/actions/workflows/collect_analytics.yml
115+
2) wait till end of step `Collect all test monitor (how long tests in state)` (about 7 min)
116+
3) run `create_new_muted_ya.py update_muted_ya` - it creates bunch of files in `%repo_path%/mute_update/`
117+
118+
| File Name | Description |
119+
|----------------------------------------|-------------------------------------------------------------------------------------------------|
120+
| deleted.txt | Tests what look like deleted (no runs 28 days in a row) |
121+
| deleted_debug.txt | With detailed info |
122+
| flaky.txt | Tests which are flaky today AND total runs > 3 AND fail_count > 2 |
123+
| flaky_debug.txt | With detailed info |
124+
| muted_stable.txt | Muted tests which are stable for the last 14 days |
125+
| muted_stable_debug.txt | With detailed info |
126+
| new_muted_ya.txt | Muted_ya.txt version with excluded **muted_stable** and **deleted** tests |
127+
| new_muted_ya_debug.txt | With detailed info |
128+
| new_muted_ya_with_flaky.txt | Muted_ya.txt version with excluded **muted_stable** and **deleted** tests and included **flaky**|
129+
| new_muted_ya_with_flaky_debug.txt | With detailed info |
130+
|muted_ya_sorted.txt| original muted_ya with resolved wildcards for real tests (not chunks)|
131+
|muted_ya_sorted_debug.txt| With detailed info|
132+
133+
134+
**1. Unmute Stable**
135+
1) replace content of [muted_ya](https://github.com/ydb-platform/ydb/blob/main/.github/config/muted_ya.txt) with content of **new_muted_ya.txt**
136+
2) create new PR and paste in PR Description
137+
- `<Unmuted tests : stable 9 and deleted 0>` from concole output
138+
- content from **muted_stable_debug** and **deleted_debug**
139+
3) Merge
140+
example https://github.com/ydb-platform/ydb/pull/11099
141+
142+
**2. Mute Flaky** (AFTER UNMUTE STABLE ONLY)
143+
1) replace content of [muted_ya](https://github.com/ydb-platform/ydb/blob/main/.github/config/muted_ya.txt) with content of **new_muted_ya_with_flaky.txt**
144+
2) create new PR
145+
2) run `create_new_muted_ya.py create_issues` - it creates issue for each flaky test in **flaky.txt**
146+
3) copy from console output text like ' Created issue ...' and paste in PR
147+
4) merge
148+
example https://github.com/ydb-platform/ydb/pull/11101

.github/scripts/analytics/get_mute_issues.py

Lines changed: 0 additions & 226 deletions
This file was deleted.

0 commit comments

Comments
 (0)