Skip to content

Commit b532321

Browse files
committed
Restructure README
Add **Running on CI** section to encapsulate use of releases and configuration. Moves release dev guide to **Contributing**.
1 parent b7ab914 commit b532321

File tree

2 files changed

+51
-47
lines changed

2 files changed

+51
-47
lines changed

.github/workflows/numpy.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,7 @@ jobs:
2525
env:
2626
ARRAY_API_TESTS_MODULE: numpy.array_api
2727
run: |
28-
# Skip test cases with known issues
28+
# Skip testing functions with known issues
2929
cat << EOF >> skips.txt
3030
3131
# copy not implemented

README.md

Lines changed: 50 additions & 46 deletions
Original file line numberDiff line numberDiff line change
@@ -50,47 +50,6 @@ a specific test case, which is useful when developing functions.
5050
$ pytest array_api_tests/test_creation_functions.py::test_zeros
5151
```
5252

53-
## Releases
54-
55-
The test suite has tagged releases on
56-
[GitHub](https://github.com/data-apis/array-api-tests/releases). If you run
57-
the test suite in your CI system, we recommend pinning against a release tag.
58-
59-
We use [calender versioning](https://calver.org/) for the releases. You should
60-
expect that any version may be "breaking" compared to the previous one, in the
61-
sense that there may have been additional tests added which cause a previously
62-
passing library to fail.
63-
64-
For now, the test suite is
65-
not installable as a Python package. You can use it by cloning the repo and
66-
running `pytest` as described above. If it would help you to be able to
67-
install it as a package, [please let us
68-
know](https://github.com/data-apis/array-api-tests/issues/85).
69-
70-
*Test suite maintainer note:* to make a release of the test suite, make an
71-
annotated tag with the version:
72-
73-
```
74-
git tag -a 2022.1
75-
```
76-
77-
(for the message, just write something like "array-api-tests version 2022.1").
78-
Be sure to use the calver version number for the tag name. Versioneer will
79-
automatically set the version number of the `array_api_tests` package based on
80-
the git tag.
81-
82-
Then push the tag to GitHub
83-
84-
```
85-
git push --tags origin 2022.1
86-
```
87-
88-
Finally go to the [tags page on
89-
GitHub](https://github.com/data-apis/array-api-tests/tags) and convert the tag
90-
into a release. If you want, you can add release notes to the release page on
91-
GitHub.
92-
93-
9453
## What the test suite covers
9554

9655
We are interested in array libraries conforming to the
@@ -147,7 +106,7 @@ of the functions and some miscellaneous things.
147106
functions interact with them correctly.
148107

149108
Be aware that some aspects of the spec are impractical or impossible to actually
150-
test, so they are not covered in the suite <!-- TODO: note what these are -->
109+
test, so they are not covered in the suite. <!-- TODO: note what these are -->
151110

152111
## Interpreting errors
153112

@@ -172,7 +131,28 @@ behaviour different from the spec, or test something that is not documented,
172131
this is a bug—please [report such
173132
issues](https://github.com/data-apis/array-api-tests/issues/) to us.
174133

175-
## Configuration
134+
135+
## Running on CI
136+
137+
See our existing [GitHub Actions workflow for
138+
Numpy](https://github.com/data-apis/array-api-tests/blob/master/.github/workflows/numpy.yml)
139+
for an example of using the test suite on CI.
140+
141+
### Releases
142+
143+
We recommend pinning against a [release tag](https://github.com/data-apis/array-api-tests/releases)
144+
when running on CI.
145+
146+
We use [calender versioning](https://calver.org/) for the releases. You should
147+
expect that any version may be "breaking" compared to the previous one, in that
148+
new tests (or improvements to existing tests) may cause a previously passing
149+
library to fail.
150+
151+
### Configuration
152+
153+
Use the `--ci` flag to run only the primary and special cases tests. You can
154+
ignore the other test cases as they are redundant for the purposes of checking
155+
compliance.
176156

177157
By default, tests for the optional Array API extensions such as
178158
[`linalg`](https://data-apis.org/array-api/latest/extensions/linear_algebra_functions.html)
@@ -183,10 +163,9 @@ The tests make heavy use
183163
[Hypothesis](https://hypothesis.readthedocs.io/en/latest/). You can configure
184164
how many examples are generated using the `--max-examples` flag, which defaults
185165
to 100. Lower values can be useful for quick checks, and larger values should
186-
result in more rigorous runs. For example, `--max-examples 10000` may find bugs
187-
where default runs don't, but will take a much longer time.
166+
result in more rigorous runs. For example, `--max-examples 10_000` may find bugs
167+
where default runs don't but will take much longer to run.
188168

189-
<!-- TODO: howto on CI -->
190169

191170
## Contributing
192171

@@ -231,6 +210,31 @@ where `path/to/array-api` is the path to a local clone of the [`array-api`
231210
repo](https://github.com/data-apis/array-api/). Edit `generate_stubs.py` to make
232211
changes to the generated files.
233212

213+
214+
### Release
215+
216+
To make a release, first make an annotated tag with the version, e.g.:
217+
218+
```
219+
git tag -a 2022.01.01
220+
```
221+
222+
Be sure to use the calver version number for the tag name. Don't worry too much
223+
on the tag message, e.g. just write "2022.01.01".
224+
225+
Versioneer will automatically set the version number of the `array_api_tests`
226+
package based on the git tag. Push the tag to GitHub:
227+
228+
```
229+
git push --tags upstream 2022.1
230+
```
231+
232+
Then go to the [tags page on
233+
GitHub](https://github.com/data-apis/array-api-tests/tags) and convert the tag
234+
into a release. If you want, you can add release notes, which GitHub can
235+
generate for you.
236+
237+
234238
## Future plans
235239

236240
Keeping full coverage of the spec is an on-going priority as the Array API

0 commit comments

Comments
 (0)