@@ -50,47 +50,6 @@ a specific test case, which is useful when developing functions.
50
50
$ pytest array_api_tests/test_creation_functions.py::test_zeros
51
51
```
52
52
53
- ## Releases
54
-
55
- The test suite has tagged releases on
56
- [ GitHub] ( https://github.com/data-apis/array-api-tests/releases ) . If you run
57
- the test suite in your CI system, we recommend pinning against a release tag.
58
-
59
- We use [ calender versioning] ( https://calver.org/ ) for the releases. You should
60
- expect that any version may be "breaking" compared to the previous one, in the
61
- sense that there may have been additional tests added which cause a previously
62
- passing library to fail.
63
-
64
- For now, the test suite is
65
- not installable as a Python package. You can use it by cloning the repo and
66
- running ` pytest ` as described above. If it would help you to be able to
67
- install it as a package, [ please let us
68
- know] ( https://github.com/data-apis/array-api-tests/issues/85 ) .
69
-
70
- * Test suite maintainer note:* to make a release of the test suite, make an
71
- annotated tag with the version:
72
-
73
- ```
74
- git tag -a 2022.1
75
- ```
76
-
77
- (for the message, just write something like "array-api-tests version 2022.1").
78
- Be sure to use the calver version number for the tag name. Versioneer will
79
- automatically set the version number of the ` array_api_tests ` package based on
80
- the git tag.
81
-
82
- Then push the tag to GitHub
83
-
84
- ```
85
- git push --tags origin 2022.1
86
- ```
87
-
88
- Finally go to the [ tags page on
89
- GitHub] ( https://github.com/data-apis/array-api-tests/tags ) and convert the tag
90
- into a release. If you want, you can add release notes to the release page on
91
- GitHub.
92
-
93
-
94
53
## What the test suite covers
95
54
96
55
We are interested in array libraries conforming to the
@@ -147,7 +106,7 @@ of the functions and some miscellaneous things.
147
106
functions interact with them correctly.
148
107
149
108
Be aware that some aspects of the spec are impractical or impossible to actually
150
- test, so they are not covered in the suite <!-- TODO: note what these are -->
109
+ test, so they are not covered in the suite. <!-- TODO: note what these are -->
151
110
152
111
## Interpreting errors
153
112
@@ -172,7 +131,28 @@ behaviour different from the spec, or test something that is not documented,
172
131
this is a bug—please [ report such
173
132
issues] ( https://github.com/data-apis/array-api-tests/issues/ ) to us.
174
133
175
- ## Configuration
134
+
135
+ ## Running on CI
136
+
137
+ See our existing [ GitHub Actions workflow for
138
+ Numpy] ( https://github.com/data-apis/array-api-tests/blob/master/.github/workflows/numpy.yml )
139
+ for an example of using the test suite on CI.
140
+
141
+ ### Releases
142
+
143
+ We recommend pinning against a [ release tag] ( https://github.com/data-apis/array-api-tests/releases )
144
+ when running on CI.
145
+
146
+ We use [ calender versioning] ( https://calver.org/ ) for the releases. You should
147
+ expect that any version may be "breaking" compared to the previous one, in that
148
+ new tests (or improvements to existing tests) may cause a previously passing
149
+ library to fail.
150
+
151
+ ### Configuration
152
+
153
+ Use the ` --ci ` flag to run only the primary and special cases tests. You can
154
+ ignore the other test cases as they are redundant for the purposes of checking
155
+ compliance.
176
156
177
157
By default, tests for the optional Array API extensions such as
178
158
[ ` linalg ` ] ( https://data-apis.org/array-api/latest/extensions/linear_algebra_functions.html )
@@ -183,10 +163,9 @@ The tests make heavy use
183
163
[ Hypothesis] ( https://hypothesis.readthedocs.io/en/latest/ ) . You can configure
184
164
how many examples are generated using the ` --max-examples ` flag, which defaults
185
165
to 100. Lower values can be useful for quick checks, and larger values should
186
- result in more rigorous runs. For example, ` --max-examples 10000 ` may find bugs
187
- where default runs don't, but will take a much longer time .
166
+ result in more rigorous runs. For example, ` --max-examples 10_000 ` may find bugs
167
+ where default runs don't but will take much longer to run .
188
168
189
- <!-- TODO: howto on CI -->
190
169
191
170
## Contributing
192
171
@@ -231,6 +210,31 @@ where `path/to/array-api` is the path to a local clone of the [`array-api`
231
210
repo] ( https://github.com/data-apis/array-api/ ) . Edit ` generate_stubs.py ` to make
232
211
changes to the generated files.
233
212
213
+
214
+ ### Release
215
+
216
+ To make a release, first make an annotated tag with the version, e.g.:
217
+
218
+ ```
219
+ git tag -a 2022.01.01
220
+ ```
221
+
222
+ Be sure to use the calver version number for the tag name. Don't worry too much
223
+ on the tag message, e.g. just write "2022.01.01".
224
+
225
+ Versioneer will automatically set the version number of the ` array_api_tests `
226
+ package based on the git tag. Push the tag to GitHub:
227
+
228
+ ```
229
+ git push --tags upstream 2022.1
230
+ ```
231
+
232
+ Then go to the [ tags page on
233
+ GitHub] ( https://github.com/data-apis/array-api-tests/tags ) and convert the tag
234
+ into a release. If you want, you can add release notes, which GitHub can
235
+ generate for you.
236
+
237
+
234
238
## Future plans
235
239
236
240
Keeping full coverage of the spec is an on-going priority as the Array API
0 commit comments