Skip to content

Commit 7af9475

Browse files
simonmichaelsimonmichael
committed
update readme
1 parent 7fbdf2a commit 7af9475

File tree

1 file changed

+126
-112
lines changed

1 file changed

+126
-112
lines changed

‎README.md

Lines changed: 126 additions & 112 deletions
Original file line numberDiff line numberDiff line change
@@ -16,11 +16,6 @@ commands, released under GPLv3+. It reads simple test specifications
1616
defining a command to run, some input, and the expected output,
1717
stderr, and exit status. It can run tests in parallel, selectively,
1818
with a timeout, in color, etc.
19-
Projects using it include
20-
[hledger](http://hledger.org),
21-
[Agda](http://wiki.portal.chalmers.se/agda),
22-
and
23-
[berp](https://github.com/bjpop/berp).
2419

2520
## Install
2621

@@ -37,102 +32,114 @@ Or, build the latest release on any major platform:
3732

3833
|||
3934
|----------------|---------------------------------------
40-
| stack: | **[get stack](https://haskell-lang.org/get-started)**, **`stack install shelltestrunner-1.10`**
35+
| stack: | **[get stack](https://docs.haskellstack.org)**, **`stack install shelltestrunner-1.10`**
4136
| cabal: | **`cabal update; cabal install shelltestrunner-1.10`**
4237

4338
## Usage
4439

45-
Here's a minimal test file containing one shell test: <!-- keep synced with tests/examples -->
40+
Here's a test file containing three simple tests.
41+
They're called "shell tests" because any shell command line can be tested.
42+
A test contains:
4643

47-
# A comment. Testing bash's builtin "echo" command (if /bin/sh is bash)
48-
echo
49-
>>>= 0
44+
- `<` and one or more lines of input to be provided on stdin (optional)
45+
- `$` and the command line to test (required)
46+
- zero or more lines of expected stdout output, or a regexp (optional)
47+
- `>2` and zero or more lines of expected stderr output, or a regexp (optional)
48+
- `>=` and an expected exit code, or a regexp (optional)
5049

51-
They're called "shell test" because any shell (`/bin/sh` on POSIX, `CMD` on Windows)
52-
command line can be tested.
53-
Each test begins with the command to test, followed by optional stdin input,
54-
expected stdout and/or stderr output, and ends with the expected exit status.
55-
Here's another file containing two tests:
50+
<!-- keep synced with tests/examples -->
51+
```
52+
# 1. Test that the "echo" command (a shell builtin, usually)
53+
# prints its argument on stdout, prints nothing on stderr,
54+
# and exits with a zero exit code.
5655
57-
# Test that the "cat" program copies its input to stdout,
58-
# nothing appears on stderr, and exit status is 0.
59-
cat
60-
<<<
61-
foo
62-
>>>
63-
foo
64-
>>>2
65-
>>>= 0
66-
67-
# Test that cat prints an error containing "unrecognized option" or
68-
# "illegal option" and exits with non-zero status if given a bad flag.
69-
cat --no-such-flag
70-
>>>2 /(unrecognized|illegal) option/
71-
>>>= !0
56+
$ echo a
57+
a
58+
59+
# 2. Test that echo with no arguments prints a blank line,
60+
# no stderr output, and exits with zero.
61+
# Since the output ends with whitespace, this time we must write
62+
# the exit code test (>=) explicitly, to act as a delimiter.
63+
64+
$ echo
65+
66+
>=
67+
68+
# 3. Test that cat with a bad flag prints nothing on stdout,
69+
# an error containing "unrecognized option" or "illegal option" on stderr,
70+
# and exits with non-zero status.
71+
72+
$ cat --no-such-flag
73+
>2 /(unrecognized|illegal) option/
74+
>= !0
75+
76+
```
7277

7378
To run these tests:
7479

75-
$ shelltest echo.test cat.test
76-
:echo.test: [OK]
77-
:cat.test:1: [OK]
78-
:cat.test:2: [OK]
80+
```
81+
$ shelltest examples.test
82+
:examples.test:1: [OK]
83+
:examples.test:2: [OK]
84+
:examples.test:3: [OK]
7985
80-
Test Cases Total
81-
Passed 3 3
82-
Failed 0 0
83-
Total 3 3
86+
Test Cases Total
87+
Passed 3 3
88+
Failed 0 0
89+
Total 3 3
90+
```
8491

8592
That's the basics!
8693
There are also some alternate test formats you'll read about below.
8794

8895
## Options
8996

90-
<!--
91-
Command to generate doc:
92-
shelltest --help | sed -e '/^shelltest file formats/,$d' -e 's/^/ /'
93-
-->
94-
95-
$ shelltest --help
96-
shelltest 1.10
97-
98-
shelltest [OPTIONS] [TESTFILES|TESTDIRS]
99-
100-
Common flags:
101-
-l --list List the names of all tests found
102-
-i --include=PAT Include tests whose name contains this glob pattern
103-
(eg: -i1 -i{4,5,6})
104-
-x --exclude=STR Exclude test files whose path contains STR
105-
-a --all Show all output without truncating, even if large
106-
-c --color Show colored output if your terminal supports it
107-
-d --diff Show differences between expected/actual output
108-
--precise Show expected/actual output precisely, with quoting
109-
--hide-successes Show only test failures
110-
--xmlout=FILE Save test results to FILE in XML format.
111-
-D --defmacro=D=DEF Define a macro D to be replaced by DEF while parsing
112-
test files.
113-
--execdir Run tests from within each test file's directory
114-
--extension=EXT File suffix of test files (default: .test)
115-
-w --with=EXE Replace the first word of test commands with EXE
116-
(unindented commands only)
117-
-o --timeout=SECS Number of seconds a test may run (default: no limit)
118-
-j --threads=N Number of threads for running tests (default: 1)
119-
--shell=EXE The shell program to use (must accept -c CMD;
120-
default: /bin/sh on POSIX, cmd.exe on Windows)
121-
--debug Show debug info while running
122-
--debug-parse Show test file parsing results and stop
123-
Print test file:
124-
--print[=FORMAT] Print test files in specified format (default: v3).
125-
126-
-h --help Display help message
127-
-V --version Print version information
128-
--numeric-version Print just the version number
129-
97+
<!-- shelltest --help | sed -e '/^shelltest file formats/,$d' -->
98+
```
99+
shelltest 1.10
100+
101+
shelltest [OPTIONS] [TESTFILES|TESTDIRS]
102+
103+
Common flags:
104+
-l --list List the names of all tests found
105+
-i --include=PAT Include tests whose name contains this glob pattern
106+
(eg: -i1 -i{4,5,6})
107+
-x --exclude=STR Exclude test files whose path contains STR
108+
-a --all Show all output without truncating, even if large
109+
-c --color Show colored output if your terminal supports it
110+
-d --diff Show differences between expected/actual output
111+
-p --precise Show expected/actual output precisely, with quoting
112+
--hide-successes Show only test failures
113+
-f --fail-fast Only hspec: stop tests on first failure
114+
--xmlout=FILE Save test results to FILE in XML format.
115+
-D --defmacro=D=DEF Define a macro D to be replaced by DEF while parsing
116+
test files.
117+
--execdir Run tests from within each test file's directory
118+
--extension=EXT File suffix of test files (default: .test)
119+
-w --with=EXE Replace the first word of test commands with EXE
120+
(unindented commands only)
121+
-o --timeout=SECS Number of seconds a test may run (default: no limit)
122+
-j --threads=N Number of threads for running tests (default: 1)
123+
--shell=EXE The shell program to use (must accept -c CMD;
124+
default: /bin/sh on POSIX, cmd.exe on Windows)
125+
--debug Show debug info while running
126+
--debug-parse Show test file parsing results and stop
127+
Print test file:
128+
--print[=FORMAT] Print test files in specified format (default: v3).
129+
--hspec Use hspec to run tests.
130+
-h --help Display help message
131+
-V --version Print version information
132+
--numeric-version Print just the version number
133+
134+
```
130135

131136
`shelltest` accepts one or more test file or directory arguments.
132137
A directory means all files below it named `*.test` (customisable with `--extension`).
133138

134-
Test commands are run with `/bin/sh` on POSIX systems and with `CMD` on Windows.
135-
By default, they are run in the directory in which you ran `shelltest`;
139+
By default, test commands are run with `/bin/sh` on POSIX systems
140+
and with `CMD` on Windows; you can change this with the `--shell` option.
141+
142+
By default, tests run in the directory in which you ran `shelltest`;
136143
with `--execdir` they will run in each test file's directory instead.
137144

138145
`--include` selects only tests whose name (file name plus intra-file sequence number) matches a
@@ -168,19 +175,22 @@ For example, the command:
168175

169176
shelltestrunner supports three test file formats:
170177

171-
| Format name | Description | Delimiters, in order |
172-
|-------------|--------------------------------------------------------------------------------------------|----------------------------|
173-
| format 1 | old; command first, exit status is required | `(none) <<< >>> >>>2 >>>=` |
174-
| format 2 | new, verbose: input first, can be reused by multiple tests, some delimiters can be omitted | `<<< $$$ >>> >>>2 >>>=` |
175-
| format 3 | new, lightweight: like format 2, but with shorter delimiters | `< $ > >2 >=` |
178+
| Format name | Description | Delimiters, in order |
179+
|-----------------------|---------------------------------------------------------------|----------------------------|
180+
| format 1 (deprecated) | command is first; exit status is required | `(none) <<< >>> >>>2 >>>=` |
181+
| format 2 (verbose) | input is first, can be reused; all but command can be omitted | `<<< $$$ >>> >>>2 >>>=` |
182+
| format 3 (preferred) | same as format 2 but with short delimiters | `< $ > >2 >=` |
176183

177-
To read each file, shelltestrunner tries the formats in this order: first format 2, then format 3, then format 1.
178-
Format 3 is the lightest and most pleasant; you should use this one, unless it clashes with your data,
179-
in which case use the more verbose format 2. Format 1 is just for backward compatibility with old tests.
180-
All tests within a file should use the same format.
184+
To read each file, shelltestrunner tries the formats in this order: format 2, then format 3, then format 1.
185+
Within a file, all tests should use the same format.
186+
187+
Here are the formats in detail, from oldest to newest.
188+
You should use format 3; or if that clashes with your data, then format 2.
181189

182190
### Format 1
183191

192+
This old format is included for backward compatibility with old tests.
193+
184194
Test files contain one or more individual tests, each consisting of a
185195
one-line shell command, optional input, expected standard output
186196
and/or error output, and a (required) exit status.
@@ -199,18 +209,16 @@ When not specified, stdout/stderr are ignored.
199209
A space before the command protects it from -w/--with.
200210

201211
Examples:
202-
[above](#usage),
203212
[shelltestrunner](https://github.com/simonmichael/shelltestrunner/tree/master/tests/format1),
204-
[hledger](https://github.com/simonmichael/hledger/tree/master/tests),
205213
[Agda](https://github.com/agda/agda/tree/master/src/size-solver/test),
206214
[berp](https://github.com/bjpop/berp/tree/master/test/regression),
207215
[cblrepo](https://github.com/magthe/cblrepo/tree/master/tests).
208216

209217
### Format 2
210218

211-
(shelltestrunner 1.9+)
212-
This improves on format 1 in two ways: it allows tests to reuse the
213-
same input, and it allows delimiters to often be omitted.
219+
This is supported by shelltestrunner 1.9+.
220+
It improves on format 1 in two ways: it allows tests to reuse the same input,
221+
and it allows delimiters/test clauses to be omitted, with more useful defaults.
214222

215223
Test files contain one or more test groups.
216224
A test group consists of some optional standard input and one or more tests.
@@ -245,10 +253,10 @@ Expected output/stderr extends to the next `>>>2` or `>>>=` if present,
245253
or to the last non-blank/comment line before the next `<<<` or `$$$` or file end.
246254
`/REGEX/` regular expression patterns may be used instead of
247255
specifying the expected output in full. The regex syntax is
248-
[regex-tdfa](http://hackage.haskell.org/package/regex-tdfa)'s, plus
256+
[regex-tdfa](https://hackage.haskell.org/package/regex-tdfa)'s, plus
249257
you can put `!` before `/REGEX/` to negate the match.
250258

251-
The [exit status](http://en.wikipedia.org/wiki/Exit_status) is a
259+
The [exit status](https://en.wikipedia.org/wiki/Exit_status) is a
252260
number, normally 0 for a successful exit. This too can be prefixed
253261
with `!` to negate the match, or you can use a `/REGEX/` pattern.
254262
A `>>>=` with nothing after it ignores the exit status.
@@ -293,8 +301,8 @@ Non-required `<<<` and `>>>` delimiters omitted:
293301

294302
### Format 3
295303

296-
(shelltestrunner 1.9+)
297-
The same as format 2, but with more convenient short delimiters: < $ > >2 >=.
304+
This is supported by shelltestrunner 1.9+.
305+
It is the preferred format - like format 2 but with more convenient short delimiters:
298306

299307
# COMMENTS OR BLANK LINES
300308
<
@@ -347,7 +355,10 @@ Non-required `<` and `>` delimiters omitted:
347355

348356
>2
349357

350-
[shelltestrunner](https://github.com/simonmichael/shelltestrunner/tree/master/tests/format3)
358+
Also:
359+
[above](#usage),
360+
[shelltestrunner](https://github.com/simonmichael/shelltestrunner/tree/master/tests/format3),
361+
[hledger](https://github.com/simonmichael/hledger/tree/master/hledger/test).
351362

352363
## Printing tests
353364

@@ -374,8 +385,8 @@ In general, always review the result of a conversion yourself before committing
374385

375386
|||
376387
|----------------------|--------------------------------------------------|
377-
| Released version: | http://hackage.haskell.org/package/shelltestrunner
378-
| Changelog: | http://hackage.haskell.org/package/shelltestrunner/changelog
388+
| Released version: | https://hackage.haskell.org/package/shelltestrunner
389+
| Changelog: | https://hackage.haskell.org/package/shelltestrunner/changelog
379390
| Code | https://github.com/simonmichael/shelltestrunner
380391
| Issues | https://github.com/simonmichael/shelltestrunner/issues
381392
| Chat | Contact sm in the #hledger:matrix.org room on matrix or the #hledger channel on libera.chat
@@ -391,23 +402,26 @@ Feedback, testing, code, documentation, packaging, blogging, and funding are mos
391402

392403
## Credits
393404

394-
[Simon Michael](http://joyful.com) wrote shelltestrunner,
405+
[Simon Michael](https://joyful.com) wrote shelltestrunner,
395406
inspired by John Wiegley's tests for Ledger.
396407

397-
Code contributors include:
398-
Taavi Väljaots,
399-
John Macfarlane,
408+
Code contributors:
409+
Andreas Abel,
400410
Andrés Sicard-Ramírez,
401-
Iustin Pop,
402-
Trygve Laugstøl,
403411
Bernie Pope,
404-
Sergei Trofimovich,
412+
Felix C. Stegerman,
413+
Iustin Pop,
414+
Jakob Schöttl
405415
John Chee.
416+
John Macfarlane,
417+
Sergei Trofimovich,
418+
Taavi Väljaots,
419+
Trygve Laugstøl,
406420

407-
shelltestrunner depends on several fine libraries, in particular Max
408-
Bolingbroke's test-framework, and of course on the Glorious Haskell
409-
Compiler.
421+
shelltestrunner depends on several fine libraries,
422+
in particular Max Bolingbroke's test-framework,
423+
and of course on the Glorious Haskell Compiler.
410424

411425
The Blade Runner font is by Phil Steinschneider.
412426

413-
<!-- http://www.explore-science-fiction-movies.com/blade-runner-movie-quotes.html -->
427+
<!-- https://www.explore-science-fiction-movies.com/blade-runner-movie-quotes.html -->

0 commit comments

Comments
 (0)