Skip to content

Commit a1873d7

Browse files
jrfnljrfnl
committed
Reports: add end-to-end tests
... for quite a few report types. The Reports which PHPCS can generate, were until now not covered by tests and therefore had no safeguards/QA whatsoever. This commit sets up an initial end-to-end test suite with tests for common report types. It doesn't have an opinion on whether or not the reports as currently generated are formatted correctly. For now, it just codifies the current functionality. For lack of documentation about this format in the PHPUnit docs, the test docs for PHP Core are the closest available to explain the available sections and how to apply these: https://qa.php.net/phpt_details.php With this initial setup done, it should be fairly straight-forward to expand this end-to-end test suite to cover more report types and to add tests for other command-line options as well. Mind: these type of tests are generally slow, so unit tests/integration tests should be preferred.
1 parent 2dc7b59 commit a1873d7

22 files changed

+710
-4
lines changed

‎package.xml

+62
Original file line numberDiff line numberDiff line change
@@ -273,6 +273,32 @@ http://pear.php.net/dtd/package-2.0.xsd">
273273
<file baseinstalldir="" name="ErrorSuppressionTest.php" role="test" />
274274
<file baseinstalldir="" name="IsCamelCapsTest.php" role="test" />
275275
</dir>
276+
<dir name="EndToEnd">
277+
<dir name="Fixtures">
278+
<dir name="Reports">
279+
<file baseinstalldir="" name="Dirty.php" role="test" />
280+
<file baseinstalldir="" name="CleanClass.php" role="test" />
281+
</dir>
282+
</dir>
283+
<dir name="Reports">
284+
<file baseinstalldir="" name="checkstyle.phpt" role="test" />
285+
<file baseinstalldir="" name="code-no-errorcodes.phpt" role="test" />
286+
<file baseinstalldir="" name="code-with-errorcodes.phpt" role="test" />
287+
<file baseinstalldir="" name="csv.phpt" role="test" />
288+
<file baseinstalldir="" name="emacs-no-errorcodes.phpt" role="test" />
289+
<file baseinstalldir="" name="emacs-with-errorcodes.phpt" role="test" />
290+
<file baseinstalldir="" name="full-no-errorcodes.phpt" role="test" />
291+
<file baseinstalldir="" name="full-with-errorcodes.phpt" role="test" />
292+
<file baseinstalldir="" name="gitblame-no-errorcodes.phpt" role="test" />
293+
<file baseinstalldir="" name="gitblame-with-errorcodes.phpt" role="test" />
294+
<file baseinstalldir="" name="json.phpt" role="test" />
295+
<file baseinstalldir="" name="junit.phpt" role="test" />
296+
<file baseinstalldir="" name="source-no-errorcodes.phpt" role="test" />
297+
<file baseinstalldir="" name="source-with-errorcodes.phpt" role="test" />
298+
<file baseinstalldir="" name="summary.phpt" role="test" />
299+
<file baseinstalldir="" name="xml.phpt" role="test" />
300+
</dir>
301+
</dir>
276302
<dir name="Standards">
277303
<file baseinstalldir="" name="AbstractSniffUnitTest.php" role="test" />
278304
<file baseinstalldir="" name="AllSniffs.php" role="test" />
@@ -2264,6 +2290,24 @@ http://pear.php.net/dtd/package-2.0.xsd">
22642290
<install as="CodeSniffer/Core/Tokenizer/TypeIntersectionTest.inc" name="tests/Core/Tokenizer/TypeIntersectionTest.inc" />
22652291
<install as="CodeSniffer/Core/Tokenizer/UndoNamespacedNameSingleTokenTest.php" name="tests/Core/Tokenizer/UndoNamespacedNameSingleTokenTest.php" />
22662292
<install as="CodeSniffer/Core/Tokenizer/UndoNamespacedNameSingleTokenTest.inc" name="tests/Core/Tokenizer/UndoNamespacedNameSingleTokenTest.inc" />
2293+
<install as="CodeSniffer/EndToEnd/Fixtures/Reports/Dirty.php" name="tests/EndToEnd/Fixtures/Reports/Dirty.php" />
2294+
<install as="CodeSniffer/EndToEnd/Fixtures/Reports/CleanClass.php" name="tests/EndToEnd/Fixtures/Reports/CleanClass.php" />
2295+
<install as="CodeSniffer/EndToEnd/Reports/checkstyle.phpt" name="tests/EndToEnd/Reports/checkstyle.phpt" />
2296+
<install as="CodeSniffer/EndToEnd/Reports/code-no-errorcodes.phpt" name="tests/EndToEnd/Reports/code-no-errorcodes.phpt" />
2297+
<install as="CodeSniffer/EndToEnd/Reports/code-with-errorcodes.phpt" name="tests/EndToEnd/Reports/code-with-errorcodes.phpt" />
2298+
<install as="CodeSniffer/EndToEnd/Reports/csv.phpt" name="tests/EndToEnd/Reports/csv.phpt" />
2299+
<install as="CodeSniffer/EndToEnd/Reports/emacs-no-errorcodes.phpt" name="tests/EndToEnd/Reports/emacs-no-errorcodes.phpt" />
2300+
<install as="CodeSniffer/EndToEnd/Reports/emacs-with-errorcodes.phpt" name="tests/EndToEnd/Reports/emacs-with-errorcodes.phpt" />
2301+
<install as="CodeSniffer/EndToEnd/Reports/full-no-errorcodes.phpt" name="tests/EndToEnd/Reports/full-no-errorcodes.phpt" />
2302+
<install as="CodeSniffer/EndToEnd/Reports/full-with-errorcodes.phpt" name="tests/EndToEnd/Reports/full-with-errorcodes.phpt" />
2303+
<install as="CodeSniffer/EndToEnd/Reports/gitblame-no-errorcodes.phpt" name="tests/EndToEnd/Reports/gitblame-no-errorcodes.phpt" />
2304+
<install as="CodeSniffer/EndToEnd/Reports/gitblame-with-errorcodes.phpt" name="tests/EndToEnd/Reports/gitblame-with-errorcodes.phpt" />
2305+
<install as="CodeSniffer/EndToEnd/Reports/json.phpt" name="tests/EndToEnd/Reports/json.phpt" />
2306+
<install as="CodeSniffer/EndToEnd/Reports/junit.phpt" name="tests/EndToEnd/Reports/junit.phpt" />
2307+
<install as="CodeSniffer/EndToEnd/Reports/source-no-errorcodes.phpt" name="tests/EndToEnd/Reports/source-no-errorcodes.phpt" />
2308+
<install as="CodeSniffer/EndToEnd/Reports/source-with-errorcodes.phpt" name="tests/EndToEnd/Reports/source-with-errorcodes.phpt" />
2309+
<install as="CodeSniffer/EndToEnd/Reports/summary.phpt" name="tests/EndToEnd/Reports/summary.phpt" />
2310+
<install as="CodeSniffer/EndToEnd/Reports/xml.phpt" name="tests/EndToEnd/Reports/xml.phpt" />
22672311
<install as="CodeSniffer/Standards/AllSniffs.php" name="tests/Standards/AllSniffs.php" />
22682312
<install as="CodeSniffer/Standards/AbstractSniffUnitTest.php" name="tests/Standards/AbstractSniffUnitTest.php" />
22692313
</filelist>
@@ -2388,6 +2432,24 @@ http://pear.php.net/dtd/package-2.0.xsd">
23882432
<install as="CodeSniffer/Core/Tokenizer/TypeIntersectionTest.inc" name="tests/Core/Tokenizer/TypeIntersectionTest.inc" />
23892433
<install as="CodeSniffer/Core/Tokenizer/UndoNamespacedNameSingleTokenTest.php" name="tests/Core/Tokenizer/UndoNamespacedNameSingleTokenTest.php" />
23902434
<install as="CodeSniffer/Core/Tokenizer/UndoNamespacedNameSingleTokenTest.inc" name="tests/Core/Tokenizer/UndoNamespacedNameSingleTokenTest.inc" />
2435+
<install as="CodeSniffer/EndToEnd/Fixtures/Reports/Dirty.php" name="tests/EndToEnd/Fixtures/Reports/Dirty.php" />
2436+
<install as="CodeSniffer/EndToEnd/Fixtures/Reports/CleanClass.php" name="tests/EndToEnd/Fixtures/Reports/CleanClass.php" />
2437+
<install as="CodeSniffer/EndToEnd/Reports/checkstyle.phpt" name="tests/EndToEnd/Reports/checkstyle.phpt" />
2438+
<install as="CodeSniffer/EndToEnd/Reports/code-no-errorcodes.phpt" name="tests/EndToEnd/Reports/code-no-errorcodes.phpt" />
2439+
<install as="CodeSniffer/EndToEnd/Reports/code-with-errorcodes.phpt" name="tests/EndToEnd/Reports/code-with-errorcodes.phpt" />
2440+
<install as="CodeSniffer/EndToEnd/Reports/csv.phpt" name="tests/EndToEnd/Reports/csv.phpt" />
2441+
<install as="CodeSniffer/EndToEnd/Reports/emacs-no-errorcodes.phpt" name="tests/EndToEnd/Reports/emacs-no-errorcodes.phpt" />
2442+
<install as="CodeSniffer/EndToEnd/Reports/emacs-with-errorcodes.phpt" name="tests/EndToEnd/Reports/emacs-with-errorcodes.phpt" />
2443+
<install as="CodeSniffer/EndToEnd/Reports/full-no-errorcodes.phpt" name="tests/EndToEnd/Reports/full-no-errorcodes.phpt" />
2444+
<install as="CodeSniffer/EndToEnd/Reports/full-with-errorcodes.phpt" name="tests/EndToEnd/Reports/full-with-errorcodes.phpt" />
2445+
<install as="CodeSniffer/EndToEnd/Reports/gitblame-no-errorcodes.phpt" name="tests/EndToEnd/Reports/gitblame-no-errorcodes.phpt" />
2446+
<install as="CodeSniffer/EndToEnd/Reports/gitblame-with-errorcodes.phpt" name="tests/EndToEnd/Reports/gitblame-with-errorcodes.phpt" />
2447+
<install as="CodeSniffer/EndToEnd/Reports/json.phpt" name="tests/EndToEnd/Reports/json.phpt" />
2448+
<install as="CodeSniffer/EndToEnd/Reports/junit.phpt" name="tests/EndToEnd/Reports/junit.phpt" />
2449+
<install as="CodeSniffer/EndToEnd/Reports/source-no-errorcodes.phpt" name="tests/EndToEnd/Reports/source-no-errorcodes.phpt" />
2450+
<install as="CodeSniffer/EndToEnd/Reports/source-with-errorcodes.phpt" name="tests/EndToEnd/Reports/source-with-errorcodes.phpt" />
2451+
<install as="CodeSniffer/EndToEnd/Reports/summary.phpt" name="tests/EndToEnd/Reports/summary.phpt" />
2452+
<install as="CodeSniffer/EndToEnd/Reports/xml.phpt" name="tests/EndToEnd/Reports/xml.phpt" />
23912453
<install as="CodeSniffer/Standards/AllSniffs.php" name="tests/Standards/AllSniffs.php" />
23922454
<install as="CodeSniffer/Standards/AbstractSniffUnitTest.php" name="tests/Standards/AbstractSniffUnitTest.php" />
23932455
<ignore name="bin/phpcs.bat" />

‎phpcs.xml.dist

+1
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@
1111
<exclude-pattern>*/src/Standards/*/Tests/*\.(inc|css|js)$</exclude-pattern>
1212
<exclude-pattern>*/tests/Core/*/*\.(inc|css|js)$</exclude-pattern>
1313
<exclude-pattern>*/tests/Core/*/Fixtures/*\.php$</exclude-pattern>
14+
<exclude-pattern>*/tests/EndToEnd/Fixtures/*</exclude-pattern>
1415

1516
<arg name="basepath" value="."/>
1617
<arg name="colors"/>

‎phpunit.xml.dist

+4
Original file line numberDiff line numberDiff line change
@@ -4,5 +4,9 @@
44
<testsuite name="PHP_CodeSniffer Test Suite">
55
<file>tests/AllTests.php</file>
66
</testsuite>
7+
<testsuite name="End2End">
8+
<directory suffix=".phpt">tests/EndToEnd/</directory>
9+
<exclude>tests/EndToEnd/Fixtures</exclude>
10+
</testsuite>
711
</testsuites>
812
</phpunit>

‎scripts/ValidatePEAR/ValidatePEARPackageXML.php

+4-4
Original file line numberDiff line numberDiff line change
@@ -131,12 +131,12 @@ protected function checkContents()
131131
$srcFiles = (new FileList(
132132
$this->projectRoot.'src/',
133133
$this->projectRoot,
134-
'`\.(css|fixed|inc|js|php|xml)$`Di'
134+
'`\.(css|fixed|inc|js|phpt|php|xml)$`Di'
135135
))->getList();
136136
$testsFiles = (new FileList(
137137
$this->projectRoot.'tests/',
138138
$this->projectRoot,
139-
'`\.(css|inc|js|php|xml)$`Di'
139+
'`\.(css|inc|js|phpt|php|xml)$`Di'
140140
))->getList();
141141
$files = array_merge($srcFiles, $testsFiles);
142142

@@ -317,7 +317,7 @@ protected function checkPHPRelease()
317317
}
318318

319319
// Check validity of the tags for files in the tests root subdirectories.
320-
if (preg_match('`^tests/.+\.(php|inc|js|css|xml)$`', $name) === 1
320+
if (preg_match('`^tests/.+\.(phpt|php|inc|js|css|xml)$`', $name) === 1
321321
&& $as === str_replace('tests/', 'CodeSniffer/', $name)
322322
) {
323323
continue;
@@ -338,7 +338,7 @@ protected function checkPHPRelease()
338338
* Verify that all files in the `tests` directory are listed in both `<phprelease>` tags.
339339
*/
340340

341-
$testFiles = (new FileList($this->projectRoot.'tests/', $this->projectRoot, '`\.(inc|php|js|css|xml)$`Di'))->getList();
341+
$testFiles = (new FileList($this->projectRoot.'tests/', $this->projectRoot, '`\.(inc|phpt|php|js|css|xml)$`Di'))->getList();
342342

343343
foreach ($testFiles as $file) {
344344
foreach ($listedFiles as $key => $listed) {

‎tests/EndToEnd/Fixtures/Reports/CleanClass.php

+12
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
<?php
2+
/*
3+
* Test fixture file for End to End Report tests.
4+
*/
5+
6+
namespace PHP_CodeSniffer\Tests\EndToEnd\Fixtures;
7+
8+
class CleanClass {
9+
const UPPERCASE = true;
10+
11+
public function myMethod() {}
12+
}

‎tests/EndToEnd/Fixtures/Reports/Dirty.php

+14
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
<?php
2+
/*
3+
* Test fixture file for End to End Report tests.
4+
*/
5+
6+
class dirty_class {
7+
const lowerCase = false;
8+
9+
public function My_Method() {}
10+
}
11+
12+
class Second_class {}
13+
14+
$obj = new dirty_class();

‎tests/EndToEnd/Reports/checkstyle.phpt

+31
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
--TEST--
2+
Report: Checkstyle
3+
4+
--SKIPIF--
5+
<?php
6+
if (is_file(__DIR__.'/../../../autoload.php') === false) {
7+
print 'skip: Test cannot run from a PEAR install.';
8+
}
9+
?>
10+
--ARGS--
11+
./tests/EndToEnd/Fixtures/Reports/ -q --no-colors --report-width=80 --basepath=./tests/EndToEnd/Fixtures/Reports/ --standard=PSR1 --report=Checkstyle
12+
13+
--FILE--
14+
<?php
15+
require_once __DIR__ . '/../../../bin/phpcs';
16+
17+
--EXPECTF--
18+
#!/usr/bin/env php
19+
<?xml version="1.0" encoding="UTF-8"?>
20+
<checkstyle version="%s">
21+
<file name="Dirty.php">
22+
<error line="1" column="1" severity="warning" message="A file should declare new symbols (classes, functions, constants, etc.) and cause no other side effects, or it should execute logic with side effects, but should not do both. The first symbol is defined on line 6 and the first side effect is on line 14." source="PSR1.Files.SideEffects.FoundWithSymbols"/>
23+
<error line="6" column="1" severity="error" message="Each class must be in a namespace of at least one level (a top-level vendor name)" source="PSR1.Classes.ClassDeclaration.MissingNamespace"/>
24+
<error line="6" column="1" severity="error" message="Class name &quot;dirty_class&quot; is not in PascalCase format" source="Squiz.Classes.ValidClassName.NotCamelCaps"/>
25+
<error line="7" column="11" severity="error" message="Class constants must be uppercase; expected LOWERCASE but found lowerCase" source="Generic.NamingConventions.UpperCaseConstantName.ClassConstantNotUpperCase"/>
26+
<error line="9" column="12" severity="error" message="Method name &quot;dirty_class::My_Method&quot; is not in camel caps format" source="PSR1.Methods.CamelCapsMethodName.NotCamelCaps"/>
27+
<error line="12" column="1" severity="error" message="Each class must be in a file by itself" source="PSR1.Classes.ClassDeclaration.MultipleClasses"/>
28+
<error line="12" column="1" severity="error" message="Each class must be in a namespace of at least one level (a top-level vendor name)" source="PSR1.Classes.ClassDeclaration.MissingNamespace"/>
29+
<error line="12" column="1" severity="error" message="Class name &quot;Second_class&quot; is not in PascalCase format" source="Squiz.Classes.ValidClassName.NotCamelCaps"/>
30+
</file>
31+
</checkstyle>

‎tests/EndToEnd/Reports/code-no-errorcodes.phpt

+73
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
--TEST--
2+
Report: Code, no error codes
3+
4+
--SKIPIF--
5+
<?php
6+
if (is_file(__DIR__.'/../../../autoload.php') === false) {
7+
print 'skip: Test cannot run from a PEAR install.';
8+
}
9+
?>
10+
--ARGS--
11+
./tests/EndToEnd/Fixtures/Reports/ -q --no-colors --report-width=80 --basepath=./tests/EndToEnd/Fixtures/Reports/ --standard=PSR1 --report=Code
12+
13+
--FILE--
14+
<?php
15+
require_once __DIR__ . '/../../../bin/phpcs';
16+
17+
--EXPECTF--
18+
#!/usr/bin/env php
19+
20+
FILE: Dirty.php
21+
--------------------------------------------------------------------------------
22+
FOUND 7 ERRORS AND 1 WARNING AFFECTING 5 LINES
23+
--------------------------------------------------------------------------------
24+
LINE 1: WARNING A file should declare new symbols (classes, functions,
25+
constants, etc.) and cause no other side effects, or it should
26+
execute logic with side effects, but should not do both. The
27+
first symbol is defined on line 6 and the first side effect is
28+
on line 14.
29+
--------------------------------------------------------------------------------
30+
>> 1: <?php
31+
2: /*
32+
3: * Test fixture file for End to End Report tests.
33+
--------------------------------------------------------------------------------
34+
LINE 6: ERROR Each class must be in a namespace of at least one level (a
35+
top-level vendor name)
36+
LINE 6: ERROR Class name "dirty_class" is not in PascalCase format
37+
--------------------------------------------------------------------------------
38+
4: */
39+
5:%w
40+
>> 6: class dirty_class {
41+
7: const lowerCase = false;
42+
8:
43+
--------------------------------------------------------------------------------
44+
LINE 7: ERROR Class constants must be uppercase; expected LOWERCASE but found
45+
lowerCase
46+
--------------------------------------------------------------------------------
47+
5:%w
48+
6: class dirty_class {
49+
>> 7: const lowerCase = false;
50+
8:%w
51+
9: public function My_Method() {}
52+
--------------------------------------------------------------------------------
53+
LINE 9: ERROR Method name "dirty_class::My_Method" is not in camel caps
54+
format
55+
--------------------------------------------------------------------------------
56+
7: const lowerCase = false;
57+
8:%w
58+
>> 9: public function My_Method() {}
59+
10: }
60+
11:
61+
--------------------------------------------------------------------------------
62+
LINE 12: ERROR Each class must be in a file by itself
63+
LINE 12: ERROR Each class must be in a namespace of at least one level (a
64+
top-level vendor name)
65+
LINE 12: ERROR Class name "Second_class" is not in PascalCase format
66+
--------------------------------------------------------------------------------
67+
10: }
68+
11:%w
69+
>> 12: class Second_class {}
70+
13:%w
71+
14: $obj = new dirty_class();
72+
--------------------------------------------------------------------------------
73+
Time: %f secs; Memory: %dMB

‎tests/EndToEnd/Reports/code-with-errorcodes.phpt

+79
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,79 @@
1+
--TEST--
2+
Report: Code, with error codes
3+
4+
--SKIPIF--
5+
<?php
6+
if (is_file(__DIR__.'/../../../autoload.php') === false) {
7+
print 'skip: Test cannot run from a PEAR install.';
8+
}
9+
?>
10+
--ARGS--
11+
./tests/EndToEnd/Fixtures/Reports/ -qs --no-colors --report-width=80 --basepath=./tests/EndToEnd/Fixtures/Reports/ --standard=PSR1 --report=Code
12+
13+
--FILE--
14+
<?php
15+
require_once __DIR__ . '/../../../bin/phpcs';
16+
17+
--EXPECTF--
18+
#!/usr/bin/env php
19+
20+
FILE: Dirty.php
21+
--------------------------------------------------------------------------------
22+
FOUND 7 ERRORS AND 1 WARNING AFFECTING 5 LINES
23+
--------------------------------------------------------------------------------
24+
LINE 1: WARNING A file should declare new symbols (classes, functions, constants,
25+
etc.) and cause no other side effects, or it should execute logic with
26+
side effects, but should not do both. The first symbol is defined on
27+
line 6 and the first side effect is on line 14.
28+
(PSR1.Files.SideEffects.FoundWithSymbols)
29+
--------------------------------------------------------------------------------
30+
>> 1: <?php
31+
2: /*
32+
3: * Test fixture file for End to End Report tests.
33+
--------------------------------------------------------------------------------
34+
LINE 6: ERROR Each class must be in a namespace of at least one level (a
35+
top-level vendor name)
36+
(PSR1.Classes.ClassDeclaration.MissingNamespace)
37+
LINE 6: ERROR Class name "dirty_class" is not in PascalCase format
38+
(Squiz.Classes.ValidClassName.NotCamelCaps)
39+
--------------------------------------------------------------------------------
40+
4: */
41+
5:%w
42+
>> 6: class dirty_class {
43+
7: const lowerCase = false;
44+
8:
45+
--------------------------------------------------------------------------------
46+
LINE 7: ERROR Class constants must be uppercase; expected LOWERCASE but found
47+
lowerCase
48+
(Generic.NamingConventions.UpperCaseConstantName.ClassConstantNotUpperCase)
49+
--------------------------------------------------------------------------------
50+
5:%w
51+
6: class dirty_class {
52+
>> 7: const lowerCase = false;
53+
8:%w
54+
9: public function My_Method() {}
55+
--------------------------------------------------------------------------------
56+
LINE 9: ERROR Method name "dirty_class::My_Method" is not in camel caps
57+
format (PSR1.Methods.CamelCapsMethodName.NotCamelCaps)
58+
--------------------------------------------------------------------------------
59+
7: const lowerCase = false;
60+
8:%w
61+
>> 9: public function My_Method() {}
62+
10: }
63+
11:
64+
--------------------------------------------------------------------------------
65+
LINE 12: ERROR Each class must be in a file by itself
66+
(PSR1.Classes.ClassDeclaration.MultipleClasses)
67+
LINE 12: ERROR Each class must be in a namespace of at least one level (a
68+
top-level vendor name)
69+
(PSR1.Classes.ClassDeclaration.MissingNamespace)
70+
LINE 12: ERROR Class name "Second_class" is not in PascalCase format
71+
(Squiz.Classes.ValidClassName.NotCamelCaps)
72+
--------------------------------------------------------------------------------
73+
10: }
74+
11:%w
75+
>> 12: class Second_class {}
76+
13:%w
77+
14: $obj = new dirty_class();
78+
--------------------------------------------------------------------------------
79+
Time: %f secs; Memory: %dMB

0 commit comments

Comments
 (0)