Skip to content

Commit bb9216e

Browse files
committed
Tokenizer / numeric separators backfill: fix parse errors being handled by the backfill
The prevents the backfill from kicking in when confronted with invalid use of the numeric literal separator which would be a parse error in PHP 7.4 anyway. This fixes the remaining unit test failures.
1 parent 58b2e59 commit bb9216e

File tree

1 file changed

+14
-4
lines changed

1 file changed

+14
-4
lines changed

src/Tokenizers/PHP.php

Lines changed: 14 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -992,14 +992,24 @@ protected function tokenize($string)
992992

993993
if ($tokens[$i][0] === T_LNUMBER
994994
|| $tokens[$i][0] === T_DNUMBER
995-
|| ($tokens[$i][0] === T_STRING
996-
&& $tokens[$i][1][0] === '_')
995+
) {
996+
$newContent .= $tokens[$i][1];
997+
continue;
998+
}
999+
1000+
if ($tokens[$i][0] === T_STRING
1001+
&& $tokens[$i][1][0] === '_'
1002+
&& ((strpos($newContent, '0x') === 0
1003+
&& preg_match('`^((?<!\.)_[0-9A-F][0-9A-F\.]*)+$`iD', $tokens[$i][1]) === 1)
1004+
|| (strpos($newContent, '0x') !== 0
1005+
&& substr($newContent, -1) !== '.'
1006+
&& substr(strtolower($newContent), -1) !== 'e'
1007+
&& preg_match('`^(?:(?<![\.e])_[0-9][0-9e\.]*)+$`iD', $tokens[$i][1]) === 1))
9971008
) {
9981009
$newContent .= $tokens[$i][1];
9991010

10001011
// Support floats.
1001-
if ($tokens[$i][0] === T_STRING
1002-
&& substr(strtolower($tokens[$i][1]), -1) === 'e'
1012+
if (substr(strtolower($tokens[$i][1]), -1) === 'e'
10031013
&& ($tokens[($i + 1)] === '-'
10041014
|| $tokens[($i + 1)] === '+')
10051015
) {

0 commit comments

Comments
 (0)