chore: Add comment-json and the current jsonlint implementation to the performance comparison

Author: prantlf

README.md

@@ -268,15 +268,13 @@ If you want to retain comments or whitespace for pretty-printing, for example, s
 
 ### Performance
 
-This is a part of an output from the [parser benchmark], when parsing a 4.2 KB formatted string ([package.json](./package.json)) with Node.js 10.15.3:
+This is a part of an output from the [parser benchmark], when parsing a 4.2 KB formatted string ([package.json](./package.json)) with Node.js 12.14.0:
 
-    the built-in parser x 68,212 ops/sec ±0.86% (87 runs sampled)
-    the pure jju parser x 10,234 ops/sec ±1.08% (89 runs sampled)
-    the extended jju parser x 10,210 ops/sec ±1.26% (88 runs sampled)
-    the tokenizable jju parser x 8,832 ops/sec ±0.92% (89 runs sampled)
-    the tokenizing jju parser x 7,911 ops/sec ±1.05% (86 runs sampled)
+    jsonlint using native JSON.parse x 97,109 ops/sec ±0.81% (93 runs sampled)
+    jsonlint using hand-coded parser x 7,256 ops/sec ±0.54% (90 runs sampled)
+    jsonlint using tokenising parser x 6,387 ops/sec ±0.44% (88 runs sampled)
 
-A custom JSON parser is [a lot slower] than the built-in one. However, it is more important to have a [clear error reporting] than the highest speed in scenarios like parsing configuration files. Extending the parser with the support for comments and single-quoted strings does not affect the performance. Making the parser collect tokens and their locations decreases the performance a bit.
+A custom JSON parser is [a lot slower] than the built-in one. However, it is more important to have a [clear error reporting] than the highest speed in scenarios like parsing configuration files. (For better error-reporting, the speed can be preserved by using the native parser initially and re-parsing with another parser only in case of failure.) Features like comments or JSON5 are also helpful in configuration files. Tokens preserve the complete input and can be used for pretty-printing without losing the comments.
 
 ### Error Handling
 

benchmarks/fail.js

@@ -6,6 +6,7 @@ const { parse: jjuParse } = require('./jju/pure')
 const { parse: pegjsParse } = require('./pegjs/pure')
 const jisonParser = require('./jison/pure').parser
 const JSON5 = require('json5')
+const { parse: parseWithComments } = require('comment-json')
 
 const inputSources = [
   `{
@@ -184,6 +185,10 @@ function parseJju () {
   }
 }
 
+function parseCommentJson () {
+  parseWithComments(inputSource)
+}
+
 function improvePegjsError (error) {
   const { message, location } = error
   const { line, column, offset } = location.start
@@ -237,6 +242,7 @@ for (const test of inputSources) {
     .add('the chevrotain parser', parseChevrotain)
     .add('the hand-built parser', parseHandbuilt)
     .add('the jju parser', parseJju)
+    .add('the comment-json parser', parseCommentJson)
     .add('the pegjs parser', parsePegjs)
     .add('the jison parser', parseJison)
     .add('the JSON5 parser', parseJSON5)

benchmarks/package-lock.json

@@ -7,6 +7,7 @@
   "devDependencies": {
     "benchmark": "2.1.4",
     "chevrotain": "6.5.0",
+    "comment-json": "2.3.1",
     "jison": "0.4.18",
     "json-to-ast": "2.1.0",
     "json5": "2.1.1",

benchmarks/package.json

@@ -15,6 +15,8 @@ const handbuiltParse = require('./hand-built/pure')
 const handbuiltExtendedParse = require('./hand-built/pure')
 const astParse = require('json-to-ast')
 const JSON5 = require('json5')
+const { parse: parseWithComments } = require('comment-json')
+const { parse: parseThis, tokenize: tokenizeThis } = require('..')
 
 const pkg = require('../package')
 const input = JSON.stringify(pkg, undefined, 2)
@@ -58,6 +60,22 @@ function parseTokenisingJju () {
   })
 }
 
+function parseCommentJson () {
+  parseWithComments(input)
+}
+
+function parseCurrentStandard () {
+  parseThis(input)
+}
+
+function parseCurrentExtended () {
+  parseThis(input, { mode: 'json5' })
+}
+
+function parseCurrentTokenising () {
+  tokenizeThis(input, { mode: 'json5' })
+}
+
 function parsePurePegjs () {
   pegjsParse(input)
 }
@@ -88,13 +106,17 @@ createSuite(`Parsing JSON data ${input.length} characters long using`)
   .add('the built-in parser', parseBuiltIn)
   .add('the pure chevrotain parser', parsePureChevrotain)
   .add('the extended chevrotain parser', parseExtendedChevrotain)
+  .add('the standard jsonlint parser', parseCurrentStandard)
+  .add('the extended jsonlint parser', parseCurrentExtended)
+  .add('the tokenising jsonlint parser', parseCurrentTokenising)
   .add('the pure hand-built parser', parseHandbuilt)
   .add('the extended hand-built parser', parseExtendedHandbuilt)
   .add('the AST parser', parseAST)
   .add('the pure jju parser', parsePureJju)
   .add('the extended jju parser', parseExtendedJju)
   .add('the tokenisable jju parser', parseTokenisableJju)
   .add('the tokenising jju parser', parseTokenisingJju)
+  .add('the comments-enabled parser', parseCommentJson)
   .add('the pure pegjs parser', parsePurePegjs)
   .add('the extended pegjs parser', parseExtendedPegjs)
   .add('the pure jison parser', parsePureJison)

benchmarks/parse.js

@@ -27,23 +27,27 @@ Results
 
 This is a result of the [benchmark] run by `npm run benchmarks`. The numbers should be understood as relative ones:
 
-    Parsing JSON data 4479 characters long using:
-      the built-in parser x 67,113 ops/sec ±0.79% (89 runs sampled)
-      the pure chevrotain parser x 15,446 ops/sec ±1.30% (90 runs sampled)
-      the extended chevrotain parser x 14,168 ops/sec ±1.23% (88 runs sampled)
-      the pure hand-built parser x 9,755 ops/sec ±0.92% (89 runs sampled)
-      the extended hand-built parser x 9,633 ops/sec ±1.13% (88 runs sampled)
-      the AST parser x 9,768 ops/sec ±0.83% (89 runs sampled)
-      the pure jju parser x 10,178 ops/sec ±0.97% (84 runs sampled)
-      the extended jju parser x 10,204 ops/sec ±1.13% (88 runs sampled)
-      the tokenisable jju parser x 8,904 ops/sec ±1.01% (88 runs sampled)
-      the tokenising jju parser x 6,750 ops/sec ±0.97% (88 runs sampled)
-      the pure pegjs parser x 3,083 ops/sec ±0.95% (89 runs sampled)
-      the extended pegjs parser x 2,786 ops/sec ±1.12% (89 runs sampled)
-      the pure jison parser x 3,033 ops/sec ±1.19% (88 runs sampled)
-      the extended jison parser x 2,810 ops/sec ±0.65% (88 runs sampled)
-      the JSON5 parser x 2,085 ops/sec ±0.45% (91 runs sampled)
-    The fastest one was the built-in parser.
+    Parsing JSON data 4589 characters long using:
+      the built-in parser x 97,039 ops/sec ±0.69% (92 runs sampled)
+      the pure chevrotain parser x 11,981 ops/sec ±0.86% (87 runs sampled)
+      the extended chevrotain parser x 11,183 ops/sec ±0.54% (88 runs sampled)
+      the standard jsonlint parser x 97,109 ops/sec ±0.81% (93 runs sampled)
+      the extended jsonlint parser x 7,256 ops/sec ±0.54% (90 runs sampled)
+      the tokenising jsonlint parser x 6,387 ops/sec ±0.44% (88 runs sampled)
+      the pure hand-built parser x 7,508 ops/sec ±0.49% (86 runs sampled)
+      the extended hand-built parser x 7,517 ops/sec ±0.45% (90 runs sampled)
+      the AST parser x 8,008 ops/sec ±0.90% (85 runs sampled)
+      the pure jju parser x 7,505 ops/sec ±0.64% (89 runs sampled)
+      the extended jju parser x 7,352 ops/sec ±0.45% (90 runs sampled)
+      the tokenisable jju parser x 6,636 ops/sec ±0.46% (89 runs sampled)
+      the tokenising jju parser x 5,373 ops/sec ±0.54% (89 runs sampled)
+      the comments-enabled parser x 6,258 ops/sec ±0.95% (88 runs sampled)
+      the pure pegjs parser x 2,803 ops/sec ±0.58% (88 runs sampled)
+      the extended pegjs parser x 2,526 ops/sec ±0.74% (87 runs sampled)
+      the pure jison parser x 2,460 ops/sec ±0.49% (89 runs sampled)
+      the extended jison parser x 2,195 ops/sec ±0.63% (88 runs sampled)
+      the JSON5 parser x 1,660 ops/sec ±0.84% (90 runs sampled)
+    The fastest one was the built-in parser,the standard jsonlint parser.
 
 I looked further at capabilities and licenses of the parsers.
 
@@ -86,6 +90,13 @@ I looked further at capabilities and licenses of the parsers.
 * Supports `reviver` for the full compatibility with JSON.parse.
 * Can generate tokens to analyse, modify and update the original JSON input.
 
+[comment-json]
+--------------
+
+* Very fast one.
+* Supports `reviver` for the full compatibility with JSON.parse.
+* Supports CJSON - JSON with JavaScript-like comments.
+
 [PEG.JS]
 --------
 
@@ -105,13 +116,20 @@ I looked further at capabilities and licenses of the parsers.
 * Hand-built parser implementing the [JSON5 specification]. Extensions mentioned above were included, and even more, like trailing commas or multi-line strings. No support for pure JSON.
 * Slower one.
 
+[JSONLint]
+----------
+
+This entry is here just to compare the current implementation to the original "contenders". Based on the [evaluation] results, the current JSONLint uses a hand-coded parser based on [JJU].
+
 [tested with a JSON grammar]: https://sap.github.io/chevrotain/performance/
 [quality of error reporting]: ./errorReportingQuality.md
+[evaluation]: ./evaluation.md
 [benchmark]: ../parse.js
 [Built-in]: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/JSON/parse
 [Chevrotain]: https://github.com/SAP/chevrotain
 [Hand-built]: https://github.com/sap/chevrotain/blob/gh-pages/performance/jsonParsers/handbuilt/handbuilt.js
 [JJU]: http://rlidwka.github.io/jju/
+[comment-json]: https://github.com/kaelzhang/node-comment-json
 [AST]: https://github.com/vtrushin/json-to-ast
 [PEG.JS]: http://pegjs.org/
 [Jison]: http://zaach.github.io/jison/