Add support for JSON Lines

Author: cdauth

README.md

@@ -82,7 +82,7 @@ const jsonStream = stringifyJsonStream({
 
 Each value anywhere the JSON document can be a synchronous value, a promise that resolves to a value, or a sync/async function returning a value. Each value can also be an object stream (created by `objectStream()`), an array stream (created by `arrayStream()`) or a string stream (created by `stringStream()`) (see [stream generators](#stream-generators) for details). String streams can also be used as property keys inside object streams. The stream creators accept an `Iterable`, an `AsyncIterable` or a `ReadableStream` data source.
 
-The stringified JSON stream created by `JsonStringifier` is a `ReadableStream<string>`. Since most JavaScript methods work with `ReadableStream<Uint8Array>` instead, we can convert it using [`TextEncoderStream`](https://developer.mozilla.org/en-US/docs/Web/API/TextEncoderStream) (Node.js >= 18 required). Here is an example of streaming the result in an Express app:
+The stringified JSON stream created by `stringifyJsonStream` is a `ReadableStream<string>`. Since most JavaScript methods work with `ReadableStream<Uint8Array>` instead, we can convert it using [`TextEncoderStream`](https://developer.mozilla.org/en-US/docs/Web/API/TextEncoderStream) (Node.js >= 18 required). Here is an example of streaming the result in an Express app:
 ```typescript
 import { stringifyJsonStream, objectStream, arrayStream, stringStream } from "json-stream-es";
 import { Writable } from "node:stream";
@@ -101,6 +101,25 @@ If you prefer the generated JSON to be indented, you can pass a number or string
 
 Please also take note of the [differences between `JSON.stringify()` and `stringifyJsonStream()`](#differences-to-jsonstringify).
 
+#### Emitting multiple JSON values (JSONL)
+
+The [JSON Lines](https://jsonlines.org/) standard is a variation of JSON that allows multiple JSON values on the root level, separated by newlines. To emit such a stream, use the [`stringifyMultiJsonStream`](#stringifymultijsonstream) function, which returns a `TransformStream<JsonValue, string>`:
+```typescript
+import { stringifyMultiJsonStream } from "json-stream-es";
+
+const values = [
+	{ test1: "object1" },
+	{ test2: "object2" }
+];
+
+iterableToStream(values)
+	.pipeThrough(stringifyMultiJsonStream())
+	.pipeThrough(new TextEncoderStream())
+	.pipeTo(Writable.toWeb(res));
+```
+
+The individual values may also contain object/array/string streams like in the examples above.
+
 ### Consume a JSON stream
 
 [`parseJsonStream()`](#parsejsonstream) parses a stringified JSON stream, selects specific array items or object properties from it and emits their values. It consumes a `ReadableStream<string>`. Since most JavaScript methods emit a `ReadableStream<Uint8Array>`, we can use [`TextDecoderStream`](https://developer.mozilla.org/en-US/docs/Web/API/TextDecoderStream) to convert that to a string stream (wide browser support since September 2022, so you might need a polyfill). Here is an example how to use it with the [Fetch API](https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API):
@@ -129,9 +148,20 @@ If you need access to not just the object property values, but also their keys,
 * `{ value: { test: "value1" }, path: ["results", 0] }`
 * `{ value: "value2", path: ["results", 1] }`
 
-#### Consuming multiple objects/arrays
+#### Consuming multiple JSON values (JSONL)
+
+The [JSON Lines](https://jsonlines.org/) standard is a variation of JSON that allows multiple JSON values on the root level, separated by newlines. To consume such a stream, pass a `multi: true` option, which will make `parseJsonStream` accept input streams with zero or multiple root values:
+```typescript
+const stream = res
+	.pipeThrough(new TextDecoderStream())
+	.pipeThrough(parseJsonStream(undefined, { multi: true }));
+```
+
+By using `undefined` as the path selector will select the root values themselves rather than their properties/elements.
+
+#### Consuming multiple nested objects/arrays
 
-Sometimes you want to consume multiple objects/arrays in a JSON stream. This would be an example JSON document:
+Sometimes you want to consume multiple objects/arrays inside a single JSON document, like in this example:
 ```json
 {
 	"apples": {
@@ -199,7 +229,7 @@ The main features of json-stream-es are:
 	* [`JsonPathStreamSplitter`](#jsonpathstreamsplitter) to split a stream of JSON values into a stream of sub streams for the values under different paths
 	* [`JsonChunk` creators](#jsonchunk-creators) to create a stream of `JsonChunk`s by hand
 * Provide convenience functions for common combinations of the above:
-	* [`stringifyJsonStream`](#stringifyjsonstream), combining `JsonSerializer` and `JsonStringifier`
+	* [`stringifyJsonStream`](#stringifyjsonstream) and [`stringifyMultiJsonStream`](#stringifymultijsonstream), combining `JsonSerializer` and `JsonStringifier`
 	* [`parseJsonStream`](#parsejsonstream) and [`parseJsonStreamWithPaths`](#parsejsonstreamwithpaths), combining `JsonParser`, `JsonPathDetector`, `JsonPathSelector` and `JsonDeserializer`
 	* [`parseNestedJsonStream`](#parsenestedjsonstream) and [`parseNestedJsonStreamWithPaths`](#parsenestedjsonstreamwithpaths), combining `JsonParser`, `JsonPathDetector`, `JsonPathSelector`, `JsonPathStreamSplitter` and `JsonDeserializer`.
 
@@ -302,31 +332,37 @@ stream.pipeThrough(parseJsonStream((path) => (
 
 A convenience function to serialize a JSON value into a stringified JSON stream. Under the hood, it creates a [`JsonSerializer`](#jsonserializer) and pipes it through a [`JsonStringifier`](#jsonstringifier). See those for details.
 
+#### `stringifyMultiJsonStream`
+
+`stringifyMultiJsonStream(space?: string | number): TransformStream<SerializableJsonValue, string>`
+
+Returns a transform stream that accepts zero or more serializable JSON values and emits a stringified JSON stream. Can be used to create a JSON stream that contains multiple values on the root level. Under the hood, it creates a transformer chain of a [`JsonSerializer`](#jsonserializer) and a [`JsonStringifier`](#jsonstringifier). See those for details.
+
 #### `parseJsonStream`
 
-`parseJsonStream(selector: JsonPathSelectorExpression): TransformStream<string, JsonValue>`
+`parseJsonStream(selector: JsonPathSelectorExpression | undefined, options?: { multi?: boolean }): TransformStream<string, JsonValue>`
 
-A convenience function to parse a stringified JSON stream, select certain arrays and/or objects from it and stream their values/elements. `selector` needs to be a [JSON path selector](#json-path-selector) that selects one or more objects/values whose values/elements should be streamed.
+A convenience function to parse a stringified JSON stream, select certain arrays and/or objects from it and stream their values/elements. `selector` needs to be a [JSON path selector](#json-path-selector) that selects one or more objects/values whose values/elements should be streamed. If `multi` is true, no error will be thrown if the input contains zero or more than one JSON values on the root level. For multi streams, `selector` can undefined to select the root values themselves.
 
 Under the hood, creates a transformer chain of a [`JsonParser`](#jsonparser), [`JsonPathDetector`](#jsonpathdetector), [`JsonPathSelector`](#jsonpathselector) and [`JsonDeserializer`](#jsondeserializer), see those for details.
 
 #### `parseJsonStreamWithPaths`
 
-`parseJsonStreamWithPaths(selector: JsonPathSelectorExpression): TransformStream<string, { value: JsonValue; path: Array<string | number> }>`
+`parseJsonStreamWithPaths(selector: JsonPathSelectorExpression | undefined, options?: { multi?: boolean }): TransformStream<string, { value: JsonValue; path: Array<string | number> }>`
 
 Like [`parseJsonStream`](#parsejsonstream), but emits a stream of `{ value: JsonValue; path: Array<string | number> }` instead, where `path` is the path of object property keys and array element indexes of each value. This allows you to to access the property keys when streaming a JSON object.
 
 #### `parseNestedJsonStream`
 
-`parseNestedJsonStream(selector: JsonPathSelectorExpression): TransformStream<string, ReadableStream<JsonValue> & { path: JsonPath }>`
+`parseNestedJsonStream(selector: JsonPathSelectorExpression | undefined, options?: { multi?: boolean }): TransformStream<string, ReadableStream<JsonValue> & { path: JsonPath }>`
 
-A convenience function to parse a stringified JSON stream, select certain arrays and/or objects emit a nested stream for each of them emitting their values/elements. `selector` needs to be a [JSON path selector](#json-path-selector) that selects one or more objects/values whose values/elements should be streamed.
+A convenience function to parse a stringified JSON stream, select certain arrays and/or objects emit a nested stream for each of them emitting their values/elements. `selector` needs to be a [JSON path selector](#json-path-selector) that selects one or more objects/values whose values/elements should be streamed. If `multi` is true, no error will be thrown if the input contains zero or more than one JSON values on the root level. For multi streams, `selector` can undefined to select the root values themselves.
 
 Under the hood, creates a transformer chain of a [`JsonParser`](#jsonparser), [`JsonPathDetector`](#jsonpathdetector), [`JsonPathSelector`](#jsonpathselector) and [`JsonPathStreamSplitter`](#jsonpathstreamsplitter), and then pipes each sub stream through [`JsonDeserializer`](#jsondeserializer).
 
 #### `parseNestedJsonStreamWithPaths`
 
-`parseNestedJsonStreamWithPaths(selector: JsonPathSelectorExpression): TransformStream<string, ReadableStream<{ value: JsonValue; path: Array<string | number> }> & { path: Array<string, number> }>`
+`parseNestedJsonStreamWithPaths(selector: JsonPathSelectorExpression | undefined, options?: { multi?: boolean }): TransformStream<string, ReadableStream<{ value: JsonValue; path: Array<string | number> }> & { path: Array<string, number> }>`
 
 Like [`parseNestedJsonStream`](#parsenestedjsonstream), but the nested streams emit `{ value: JsonValue; path: Array<string | number> }` instead, where `path` is the path of object property keys and array element indexes of each value. In the sub streams, the paths have the path prefix of their containing streamed object/array removed.
 
@@ -336,11 +372,11 @@ Like [`parseNestedJsonStream`](#parsenestedjsonstream), but the nested streams e
 
 A `TransformStream<string, JsonChunk>` that parses the incoming stringified JSON stream and emits [`JsonChunk` objects](#jsonchunk-objects) for the different tokens that the JSON document is made of.
 
-Construct one using `new JsonParser()` and use it by calling [`.pipeThrough()`](https://developer.mozilla.org/en-US/docs/Web/API/ReadableStream/pipeThrough) on a `ReadableStream<string>`.
+Construct one using `new JsonParser(options?: { multi?: boolean })` and use it by calling [`.pipeThrough()`](https://developer.mozilla.org/en-US/docs/Web/API/ReadableStream/pipeThrough) on a `ReadableStream<string>`. If `multi` is true, accepts zero or multiple JSON values on the root level, otherwise an exception is thrown if the data contains zero or more than one values.
 
 Pass the output on to [`JsonPathDetector`](#jsonpathdetector), [`JsonPathSelector`](#jsonpathselector) and [`JsonDeserializer`](#jsondeserializer) to consume a JSON stream.
 
-The input stream is expected to contain one valid JSON document. If the document is invalid or the input stream contains zero or multiple documents, the stream aborts with an error. This also means that you can rely on the order of the emitted `JsonChunk` objects to be valid (for example, when a `JsonChunkType.STRING_CHUNK` object is emitted, you can be sure that it was preceded b a `JsonChunkType.STRING_START` object).
+The input stream is expected to contain valid JSON documents. If the input is not valid JSON (or if the input contains zero or more than one JSON documents and `multi` is not true), the stream aborts with an error. This also means that you can rely on the order of the emitted `JsonChunk` objects to be valid (for example, when a `JsonChunkType.STRING_CHUNK` object is emitted, you can be sure that it was preceded b a `JsonChunkType.STRING_START` object).
 
 #### `JsonStringifier`
 
@@ -382,6 +418,8 @@ The `SerializableJsonValue` input chunks can be any valid JSON values, that is `
 
 As the `space` constructor argument, you can specify a number of indentation spaces or an indentation string, equivalent to the [space](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/JSON/stringify#space) parameter of `JSON.stringify()`. This will cause `WHITESPACE` chunks to be emitted in the appropriate places.
 
+Multiple JSON values on the root level are always separated by a newline (`\n`). This means that when no `space` is defined, the output produced by `JsonSerializer` piped through `JsonStringifier` fulfills the [JSON Lines](https://jsonlines.org/) (JSONL) standard.
+
 ##### Differences to `JSON.stringify()`
 
 `JsonSerializer` aims to mimic the behaviour of [`JSON.stringify()`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/JSON/stringify), with the following differences.

package.json

@@ -1,11 +1,7 @@
 {
 	"name": "json-stream-es",
-	"description": "A streaming JSON parser/stringifier using web streams.",
-	"tags": [
-		"json",
-		"stream",
-		"webstreams"
-	],
+	"description": "A streaming JSON/JSONL parser/stringifier using web streams.",
+	"keywords": ["json", "jsonl", "stream", "webstreams"],
 	"version": "1.0.0",
 	"author": "Candid Dauth <cdauth@cdauth.eu>",
 	"repository": {

src/__tests__/convenience.test.ts

@@ -34,6 +34,30 @@ test("parseJsonStream", async () => {
 	expect(await streamToArray(stream)).toEqual(["apple1", "apple2", "cherry1", "cherry2"]);
 });
 
+test("parseJsonStream fails for multiple documents", async () => {
+	const stream = stringToStream(`"value1"\n"value2"`)
+		.pipeThrough(parseJsonStream([]));
+	await expect(async () => await streamToArray(stream)).rejects.toThrowError("Unexpected character");
+});
+
+test("parseJsonStream multi", async () => {
+	const documents = [
+		"value1",
+		2,
+		{ test3: "value3" },
+		["value4"]
+	];
+	const stream = stringToStream(documents.map((v) => JSON.stringify(v)).join("\n"))
+		.pipeThrough(parseJsonStream(undefined, { multi: true }));
+	expect(await streamToArray(stream)).toEqual(documents);
+});
+
+test("parseJsonStream multi no values", async () => {
+	const stream = stringToStream("")
+		.pipeThrough(parseJsonStream(undefined, { multi: true }));
+	expect(await streamToArray(stream)).toEqual([]);
+});
+
 test("parseJsonStreamWithPaths", async () => {
 	const stream = stringToStream(JSON.stringify(testObject))
 		.pipeThrough(parseJsonStreamWithPaths([["apples", "cherries"], "results"]));

src/convenience.ts

@@ -1,6 +1,6 @@
 import { JsonDeserializer, type JsonValueAndPath } from "./json-deserializer";
-import { JsonParser } from "./json-parser";
-import { serializeJsonValue, type SerializableJsonValue } from "./json-serializer";
+import { JsonParser, type JsonParserOptions } from "./json-parser";
+import { JsonSerializer, serializeJsonValue, type SerializableJsonValue } from "./json-serializer";
 import { JsonStringifier } from "./json-stringifier";
 import { JsonPathDetector, type JsonPath } from "./json-path-detector";
 import { JsonPathSelector, matchesJsonPathSelector, type JsonPathSelectorExpression } from "./json-path-selector";
@@ -12,40 +12,59 @@ export function stringifyJsonStream(value: SerializableJsonValue, space?: string
 	return serializeJsonValue(value, space).pipeThrough(new JsonStringifier());
 }
 
+export function stringifyMultiJsonStream(space?: string | number): TransformStream<SerializableJsonValue, string> {
+	return new PipeableTransformStream((readable) => {
+		return readable
+			.pipeThrough(new JsonSerializer(space))
+			.pipeThrough(new JsonStringifier());
+	});
+}
+
 class ValueExtractor extends AbstractTransformStream<JsonValueAndPath, JsonValue> {
 	protected override transform(chunk: JsonValueAndPath, controller: TransformStreamDefaultController<JsonValue>) {
 		controller.enqueue(chunk.value);
 	}
 }
 
 export function parseJsonStreamWithPaths(
-	selector: JsonPathSelectorExpression
+	selector: JsonPathSelectorExpression | undefined,
+	options?: JsonParserOptions
 ): TransformStream<string, JsonValueAndPath> {
 	return new PipeableTransformStream((readable) => {
-		return readable
-			.pipeThrough(new JsonParser())
-			.pipeThrough(new JsonPathDetector())
-			.pipeThrough(new JsonPathSelector((path) => path.length > 0 && matchesJsonPathSelector(path.slice(0, -1), selector)))
-			.pipeThrough(new JsonDeserializer());
+		let result = readable
+			.pipeThrough(new JsonParser(options))
+			.pipeThrough(new JsonPathDetector());
+		if (selector) {
+			result = result.pipeThrough(new JsonPathSelector((path) => path.length > 0 && matchesJsonPathSelector(path.slice(0, -1), selector)));
+		}
+		return result.pipeThrough(new JsonDeserializer());
 	});
 }
 
 export function parseJsonStream(
-	selector: JsonPathSelectorExpression
+	selector: JsonPathSelectorExpression | undefined,
+	options?: JsonParserOptions
 ): TransformStream<string, JsonValue> {
 	return new PipeableTransformStream((readable) => {
-		return readable
-			.pipeThrough(parseJsonStreamWithPaths(selector))
+		let result = readable.pipeThrough(new JsonParser(options))
+		if (selector) {
+			result = result
+				.pipeThrough(new JsonPathDetector())
+				.pipeThrough(new JsonPathSelector((path) => path.length > 0 && matchesJsonPathSelector(path.slice(0, -1), selector)));
+		}
+		return result
+			.pipeThrough(new JsonDeserializer())
 			.pipeThrough(new ValueExtractor());
 	});
 }
 
 export function parseNestedJsonStreamWithPaths(
-	selector: JsonPathSelectorExpression
+	selector: JsonPathSelectorExpression,
+	options?: JsonParserOptions
 ): TransformStream<string, ReadableStream<JsonValueAndPath> & { path: JsonPath }> {
 	return new PipeableTransformStream((readable) => {
 		return readable
-			.pipeThrough(new JsonParser())
+			.pipeThrough(new JsonParser(options))
 			.pipeThrough(new JsonPathDetector())
 			.pipeThrough(new JsonPathSelector(selector))
 			.pipeThrough(new JsonPathStreamSplitter())
@@ -63,11 +82,12 @@ export function parseNestedJsonStreamWithPaths(
 }
 
 export function parseNestedJsonStream(
-	selector: JsonPathSelectorExpression
+	selector: JsonPathSelectorExpression,
+	options?: JsonParserOptions
 ): TransformStream<string, ReadableStream<JsonValue> & { path: JsonPath }> {
 	return new PipeableTransformStream((readable) => {
 		return readable
-			.pipeThrough(parseNestedJsonStreamWithPaths(selector))
+			.pipeThrough(parseNestedJsonStreamWithPaths(selector, options))
 			.pipeThrough(new TransformStream({
 				transform: (chunk, controller) => {
 					controller.enqueue(Object.assign(

src/json-deserializer.ts

@@ -39,6 +39,10 @@ export type JsonValueAndPath = JsonValueAndOptionalPath<JsonChunkWithPath>;
 export class JsonDeserializer<C extends JsonChunk & { path?: JsonPath } = JsonChunkWithPath> extends AbstractTransformStream<C, JsonValueAndOptionalPath<C>> {
 	protected state: State<C> = { type: StateType.ROOT, value: undefined, path: [] };
 
+	constructor() {
+		super();
+	}
+
 	protected handleValueEnd(controller: TransformStreamDefaultController<JsonValueAndOptionalPath<C>>): void {
 		if (this.state.type === StateType.ROOT) {
 			if (this.state.value !== undefined) {