How to read large files in nodejs (v16)

[gowy222]

const zipReader = new zip.ZipReader(readable)

ERR:
reader = new BlobReader(await new Response(reader.readable).blob()); //put all data into memory? if file over 2GB..
=> ReferenceError: Response is not defined

---

Response is Web API? is there any way work with big file(>2GB) use fs.createReadStream(local_path)... low memory! ;
and noted NO Readable.toWeb() in pure nodejs tho..

[gildas-lormeau]

If you cannot use web streams natively, you can write a class extending the Reader class of zip.js and use it to read data.

You have 2 methods to implement (you can omit the constructor):

• The method init should initialize this.size with the size in bytes returned by filehandle#stat for example, see https://nodejs.org/api/fs.html#filehandlestatoptions.
• The method readUint8Array should rely on fs.createReadStream and use the options start, end in order to read sections of the file as Uint8Array instances, see https://nodejs.org/api/all.html#fs_fs_createreadstream_path_options.

There is also an example of such a custom Reader class in the documentation or below.

zip.js/tests/all/test-custom-io.js

Lines 6 to 25 in 151b598

	class BinaryStringReader extends zip.Reader {
	constructor(binaryString) {
	super();
	this.binaryString = binaryString;
	}
	init() {
	super.init();
	this.size = this.binaryString.length;
	}
	readUint8Array(offset, length) {
	const result = new Uint8Array(length);
	for (let indexCharacter = 0; indexCharacter < length; indexCharacter++) {
	result[indexCharacter] = this.binaryString.charCodeAt(indexCharacter + offset) & 0xFF;
	}
	return result;
	}
	}

[TKdevlop]

Hi sorry, I am new to this lib and still not sure how to fix the above issue
for example to fix ReferenceError: TransformStream is not defined

need to add this

import { TransformStream } from "node:stream/web";

//@ts-ignore
globalThis.TransformStream = TransformStream;

can you suggest a fix for the above would be greatly appreciated?

[gildas-lormeau]

Didn't the code you added solve the problem? The alternative is to update Node.js.