How to download 10'000+ images with undici

[higoka]

I have a large json file which contains urls to images which i need to download to my machine.

I parse the json file and now i have an array of 10'000+ urls of images. Now my question is how can i efficiently download every image using undici to my machine. Preferably i want to download them in parallel so its fast but also throttle the request rate so i don't get rate-limited?

[mcollina]

Where would you store your images? In any case, I would just use https://www.npmjs.com/package/fastq and with a worker to download and save them one by one. I would not process more than 100 in parallel due to file descriptors limitation on the OS, but that number could be easily increased.

[higoka]

How would i download a file? Can you show a code example, i am new to undici.

[mcollina]

The readme covers how to call an http endpoint https://github.com/nodejs/undici#quick-start.

[higoka]

Can i use a fs.createWriteStream() factory with the undici.stream() method to save the file on the disk?

[mcollina]

Definitely

[higoka]

import { stream } from 'undici'
import { createWriteStream } from 'fs'

function downloadFactory ({ statusCode, opaque }) {
  if (statusCode !== 200) {
    throw new Error(`failed to download, ${statusCode}`)
  }
  return createWriteStream(opaque.target)
}

const urls = [
  { source: 'https://www.example.com/foo.png', target: 'images/foo.png' },
  { source: 'https://www.example.com/bar.png', target: 'images/bar.png' },
  { source: 'https://www.example.com/baz.png', target: 'images/baz.png' },
  // ... many more
]

await Promise.allSettled(
  urls.map(({ source, target }) => stream(source, { opaque: { target } }, downloadFactory))
)

console.log('all images downloaded')

@mcollina I came up with this. Is this the correct way to download files?