Is it necessary to specify a specific requestQueue if RequestQueueV2 was previously passed to the crawler?

[Cit1zeN4]

I'm reading a document about parallel use of multiple crawlers and I can't understand why I need to specify a specific requestQueue when executing enqueueLinks. Won't the RequestQueueV2 that was previously passed to the crawler be used?

code from the guide:

router.addHandler('CATEGORY', async ({ page, enqueueLinks, request, log }) => {
    log.debug(`Enqueueing pagination for: ${request.url}`);
    // We are now on a category page. We can use this to paginate through and enqueue all products,
    // as well as any subsequent pages we find

    await page.waitForSelector('.product-item > a');
    await enqueueLinks({
        selector: '.product-item > a',
        label: 'DETAIL',
        // highlight-next-line
        requestQueue: await getOrInitQueue(), // <= Is this really necessary?
    });

    const nextButton = await page.$('a.pagination__next');
    if (nextButton) {
        await enqueueLinks({
            selector: 'a.pagination__next',
            label: 'CATEGORY',
        });
    }
});

https://crawlee.dev/js/docs/guides/parallel-scraping
https://github.com/apify/crawlee/blob/master/docs/guides/parallel-scraping/adapted-routes.mjs

[janbuchar]

Hi there! Judging from a quick glance at the guide, it probably isn't necessary to pass the requestQueue to enqueueLinks. The reason it's mentioned in the guide is that this code is shown before writing the "consumer" code, so I guess that the author wanted to make it explicit that we're using a shared named queue here. Is that correct @vladfrangu? 🙂

[vladfrangu]

Got to answer this before me - yes! The guide [which could use some love possibly] was built to show you a split between the queue that collects what should be processed and the queue actually does the processing and can be parallelized. You totally can just use the default request queue, which to my knowledge it's now RequestQueueV2, with 0 issues! It depends on your crawler of course!

[janbuchar]

Huh, can you use the default one? If you do that on Apify, then the individual "consumers" will have their own queues and won't share anything. I believe you still need a named queue.

[vladfrangu]

Yes, you still need to make a named queue and set it at the crawler leve, but I'm semi certain that after you've done that, it propagates down to the utility functions, right? If not, that sounds a bit like a bug 😅

[janbuchar]

Yeah, just setting it at the crawler level is enough.