Generic Error When Repeatedly Calling Submit-PnPSearchQuery

[stuck-on-starting]

I have a PowerShell script that uses the Submit-PnPSearchQuery cmdlet to find all documents in a SPO document library that meet certain filter criteria.

Some libraries in our environment are very large. So I have a loop that repeatedly calls Submit-PnPSearchQuery to retrieve documents in batches of 500 (the maximum) until all results have been retrieved.

The issue that I'm having is that the loop never completes successfully on these large libraries because I eventually receive a generic error, 'Search has encountered a problem that prevents results from being returned'. This usually occurs around 50,000 - 70,000 results.

Has anyone ever encountered this before or know how to avoid this error?