SPO throttling issue unable to track in PnP PowerShell script

[patleyrita]

Hi All ,

I am trying to create large number of sites(995) and document libraries(600K) in it, which is taking number of days(22) time to complete given volume of site collection and document libraries using PnP PS. In order to reduce time I have written Parallel execution of PowerShell scripts by passing CSV paths to below code

$scriptPaths = @(
"Merged1Run3PS.csv",
"Merged2Run3PS.csv"

)

Start jobs for each script in parallel

$scriptPaths | ForEach-Object {
write-host $_
Start-Process -FilePath "powershell.exe" -ArgumentList "-File C:\ABC\PNPOnineDcLibRn.ps1 $" -RedirectStandardOutput C:\ABC\CSVToRun3NDParalleSPO$.txt
}

which start all number of PS execution parallelly , however i can see after couple of hrs only one PS is writing document libraries and other are getting stopes with below error .

Error: at PnP.PowerShell.Commands.Base.PnPConnectedCmdlet.ProcessRecord()
at System.Management.Automation.CommandProcessor.ProcessRecord()

It seams generic error how can i track the error to get the issue details .

[AndersRask]

Read my answer here regarding throttling and especially Rate Limits.

Running a huge amount of requests in parallel could get your AppId banned for not obeying 429 throttling, and it is of no use, as you only have so many RU to use (based on licenses) per minute and per day.

[patleyrita]

Thanks AndersRask for your valuable inputs. I will check the details mentioned in the post .