TerminatingError(Get-PnPListItem)The remote server returned an error: (503)

[IfEyeOnlyKnew]

SharePoint Online / PowerShell 7.3.6
I am using PnPListItem to get around 754,000 files to look for path count greater than or equal to 400. At the 712,000 item query I get this error message. I am not sure why it says it is null? 503 error? I should have about 40,000 more files to query. The error does not stop querying the other libraries.

I am using -PageSize 1000 -ScriptBlock, so kind of confused what this 503 means.

TerminatingError(Get-PnPListItem): "Exception calling "ExecuteQuery" with "0" argument(s): "The remote server returned an error: (503)"

`Set-Location C:\Scripts\SPOCharacterCount
Clear-Host

$filerandom = get-random
Start-Transcript -Path C:\Scripts\SPOScript\Transcript\transcript_$filerandom.txt -NoClobber

#Specify SharePoint Online Site URL or User's OneDrive Site URL
$SiteURL = "https://tenant.sharepoint.com/teams/XYZ"

Foreach ($Site in $SiteURL)
{
$AllFiles = @() # Result array to keep all file details

#Connect to SharePoint Online site
$password = ConvertTo-SecureString -String "Secret" -AsPlainText -Force
$Connection= Connect-PnPOnline -ClientId 00000008-0003-000f-0003-000000001 -Url $Site -Tenant tenant.onmicrosoft.com -CertificatePath 'C:\scripts\PnPCertificate\certificate.pfx' -CertificatePassword $password

#Get all document libraries - Exclude Hidden Libraries
$DocumentLibraries = Get-PnPList | Where-Object {$_.BaseType -eq "DocumentLibrary" -and $_.Hidden -eq $false -and $_.Title -ne "Style Library" -and $_.Title -ne "Site Assets" -and $_.Title -ne "Site Pages" -and $_.Title -ne "Site Collection Images" -and $_.Title -ne "Site Collection Documents"} #Or $_.BaseType -eq "DocumentLibrary"

#Get Document Libraries Name, Default URL and Number of$_s
$DocumentLibraries | Select-Object Title, DefaultViewURL, ItemCount | Format-Table -AutoSize

Foreach ($Document in $DocumentLibraries)
{
    #Provide name of your List/Document library
    $DocumentLibrary = $Document.Title

    #Retrieve all Files from the document library - | Where-Object {$_["FileLeafRef"] -like "*.*"}
    Get-PnPListItem -List $DocumentLibrary -PageSize 1000  -ScriptBlock { Param($_s) $_s.Context.ExecuteQuery() } | Where-Object {$_["FileLeafRef"] -like "*.*"} | ForEach-Object {
    
        #Find files that FileRef is greater than 400 characters add to list
        Write-Host "Checking file length:"($_.FieldValues.FileRef).length " <Filename> " $_.FieldValues["FileRef"]
        $GetCount = ($_.FieldValues.FileRef).length
        
        #Add file details to Result array
        If ($GetCount -ge 400)
        {
            Write-host "File is greater than or equal to 400" ($_.FieldValues.FileRef).length " <> " $_.FieldValues["FileRef"] -ForegroundColor Red
                   
                $AllFiles += New-Object PSObject -property $([ordered]@{ 
                FileName  = $_.FieldValues["FileLeafRef"]
                RelativeURL = $_.FieldValues["FileRef"]   
                RelativeUrlCharCount  = ($_.FieldValues["FileRef"]).Length
                FileID = $_.FieldValues["UniqueId"]
                FileType = $_.FieldValues["File_x0020_Type"]
                CreatedByEmail = $_.FieldValues["Author"].Email
                CreatedTime   = $_.FieldValues["Created"]
                LastModifiedTime   = $_.FieldValues["Modified"]
                ModifiedByEmail  = $_.FieldValues["Editor"].Email
                FileSize_KB = [Math]::Round(($_.FieldValues["File_x0020_Size"]/1024), 2) #File size in KB
            })  
        }      
    } 
        $Filename = $site.url -replace("https://tenant.sharepoint.com/teams/","") -replace("https://tenant.sharepoint.com/sites/","") -replace("/","_") -replace("/","") 
        [IO.Path]::GetinvalidFileNameChars() | ForEach-Object {$Filename = $Filename.Replace($_," ")}
        $CSVPath = "C:\Scripts\SPOCharacterCount\Report\"+$Filename.trim() + "_" +"_Rpt"+".csv"
    
        If ($null -eq $AllFiles)
        {$AllFiles | Export-CSV $CSVPath -NoTypeInformation -Encoding UTF8
        Write-Host "SharePoint files report exported successfully" –f Green}  
        Else{
        Write-Host "moving on...."}      
}

}
Disconnect-PnPOnline
Stop-Transcript

`

[IfEyeOnlyKnew]

Here is a random 503 error, @ CIM8186.JPG there are actually over another 100 files it does not query, but the 503 appears and stops querying the library, then moves on the next library.

Checking file length: 91 /teams/SiteCollection_001/GCG-Corporate_00/00 Meeting 2012 at burg/CIMG8184.JPG
Checking file length: 91 /teams/SiteCollection_001/GCG-Corporate_00/00 Meeting 2012 at burg/CIMG8186.JPG
PS>TerminatingError(Get-PnPListItem): "Exception calling "ExecuteQuery" with "0" argument(s): "The remote server returned an error: (503) Service Unavailable.""

Get-PnPListItem: C:\Scripts\SPOCharacterCount\GetCharacterCount_New.ps1:30:9
Line |
30 | Get-PnPListItem -List $DocumentLibrary -PageSize 1000 -Scrip …
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| Exception calling "ExecuteQuery" with "0" argument(s): "The remote server returned an error: (503) Service Unavailable."

Get-PnPListItem: C:\Scripts\SPOCharacterCount\GetCharacterCount_New.ps1:30:9
Line |
30 | Get-PnPListItem -List $DocumentLibrary -PageSize 1000 -Scrip …
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| Exception calling "ExecuteQuery" with "0" argument(s): "The remote server returned an error: (503) Service Unavailable."

moving on....
Checking file length: 75 /teams/SiteCollection_001/My Team template/lagen.zip
Checking file length: 78 /teams/SiteCollection_001/My Team template/Guide_Final_DE.doc

[AndersRask]

503 means that you are getting hard throttled.
Meaning that you have ignored soft throttling (429 error code) and are currently blocked.

I recommend read up on throttling, because when you attack the endpoints hard enough to get 503 (it takes an effort!) you will also affect other services on your tenant such as Onedrive sync, backup, other tasks using the endpoints.

Your tenant has an amount of Resource Units available per minute/day. How many depends on your number of licenses. Every request takes between 1 and 5 RU.

Once the request per minute are 80% used you will begin to receive rate limit headers telling you to slow down sending requests. Once you have used 90% of RU within a minute you will receive 429 responses telling you to pause (typically 120 seconds) before sending another request. Ignoring this, or running too many parallel requests will get you hard throttled (503 server too busy). Keeping ignoring that too, will get the application I'd you are using banned on the tenant.

Also note that the pnp.powershell framework has a built in retry handler and will handle soft throttling by delaying requests. Rate Limit support is coming, but isn't available yet. This is not foolsafe, so you can still become throttled, especially if you run many threads or other processes are using up your RU (they are shared across tenant).

Without having looked at your code, I suggest spreading the load of requests over time, so you do not hammer the endpoints quite so hard.

Read this article for more info on avoiding getting throttled
https://learn.microsoft.com/en-us/sharepoint/dev/general-development/how-to-avoid-getting-throttled-or-blocked-in-sharepoint-online

[IfEyeOnlyKnew]

I see this post [https://github.com//discussions/62] here as well about 429 and powershell is already trying to resolve the issue.

Would this work where the 429 happens and the catch returns back to the function? I ask, because the $retryAfter does not go away.
`
Function Try-This{
Try {
Do something
}

catch [System.Net.WebException] {
$response = $_.Exception.Response
if ($response.StatusCode -eq 429 -or $response.StatusCode -eq 503) {
$retryAfter = $response.Headers['Retry-After']
Start-Sleep -Seconds $retryAfter
# Retry your code here
Try-This
}
}
`

[AndersRask]

Retry is already implemented in pnp.powershell so you don't have to test for a 429 response

As I wrote above, rate limit is on the drawing board, and is already implemented in core (and code can basically be implemented in retry module exactly as in core)

When we get rate limit support is hard to say, as this framework is done by people in their spare time, but maybe @erwinvanhunen or @gautamdsheth can give us a hint on the time frame?

[IfEyeOnlyKnew]

Is someone working to implement this in PnP PowerShell: https://github.com/pnp/pnpframework/blob/dev/src/lib/PnP.Framework/Extensions/ClientContextExtensions.cs#L178