Sharepoint Throttling

[ayearsley1984]

Hi

We are using the Add-PNPFile to migrate files from an on prem file server to Sharepoint sites. We've got this working well and it is migrating about 40 files a minute at peak time which is acceptable. However, every 5-10 minutes it seems to get stuck on teh Add-PNPFile step for around 5 minutes and then will start going again. I assume this is throttling but not getting any error messages back.

I know it is supposed to help if you decorate your traffic, but unsure if this is done or if it can be done with PNP Powershell?

We are using a clientid and clientsecrete for the Connect-PNPOnline step. Is there a better method to use for unattended mode running on an onprem server.

We are using version 1.10 but looking throught eh change log there doesn't seem to be anything relevant for us to need to upgrade, so holding off on that at present given the main process is working, albeit about half the rate we were expecting.

Any help greatly appreciated.

Thanks
Andrew

[AndersRask]

Sounds like you're hitting the per minute rate limit. You won't see this decorated in headers, as requests are wrapped inside pnp framework request manager. I've asked about rate limit inclusion before, and hope this is made a priority (nudge nudge) but I guess getting to .Net6 in 2.1.1 has been the priority so far 😊 Read more about rate limit below. In short it's a pool of resources that depend on your number of licenses. Introduced to protect endpoints against too many requests. https://learn.microsoft.com/en-us/sharepoint/dev/general-development/how-to-avoid-getting-throttled-or-blocked-in-sharepoint-online søn. 14. maj 2023 14.59 skrev ayearsley1984 ***@***.***>:

…

Hi We are using the Add-PNPFile to migrate files from an on prem file server to Sharepoint sites. We've got this working well and it is migrating about 40 files a minute at peak time which is acceptable. However, every 5-10 minutes it seems to get stuck on teh Add-PNPFile step for around 5 minutes and then will start going again. I assume this is throttling but not getting any error messages back. I know it is supposed to help if you decorate your traffic, but unsure if this is done or if it can be done with PNP Powershell? We are using a clientid and clientsecrete for the Connect-PNPOnline step. Is there a better method to use for unattended mode running on an onprem server. We are using version 1.10 but looking throught eh change log there doesn't seem to be anything relevant for us to need to upgrade, so holding off on that at present given the main process is working, albeit about half the rate we were expecting. Any help greatly appreciated. Thanks Andrew — Reply to this email directly, view it on GitHub <#3109>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAMTDTRUQWNZGZNZYOBWCHDXGDJKBANCNFSM6AAAAAAYBEXLNQ> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[ayearsley1984]

Thanks for your response. I've read that Microsoft article a few times now.

By decorating I mean decorating the request being sent to Sharepoint with appid, appname and user-agent as recommended, rather than the response coming back?

in terms of hitting the rate time, for our size of organisation this is 1200 per minute, we are only processing 40 files a minute so that would suggest each file is using 30 credits from the limit, does that sound right. Even reducing down the number to 30 files a minute doesn't seem to help

[AndersRask]

Decorating with user agent is mainly doe ISVs that create a product, so that the ULS log will show this, so it has no effect on throttling as such. How do you connect to SharePoint though? Throttling for a user happens a lot faster than if you create an app registration and connect with that. In general moving many files using pnp is a very slow process. If you have a lot of files and not much Metadata, you could consider the migration frameworks dedicated for this, that most migration tools also use. They will move the files to azure in a queue and handle upload. søn. 14. maj 2023 23.31 skrev ayearsley1984 ***@***.***>:

…

Thanks for your response. I've read that Microsoft article a few times now. By decorating I mean decorating the request being sent to Sharepoint with appid, appname and user-agent as recommended, rather than the response coming back? — Reply to this email directly, view it on GitHub <#3109 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAMTDTSCHICRRX5P4OOPWNDXGFFKFANCNFSM6AAAAAAYBEXLNQ> . You are receiving this because you commented.Message ID: ***@***.***>

[ayearsley1984]

We are using an app registration with clientid and clientsecret to connect.

We have lots of files (6 million+) and lots of metadata (probably 10 fields for each file) so from the above it doesn't sound liek the migration framework would work!?

[AndersRask]

That's quite a lot of files. The problem with pnp framework as you noticed is that you do now get rate limit reply, as this is not implemented yet, so you probably will run into being throttled due to depleted rate limit quota, without even knowing. For large migration style tasks like yours, I would go with a migration tool. Personally I would recommend a dedicated migration tool. My preferred tool after working with many migration tools are Migration Flow for SharePoint https://www.migratedms.com/product tir. 16. maj 2023 10.13 skrev ayearsley1984 ***@***.***>:

…

We are using an app registration with clientid and clientsecret to connect. We have lots of files (6 million+) and lots of metadata (probably 10 fields for each file) so from the above it doesn't sound liek the migration framework would work!? — Reply to this email directly, view it on GitHub <#3109 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAMTDTVTFGYAJMEAGQOH2A3XGMZJJANCNFSM6AAAAAAYBEXLNQ> . You are receiving this because you commented.Message ID: ***@***.***>