Server Crash (Server CPU 100%)

[bekirbostanci]

Hi
We tried to organize for 500 people for an event. But after 100 people entered the site during the event, the server could not respond to users. Server CPU working rate 100%. We had lots of pictures, videos and a high polygon building model in the room. We were getting the building model on https://poly.google.com/. Videos added via youtube links.

Questions

1. Do the images and videos I upload outside the server negatively affect the server performance?
2. High polygons buildings cause the server to crash?
3. Are the 3d objects created by the users a problem for the server?

Server Property
Server : N. Virginia

Server Type: c5.24 xLarge ( 
c5.24xlarge 96 vCPU 192 GIB

CDN : Cloudflare don’t


What should I do to avoid facing this problem again?

[robinkwilson]

To know what went wrong, I'd need to know:

1. What did the "server not responding to users" look like?
2. How spread out were your guests? - we recommend 25 or less per room to help everyone's performance
3. If you're using Cloudflare WITH videos, did you check your email to see if they suddenly stopped supporting your event in the middle of it? - we do not recommend using Cloudflare workers if you have ANY videos in your scenes, we're in the process of replacing Cloudflare: Content CDN Cloudflare to Stackpath hubs-cloud#130

Each of your questions do impact user performance significantly. We have recommendations for how may polygons for a scene on our documentation and recommend limiting videos to 3 at most per room. The bottleneck is everyone's internet connections here.

If someone brought in too large of an asset, I'm wondering if the room would freeze or become unresponsive for a while while the server uploads then sends that large asset to everyone connected. Or if there was already a high polygon asset, it's going to take a long time for someone to load into the room. There have been rooms with videos where I've had to wait 5 minutes for a 100+MB download before even entering a room.

The c5.24xlarge (Personal 1 server I'm guessing?) should have been able to handle that many people at a time (spread out to 25 people per room), unless the server encountered an unexpected error suddenly and crashed.

In the future, I might try Hubs Cloud Enterprise multi-server with 2 app servers x 2 streaming servers for added resiliency.
If an unexpected problem occurs with a server and it crashes, another takes it's place.

To keep costs down leading up to the next event, I'd suggest using 1 app server x 1 streaming server and a small instance size with database pausing on. Then scale up to 2 app servers x 2 streaming servers during your event.

Summarizing:

• 25 people per room
• Limit videos to 3 in a room
• Lower poly assets the better (In Spoke when you save a scene, Spoke will tell you how your polycount is in your scene: red, yellow, green)
• Ideally users bring in low polycount 3D assets
• If you have videos, remove Cloudflare workers
• Maybe try Hubs Cloud Enterprise for that server resiliency function having extra server to replace a failed one

Hope this helps!

[bekirbostanci]

Thank you for your response. Your response answered all my questions.