I disabled parallel executor, and my game's performance improved

[shuoli84]

I am developing a game for iOS with bevy. When I test the code in my iPhone 12 pro max, the game works but fresh rate is unstable, cpu usage is also relative high (around 50%). After some guessing and trouble shooting, I think the parallel executor introduced some significant overhead.
I did a quick and dirty verification, by modify Stage::parallel() return the single_threaded(). So all ecs systems are using single threaded scheduler, and my game performance improved. Following data is frame time distribution for 5s. It seems parallel scheduler has much better min frame time and much worse max frame time.
I instrumented the code, there are cpu spikes in parallel scheduler, which calls out some stack like "waker.clone()" / "async_tasks::header::Header::register".
Is this a known issue?

iOS
Single threaded (hacked)  cpu usage ~40%.
min:0ms p50:16ms p90:33ms max:57ms
min:3ms p50:16ms p90:17ms max:31ms
min:13ms p50:16ms p90:17ms max:19ms
min:13ms p50:16ms p90:17ms max:19ms
min:13ms p50:16ms p90:17ms max:20ms
min:14ms p50:16ms p90:17ms max:20ms
min:14ms p50:16ms p90:17ms max:19ms
min:13ms p50:16ms p90:17ms max:19ms
min:0ms p50:16ms p90:17ms max:19ms
min:14ms p50:16ms p90:17ms max:19ms

Default impl (without the hack)  cpu usage ~50%
min:0ms p50:16ms p90:20ms max:75ms
min:0ms p50:16ms p90:18ms max:49ms
min:3ms p50:16ms p90:17ms max:26ms
min:4ms p50:16ms p90:17ms max:25ms
min:4ms p50:16ms p90:17ms max:25ms
min:3ms p50:16ms p90:17ms max:28ms
min:2ms p50:16ms p90:17ms max:32ms
min:1ms p50:16ms p90:17ms max:36ms
min:1ms p50:16ms p90:17ms max:35ms
min:1ms p50:16ms p90:17ms max:36ms
min:0ms p50:16ms p90:17ms max:22ms
min:5ms p50:16ms p90:17ms max:24ms

[shuoli84]

The data in above is measured wrongly, it is time of delta(). Which equals 18ms for most of time. Today I tried to measure frame's First -> Last stage, and following is new data. Kinda same conclusion, in my case(a simple 2d game), single thread is way better than parallel scheduler.

iOS
Single threaded
min:96us p50:617us p90:742us max:11181us
min:128us p50:599us p90:727us max:1999us
min:341us p50:604us p90:748us max:1251us
min:102us p50:605us p90:754us max:7134us
min:399us p50:583us p90:756us max:2072us
min:323us p50:575us p90:692us max:900us
min:396us p50:555us p90:677us max:809us
min:142us p50:578us p90:732us max:949us
min:292us p50:562us p90:698us max:1127us
min:183us p50:595us p90:731us max:1030us
min:398us p50:590us p90:743us max:1023us
min:94us p50:588us p90:737us max:999us

Default
min:200us p50:1113us p90:1499us max:7393us
min:227us p50:1134us p90:1457us max:6220us
min:221us p50:1114us p90:1355us max:16982us
min:204us p50:1103us p90:1314us max:4554us
min:367us p50:1099us p90:1298us max:3098us
min:224us p50:1098us p90:1305us max:6146us
min:217us p50:1147us p90:1431us max:8181us
min:200us p50:1163us p90:1442us max:8386us

[hymm]

I would say that it's not super surprising that single threaded is faster than multithreaded. The multithreaded executor has some overhead associated with it and it'll depend on how many systems can actually execute in parallel whether it'll be worth it or not. There is a small improvement to the overhead that has been merged into main that might help some, but probably doesn't completely close the gap for your case #4919.

The implementation of the task scope used in the system executor has a tight loop in it, which is what I would expect to be causing the cpu spikes. Some discussion here about it #4718.