[Apple Silicon] Manually recreating swapchains creates weird artifacts

If I e.g. recreate the swapchain every frame it flickers (in exclusive fullscreen!).

I don't do this, but there was a case where I recreated the swapchain manually and it caused the screen to turn black (weird viewport).

I cannot find anything in the vulkan spec that doesn't allow me to recreate the swapchain as I want.

(Running from separate thread)