Why is it more difficult to abort a generation in SwarmUI as opposed to Comfy backend?

[pborah88]

I've used comfy both by itself and from the swarmUI and in both cases even if I'm generating 100+ images by cranking up the batch count under queue prompt, it's very easy to stop suddenly. I just have to clear the queue and cancel the current on running. However using "batch" parameter in swarmUI only uses single GPU. In order to use both GPUs I have to edit the images parameter to a number that I want. But since I started using it a couple weeks ago, I've been finding it very difficult to abort if all of a sudden I want to make changes to the current parameters because I don't like what I see in the preview. I can click x multiple times, I can select the various interrupt options several times but it almost always takes a long time to acknowledge that request upwards of 10 minutes. And sometimes I have to reboot server because it's just stuck. Another quirk I noticed is when I'm generating more than 2 images (1/GPU), when I select "interrupt all generations", if it does work, it will only abort the first two and start the next ones in the queue, so then I end up restarting the backends. I added more system RAM since my last question, 64GB, so that I have enough headroom.

[pborah88]

Actually, please hold off on this. I enabled verbose logging after my last question, but somehow it went back to "info" under server configuration. I just took a look around and noticed that. I've reenabled it. Will keep an eye on the logs the next time I'm faced with this issue. Thank you.

[mcmonkey4eva]

So for a bit of info here:

• the current vs all sessions thing: that's so you can open multiple tabs of swarm and work independently between (or even share your swarm instance with somebody else connecting remotely). "Current session" will only interrupt generations that come from the tab you're in, whereas "All sessions" interrupts everything running on the server regardless of what tab requested it (*this will be different in the future when there's an explicit multi-user-account system). Notably "all sessions" is also useful for if you accidentally refreshed/closed the tab and want to go cancel what was running previously.
• When using local comfy self-start backends, you should see in Debug logs [Debug] ComfyUI Interrupt requested, that is swarm seeing the interrupt and sending it forward to comfy, then ComfyUI-0 on port 7821 stderr: Processing interrupted which is comfy seeing the interrupt and stopping. If the first one is delayed, either something has gone wrong on the Swarm side, or it is trapped waiting on comfy (it only runs the check each time comfy sends a status update, so eg if comfy is loading a model, it cannot be interrupted until it finishes loading. Comfy will freeze up the same way if you cancel directly in the comfy tab, so not much changes here. The only difference would be if you have two models loading in a row, it's possible it'll finish loading the first, then swarm sees the interrupt and forwards it, but comfy has already started loading the second and so that needs to finish before comfy processes the forwarded interrupt. In practice, outside of model loads, there's extremely little delay between updates usually, maybe a few seconds if doing a rather large gen).
• When doing a Swarm-API-Backend (ie a remote machine), it's a very similar process with one extra step in between (wait for swarm to see the interrupt, then forward it, then wait for remote swarm to see it, then forward it, then it hits the comfy)
• The queue handler does check for if a request was cancelled, to remove it from the queue right away. It shouldn't take more than a couple seconds for this to self-clear.
• The interrupt signal is checked for all over, it shouldn't be possible for an interrupted generation to get sent forward if it wasn't already running at the time.
• By default, if you have more gen requests than GPUs, Swarm queues 2 generations onto 1 GPU at a time (this is to guarantee that swarm's processing time and network overhead don't slow the requests down, since comfy has a linear queue built in internally). (You can control this in the backends tab, it's the "OverQueue" setting on a comfy self start backend). This might affect interrupt handling a bit - but, again, interrupts are checked for, it should clear.
• Sometimes the UI view indicating how many gens are waiting gets desynced (shows you have gens still queued after the server is emptied). When this happens, triggering a gen will immediately update it to the real number.
• When generating locally single images or small sets of eg 4 images in a row, on my machine interrupts are near-instant as long as they happen while the generation is stepping (ksampler, when there's a preview image showing up).
• Thaaaat all said, if you're able to replicate consistent delays with interrupts please do post an issue with a copy of your debug log and any info you can add about when you're cancelling or configuration setups (is it only when multiple GPUs? a remote backend? only when about X many images in queue? does it happen when you're mixing different models in the queue?)

[Mushy-Snugglebites-badonkadonk]

By default, if you have more gen requests than GPUs, Swarm queues 2 generations onto 1 GPU at a time (this is to guarantee that swarm's processing time and network overhead don't slow the requests down, since comfy has a linear queue built in internally). (You can control this in the backends tab, it's the "OverQueue" setting on a comfy self start backend). This might affect interrupt handling a bit - but, again, interrupts are checked for, it should clear.

I went down the rabbit hole on this one. So, I had set the Over Queue setting to 30 for the RTX 4090 and the GTX 1060 Over Queue value to 0. The idea was the bulk of the work would be done on the 4090 before handing off tasks to the aging 1060 with limited VRAM.

The objective was to see if there was an increase or decrease in efficiency by adding the 1060 to the swarm. As I suspected, the limited VRAM and processing power introduced bottlenecks into the workflow.

I took the logs and had GPT-4o look for any patterns and then report its findings which I have included below. 🤓

Summary Report: Analysis of Grid Generation Efficiency in Stable Diffusion Using ComfyUI with RTX 4090 and GTX 1060 GPUs

Objective:

To determine the efficiency and performance of grid generation in Stable Diffusion using ComfyUI with both the RTX 4090 and GTX 1060 GPUs versus using the RTX 4090 GPU alone.

Methodology:

Two sets of logs were analyzed:

1. RTX 4090 + GTX 1060 Combined Setup
2. RTX 4090 Solo Setup

The logs were reviewed to identify the start and stop times of the grid generation tasks, allowing for the calculation of task durations. Additionally, average prep and image generation times were analyzed.

Data Extracted:

RTX 4090 + GTX 1060 Combined Setup:

• Start Time: 2024-05-16 11:49:35.261
• Stop Time: 2024-05-16 11:59:32.733
• Duration: 9 minutes, 57.472 seconds

RTX 4090 Solo Setup:

• Start Time: 2024-05-16 12:05:57.866
• Stop Time: 2024-05-16 12:10:30.111
• Duration: 4 minutes, 32.245 seconds

Average Times:

Metric	RTX 4090 + GTX 1060	RTX 4090 Only
Average Prep Time	7-10 seconds	5-8 seconds
Average Gen Time	1-28 seconds	1-15 seconds
Error Instances	Multiple backend model loading errors	Minimal to none
Task Consistency	Varied performance, higher fluctuation	More consistent performance
Micro-pausing Instances	Frequent due to load balancing across GPUs	Rare
Overall Efficiency	Lower due to load balancing and older GPU bottleneck	Higher, stable, and efficient

Detailed Breakdown:

Initial Findings:

1. Prep and Gen Times:

   • RTX 4090 + GTX 1060: The combination showed higher fluctuation in both preparation (prep) and generation (gen) times. For example, prep times varied from 7 to 10 seconds, while gen times ranged from 1 to 28 seconds.
   • RTX 4090 Only: The times were more consistent, with prep times typically between 5 to 8 seconds and gen times between 1 to 15 seconds. The smaller range in values suggested more stable and predictable performance when using a single, powerful GPU.

2. Errors and Warnings:

   • RTX 4090 + GTX 1060: Multiple instances of backend model loading errors were noted, potentially stemming from the limitations of the GTX 1060, which might be due in part to the limited 6GB of VRAM.
   • RTX 4090 Only: Minimal errors were noted, showcasing the stability and reliability of using a single, more capable GPU.

3. Micro-pausing:

   • RTX 4090 + GTX 1060: Frequent micro-pausing to manage the load between the GPUs, introducing delays and inefficiencies.
   • RTX 4090 Only: Rare instances of micro-pausing, leading to smoother operation and faster completion times.

Final Analysis:

1. Performance Difference:

   • The solo RTX 4090 setup completed the grid generation task significantly faster (4 minutes, 32.245 seconds) compared to the combined RTX 4090 + GTX 1060 setup (9 minutes, 57.472 seconds).
   • This indicates that the GTX 1060 acts as a bottleneck when used alongside the RTX 4090, causing an overall increase in the time taken to complete the task.

2. Resource Efficiency:

   • Using only the RTX 4090 reduces the duration of the task and minimizes potential errors and synchronization issues from balancing the workload across GPUs with different capabilities.
   • This suggests that the system performs more efficiently and reliably when relying on the more powerful GPU alone.

3. Load Balancing and Bottleneck:

   • The significant time difference underscores the inefficiency introduced by the older GTX 1060. The powerful RTX 4090 is likely throttled by the need to wait for the GTX 1060 to complete its portion of the workload.
   • This highlights that in setups with mismatched GPU capabilities, the slower GPU with lower VRAM can degrade overall performance.

Conclusion:

For grid generation tasks in Stable Diffusion using ComfyUI, it is significantly more efficient and faster to use the RTX 4090 alone rather than combining it with the GTX 1060. The older GTX 1060 introduces unnecessary delays and errors, making the system less stable and predictable. Therefore, focusing on the single, more powerful RTX 4090 GPU is the optimal approach for achieving the best performance and efficiency.

_{(This report was generated with the help of AI)}