Bad outputs with Pony models

[moonwhaler]

First I used my old styles and tried to inpaint older gens. This resulted in very strange looking output at the inpainted area. So I tried to create a completely new image and generate with 100% strength:

The output is not really recognizable and looks similar to the inpainted results. I really didn't change the Sampler and other settings since a few months ago.

Sampler: Euler
Steps: 20
CFG: 4.5
Pony Model: cashmoney_v10

This is a local gen with a local managed server (by Krita AI). Core XL components installed including ControlNets (Universal XL, Stencil, IP Adapter Face). I also reinstalled Comfy using the in-application options + verified the models.

No performance options active.

[dauwpe]

Could this be due to Clipskip being on 1?

[Acly]

Maybe, hard to tell.

Try to dump the workflow that is produced and we might be able to say more.

[moonwhaler]

I rechecked, but always used Clip 2 for Pony - this was already set in my style, since ever.
Here is the workflow.

workflow.json

Settings:

[Acly]

I don't have the checkpoint, but substituting a different Pony finetune the workflow you posted works fine.

Produces stuff like this

[moonwhaler]

I don't have the checkpoint, but substituting a different Pony finetune the workflow you posted works fine.

Produces stuff like this

Which also looks quite strange. I will try again with ComfyUI directly, but as I said, nothing much changed on my side. My guess it has something to do with things that are not included in the checkpoint (pruned), but are in others. So those are working more or less OK.

[moonwhaler]

I now went ahead and loaded the dumped workflow into (a personal) ComfyUI instance. Using the dumped workflow "as is" also generated a garbled image. I then changed the checkpoint loader so that CLIP is also used from the model and then the output was fine. My guess is, the CLIP models is/are the culprit.

With default dump:

With changed loader / CLIP:

New workflow:

workflow_pony_working.json

I redownloaded the (I guess SDXL 3.5, which may be wrong) CLIP models, but no change.
From the SDXL docs:

text_encoder (CLIPTextModel) — Frozen text-encoder. Stable Diffusion XL uses the text portion of CLIP, specifically the clip-vit-large-patch14 variant.

text_encoder_2 ( CLIPTextModelWithProjection) — Second frozen text-encoder. Stable Diffusion XL uses the text and pool portion of CLIP, specifically the laion/CLIP-ViT-bigG-14-laion2B-39B-b160k variant.

Per default CLIP_L and CLIP_G seem to be used, which are not the correct CLIP for Pony. Not sure, if that's an issue with my installation or not. Thank you.

[Acly]

Per default CLIP_L and CLIP_G seem to be used, which are not the correct CLIP for Pony

Looks like it, I think Pony may have trained the clip-l so it probably has to be a different one.

If you really want to you can extract it from a pony checkpoint (there is a ClipSave node in comfyui). This all seems to be a lot of hassle though compared to just using regular checkpoint with embedded clip.

[moonwhaler]

It's interesting, that this was working with the same models just a few weeks or months ago. Is there a way to select a default CLIP in a style setting - or maybe a standard workflow per style? That way one could mitigate those issues.

[moonwhaler]

The culprit was me using extra model paths and pointing UNET and CHECKPOINTS to the same folder. It works now.
Thanks @Acly