Error - I think dealing with models that have instruct pix2pix

[DirtyHamster]

@s1dlx Was attempting to do a re-basin merge with the following model: photosomnia_v16.instruct-pix2pix.safetensors The only thing different with this model is that it has pix2pix in the architecture, so I was thinking that might be the cause of the issue. I was doing one of my follow up test. I have to check inpainting too.

CLI prompt:
H:\Users\adamf\AI_Progs\sd-meh-merge\meh>merge_models.py -a H:\Users\adamf\AI_Progs\AI_Models\test\00-regit-pwddxWDP1x5EV5SMA5URPM440EMxdream6rellyrevebmev2_dblback2_20prune.safetensors -b H:\Users\adamf\AI_Progs\AI_Models\Stable_Diffusion\photosomnia_v16.instruct-pix2pix.safetensors -m weighted_sum -p 32 -o H:\Users\adamf\AI_Progs\AI_Models\test\00-regit-pwddxWDP1x5EV5SMA5URPM440EMxdream6rellyrevebmev2_dblback2pix_20prune -f safetensors -ba 0.5 -bb 0.5 -pr -rb -rbi 20

I get the following error:

before loading models: 0.000
loading: H:\Users\adamf\AI_Progs\AI_Models\test\00-regit-pwddxWDP1x5EV5SMA5URPM440EMxdream6rellyrevebmev2_dblback2_20prune.safetensors
loading: H:\Users\adamf\AI_Progs\AI_Models\Stable_Diffusion\photosomnia_v16.instruct-pix2pix.safetensors
models loaded: 0.000
permuting
0 iteration start: 0.000
weights & bases, before simple merge: 0.000
stage 1: 100%|████████████████████████████████████████████████████████████████████▉| 1130/1131 [00:42<00:00, 26.46it/s]
Traceback (most recent call last):
File "H:\Users\adamf\AI_Progs\sd-meh-merge\meh\merge_models.py", line 151, in
main()
File "C:\Users\adamf\AppData\Local\Programs\Python\Python310\lib\site-packages\click\core.py", line 1157, in call
return self.main(*args, **kwargs)
File "C:\Users\adamf\AppData\Local\Programs\Python\Python310\lib\site-packages\click\core.py", line 1078, in main
rv = self.invoke(ctx)
File "C:\Users\adamf\AppData\Local\Programs\Python\Python310\lib\site-packages\click\core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "C:\Users\adamf\AppData\Local\Programs\Python\Python310\lib\site-packages\click\core.py", line 783, in invoke
return __callback(*args, **kwargs)
File "H:\Users\adamf\AI_Progs\sd-meh-merge\meh\merge_models.py", line 132, in main
merged = merge_models(
File "H:\Users\adamf\AI_Progs\sd-meh-merge\meh\sd_meh\merge.py", line 146, in merge_models
merged = rebasin_merge(
File "H:\Users\adamf\AI_Progs\sd-meh-merge\meh\sd_meh\merge.py", line 286, in rebasin_merge
thetas["model_a"] = simple_merge(
File "H:\Users\adamf\AI_Progs\sd-meh-merge\meh\sd_meh\merge.py", line 244, in simple_merge
res.result()
File "C:\Users\adamf\AppData\Local\Programs\Python\Python310\lib\concurrent\futures_base.py", line 451, in result
return self.__get_result()
File "C:\Users\adamf\AppData\Local\Programs\Python\Python310\lib\concurrent\futures_base.py", line 403, in __get_result
raise self._exception
File "C:\Users\adamf\AppData\Local\Programs\Python\Python310\lib\concurrent\futures\thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "H:\Users\adamf\AI_Progs\sd-meh-merge\meh\sd_meh\merge.py", line 342, in simple_merge_key
with merge_key_context(key, thetas, *args, **kwargs) as result:
File "C:\Users\adamf\AppData\Local\Programs\Python\Python310\lib\contextlib.py", line 135, in enter
return next(self.gen)
File "H:\Users\adamf\AI_Progs\sd-meh-merge\meh\sd_meh\merge.py", line 428, in merge_key_context
result = merge_key(*args, **kwargs)
File "H:\Users\adamf\AI_Progs\sd-meh-merge\meh\sd_meh\merge.py", line 403, in merge_key
merged_key = merge_method(**merge_args).to(storage_device)
File "H:\Users\adamf\AI_Progs\sd-meh-merge\meh\sd_meh\merge_methods.py", line 27, in weighted_sum
return (1 - alpha) * a + alpha * b
RuntimeError: The size of tensor a (4) must match the size of tensor b (8) at non-singleton dimension 1

[s1dlx]

I'm not familiar with pix2pix architecture, but it seems that you have a layer with a different size from the other model one, hence it's not possible to merge those

also, you are rebasin the two, which may complicate further the situation

try to weight_sum merge the two, do you get the same error?

[DirtyHamster]

Same error when just dropping the rebasin args and having it run as a normal weight_sum. I'm not that familiar with the architecture of it either.

https://huggingface.co/blog/instruction-tuning-sd
https://www.tensorflow.org/tutorials/generative/pix2pix

I'm actually curious if the layer isn't present in both models then perhaps it could just be pruned and returned later, so it can retain the functionality.

H:\Users\adamf\AI_Progs\sd-meh-merge\meh>merge_models.py -a H:\Users\adamf\AI_Progs\AI_Models\test\00-regit-pwddxWDP1x5EV5SMA5URPM440EMxdream6rellyrevebm_10prune.safetensors -b H:\Users\adamf\AI_Progs\AI_Models\Stable_Diffusion\photosomnia_v16.instruct-pix2pix.safetensors -m weighted_sum -p 32 -o H:\Users\adamf\AI_Progs\AI_Models\test\testweightsum -f safetensors -ba 0.5 -bb 0.5 -pr

before loading models: 0.000
loading: H:\Users\adamf\AI_Progs\AI_Models\test\00-regit-pwddxWDP1x5EV5SMA5URPM440EMxdream6rellyrevebm_10prune.safetensors
loading: H:\Users\adamf\AI_Progs\AI_Models\Stable_Diffusion\photosomnia_v16.instruct-pix2pix.safetensors
models loaded: 0.000
stage 1: 100%|████████████████████████████████████████████████████████████████████▉| 1130/1131 [01:30<00:00, 12.54it/s]
Traceback (most recent call last):
File "H:\Users\adamf\AI_Progs\sd-meh-merge\meh\merge_models.py", line 151, in
main()
File "C:\Users\adamf\AppData\Local\Programs\Python\Python310\lib\site-packages\click\core.py", line 1157, in call
return self.main(*args, **kwargs)
File "C:\Users\adamf\AppData\Local\Programs\Python\Python310\lib\site-packages\click\core.py", line 1078, in main
rv = self.invoke(ctx)
File "C:\Users\adamf\AppData\Local\Programs\Python\Python310\lib\site-packages\click\core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "C:\Users\adamf\AppData\Local\Programs\Python\Python310\lib\site-packages\click\core.py", line 783, in invoke
return __callback(*args, **kwargs)
File "H:\Users\adamf\AI_Progs\sd-meh-merge\meh\merge_models.py", line 132, in main
merged = merge_models(
File "H:\Users\adamf\AI_Progs\sd-meh-merge\meh\sd_meh\merge.py", line 162, in merge_models
merged = simple_merge(
File "H:\Users\adamf\AI_Progs\sd-meh-merge\meh\sd_meh\merge.py", line 244, in simple_merge
res.result()
File "C:\Users\adamf\AppData\Local\Programs\Python\Python310\lib\concurrent\futures_base.py", line 451, in result
return self.__get_result()
File "C:\Users\adamf\AppData\Local\Programs\Python\Python310\lib\concurrent\futures_base.py", line 403, in __get_result
raise self._exception
File "C:\Users\adamf\AppData\Local\Programs\Python\Python310\lib\concurrent\futures\thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "H:\Users\adamf\AI_Progs\sd-meh-merge\meh\sd_meh\merge.py", line 342, in simple_merge_key
with merge_key_context(key, thetas, *args, **kwargs) as result:
File "C:\Users\adamf\AppData\Local\Programs\Python\Python310\lib\contextlib.py", line 135, in enter
return next(self.gen)
File "H:\Users\adamf\AI_Progs\sd-meh-merge\meh\sd_meh\merge.py", line 428, in merge_key_context
result = merge_key(*args, **kwargs)
File "H:\Users\adamf\AI_Progs\sd-meh-merge\meh\sd_meh\merge.py", line 403, in merge_key
merged_key = merge_method(**merge_args).to(storage_device)
File "H:\Users\adamf\AI_Progs\sd-meh-merge\meh\sd_meh\merge_methods.py", line 27, in weighted_sum
return (1 - alpha) * a + alpha * b
RuntimeError: The size of tensor a (4) must match the size of tensor b (8) at non-singleton dimension 1

[DirtyHamster]

@s1dlx I tried running it on the default merger via Vlad's and got the following error on his:
RuntimeError: When merging instruct-pix2pix model with a normal one, A must be the instruct-pix2pix model.

I tried swapping A and B on yours but got the same error as in the previous message. Swapping them did work on vlad's so there might be some code in there that you can look at to see how it deals with the architecture for pix2pix at least for the normal merge.

Full read out:

23:51:36-222934 INFO Model merge loading secondary model:
H:\Users\adamf\AI_Progs\AI_Models\Stable_Diffusion\photosomnia_v16.instruct-pix2pix.safetensors
Loading weights: H:\Users\adamf\AI_Progs\AI_Models\Stable_Diffusion\photosomnia_v16.instruct-pix2pix.safetensors ━ 0… -…
GB
23:51:37-917656 INFO Model merge loading primary model:
H:\Users\adamf\AI_Progs\AI_Models\Stable_Diffusion\HD-22-fp16-fixclip-noema.safetensors
Loading weights: H:\Users\adamf\AI_Progs\AI_Models\Stable_Diffusion\HD-22-fp16-fixclip-noema.safetensors ━━━━ 0.0… -:--…
GB
23:51:41-717749 INFO Model merge: running
39%|███████████████████████████████ | 450/1143 [00:09<00:14, 47.69it/s]
23:51:51-155872 ERROR model merge: RuntimeError
╭───────────────────────────────────────── Traceback (most recent call last) ──────────────────────────────────────────╮
│ H:\Users\adamf\AI_Progs\VladDiffusion\automatic\modules\ui_models.py:100 in modelmerger │
│ │
│ 99 │ │ │ │ │ try: │
│ ❱ 100 │ │ │ │ │ │ results = extras.run_modelmerger(*args) │
│ 101 │ │ │ │ │ except Exception as e: │
│ │
│ H:\Users\adamf\AI_Progs\VladDiffusion\automatic\modules\extras.py:154 in run_modelmerger │
│ │
│ 153 │ │ │ │ if a.shape[1] == 4 and b.shape[1] == 8: │
│ ❱ 154 │ │ │ │ │ raise RuntimeError("When merging instruct-pix2pix model with a norma │
│ 155 │ │ │ │ if a.shape[1] == 8 and b.shape[1] == 4:#If we have an Instruct-Pix2Pix m │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
RuntimeError: When merging instruct-pix2pix model with a normal one, A must be the instruct-pix2pix model.
23:51:58-233415 INFO Available models: H:\Users\adamf\AI_Progs\AI_Models\Stable_Diffusion 117

[s1dlx]

just to clarify: using pix2pix as model a doesn't work on both vlad and meh?

[WhiteWipe]

That extra layer is trained along with the rest of the weights. If this is anything like inpainting, just adding the layer to a normal model works, but causes a drop in quality. I am really curios if there is a way when merging to modify the weights to the extra layer relative to model that's being merged with inapinting/pix2pix model to improve quality. A question I can't answer: Can git rebasin be repurposed for this use case where the normal sd weights are merged normally, but the entire inpainting/pix2pix model is permutated towards the normal model separately and then add at the end just the permutated layer to the merge.

Edit: Git rebasin can presume the normal sd model has the extra layer, but with weights zero.

[DirtyHamster]

just to clarify: using pix2pix as model a doesn't work on both vlad and meh?

It works in vlad's only as model a
It doesn't work in meh as a or b.

[DirtyHamster]

Sorry non-intended mouse click closed the issue for a sec. lol.

[DirtyHamster]

That extra layer is trained along with the rest of the weights. If this is anything like inpainting, just adding the layer to a normal model works, but causes a drop in quality. I am really curios if there is a way when merging to modify the weights to the extra layer relative to model that's being merged with inapinting/pix2pix model to improve quality. A question I can't answer: Can git rebasin be repurposed for this use case where the normal sd weights are merged normally, but the entire inpainting/pix2pix model is permutated towards the normal model separately and then add at the end just the permutated layer to the merge.

Edit: Git rebasin can presume the normal sd model has the extra layer, but with weights zero.

It looks from what I'm reading that unless they fine tuned their pix2pix model for extra functionality the quality loss would be negligible and it would only ever be as good as the pix2pix base model can generate. https://huggingface.co/blog/instruction-tuning-sd From that an a few other small articles that weren't a ton of help it seems like pix2pix is it's own complete model that can be added in or run separately. Correct me if I'm reading into this wrong though.

So it probably could just be pruned out and added in before saving. I'm not sure how a merge should handle 2 models with pix2pix though and I have yet to see if that generates any other errors. Similarly I'm not sure how to identify if a pix2pix model has been fine tuned.

[WhiteWipe]

Looked into it just to be sure. It's the same story as an inpainting model. You take a model and train all of it with an extra layer on top. It should work just by appending the layer to a normal model, but there still should be some quality loss simply because the extra layer is slightly out of sync with the normal weights. Nonetheless, I'll try to do a PR soon for meh for pix2pix(and inpainting if it doesn't already work) and somebody can test how well the resulting merge works.

[DirtyHamster]

@WhiteWipe I can retest it if you can fix it those portions..

[s1dlx]

0.9.0 should fix this