Is retraining/fine tuning possible?

[zarg1]

I have several thousand image/label pairs that I would like stable diffusion to perform better on. Is it possible to re-train the model or fine tune it so that my image/label pairs can influence the model?

I am not sure if it's possible to do that with this repo. But if not, do you know of any ways to do this, even outside this repo? I did see the 5-image fine tuning tutorial using textual inversion here, but I do not have access to a GPU with 20GB memory.

[bitRAKE]

The model parameters can have such a drastic effect on the output.

What kind of improvement are you wanting?

img2img can reapply a prompt & seed to the same image - have you tried that?

[zarg1]

Thank you for your reply. I am aware of the prompt & seed image, but unfortunately I do not have a specific seed image I want for an output. You can assume the prompt creator does not have a seed image. Instead I would like to influence the output so that it incorporates the image/label pairs that I have.

For instance, let's say I have lots of labeled pictures of 'ZargBots' in different contexts which do not exist in the original training dataset. I would like to influence the model so that it can create pictures that include ZargBots. All publicly published models do not know what a ZargBot is, so I would have to somehow train my own.

Usually fine tuning is freezing all but the last layers of a model, at least for image classification, and retraining the last layers on your custom data. I am used to doing this with image classification models, but I don't know how I would do that with stable diffusion yet. Or if it's possible.