Skip to content

AutoModel #11115

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 13 commits into from
Apr 9, 2025
Merged

AutoModel #11115

merged 13 commits into from
Apr 9, 2025

Conversation

hlky
Copy link
Contributor

@hlky hlky commented Mar 19, 2025

What does this PR do?

Fixes #10059

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@hlky hlky requested review from DN6 and sayakpaul March 19, 2025 08:22
@HuggingFaceDocBuilderDev

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Copy link
Member

@sayakpaul sayakpaul left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! I think this will be so much better as an UX improvement. Cc: @vladmandic you might like this :)

Let's add docs and tests after @DN6 reviews.

@ParagEkbote
Copy link
Contributor

I anticipate this functionality to be quite helpful in a future version of diffusers. Does it make sense to add this PR to the future release section of the diffusers roadmap?

@hlky hlky added the roadmap Add to current release roadmap label Mar 20, 2025
@hlky
Copy link
Contributor Author

hlky commented Mar 20, 2025

Hi @ParagEkbote, I've added this to the roadmap, we will aim to get it included with the next release.

@vladmandic
Copy link
Contributor

an idea - to really make automodel as simple as possible, add default pretrained_model_or_path to mappings for each model where possible.

@hlky
Copy link
Contributor Author

hlky commented Apr 2, 2025

@vladmandic Do you have an example pseudo-code of your expected usage?

@vladmandic
Copy link
Contributor

@vladmandic Do you have an example pseudo-code of your expected usage?

sorry, i was too fast with my comment. if using auto-pipeline, we dont know what's the model ahead of time.
my comment was more general, if we do use specific pipeline, there should be a default model specified.

one thing with automodel i'd love to see is the model type.
e.g., i want to list available txt2img pipelines so i can display to user which models are supported. right now everything is a flat list without any way to differentiate them.

config = cls.load_config(pretrained_model_or_path, **load_config_kwargs)
orig_class_name = config["_class_name"]

model_cls = _get_task_class(AUTO_MODEL_MAPPING, orig_class_name)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rather than maintain this big mapping can't we use importlib? All the models are available in the main diffusers init right? Similar to how we do it here?

# else we just import it from the library.
library = importlib.import_module(library_name)
class_obj = getattr(library, class_name)
class_candidates = {c: getattr(library, c, None) for c in importable_classes.keys()}

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, agree, probably don't need a mapping here, could use the same logic as in from_pretrained

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if we can get rid of mapping, everything will be supported automatically too I think

@@ -895,6 +910,21 @@ def from_pretrained(cls, *args, **kwargs):
requires_backends(cls, ["torch"])


class TransformerTemporalModel(metaclass=DummyObject):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was missing, needed to be added for the test to pass.

Copy link
Collaborator

@DN6 DN6 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work 👍🏽

@DN6 DN6 merged commit 437cb36 into huggingface:main Apr 9, 2025
28 of 29 checks passed
@github-project-automation github-project-automation bot moved this from In Progress to Done in Diffusers Roadmap 0.35 Apr 9, 2025
@hlky hlky mentioned this pull request Apr 9, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
roadmap Add to current release roadmap
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[core] introduce AutoModel
7 participants