Plugin support for custom LLM providers now available

[aksg87]

Hi all,

LangExtract now supports third-party model providers through a new plugin and registry infrastructure. You can integrate custom LLM backends (Azure OpenAI, AWS Bedrock, custom inference servers, etc.) without modifying core LangExtract code.

Please check out the example and documentation:
https://github.com/google/langextract/tree/main/examples/custom_provider_plugin

Feel free to provide feedback or report any issues. Thank you!

@YunfanGoForIt @mariano @zc1175 @praneeth999 @JustStas @jkitaok

[JustStas]

@aksg87 thanks for the feature!
Do you think it would make sense to include some of the more popular providers (azure, bedrock) as core providers?
Happy to provide a PR in the new format.
Especially considering Azure OpenAI will be quite similar to the already supported OpenAI

[JustStas]

@aksg87 I have updated #38 with the new approach and switched to inheriting the bulk of the functionality from the openai.py provider. I think this way it doesn't generate much additional effort for long-term support while covering a recurrent need for many users (especially in corporate projects).

Otherwise, we could also add a LiteLLM core provider to cover many LLMs with minimum long-term support overhead - please tell me if you think this makes sense.

[mariano]

@aksg87 Looks great. Question, currently extract() builds the model config and instance from params. Shouldn't it also be possible to pass either ModelConfig or a factory-instantiated model? So instead of:

result = extract(
    prompt_description=settings.specs.header.prompt,
    examples=settings.specs.header.examples,
    text_or_documents=content,
    language_model_type=OpenAILanguageModel,
    language_model_params={
        "base_url": "http://localhost:9001/v1",
        "api_key": environ["OPENAI_API_KEY"]
    },
    model_id="gpt-4o-mini",
    use_schema_constraints=False,
    fence_output=True,
    debug=False
)

We could do:

model_config = ModelConfig(
    model_id="gpt-4o-mini",
    provider=OpenAILanguageModel,
    provider_kwargs={
        "base_url": "http://localhost:9001/v1",
        "api_key": environ["OPENAI_API_KEY"]
    }
)
result = extract(
    prompt_description=settings.specs.header.prompt,
    examples=settings.specs.header.examples,
    text_or_documents=content,
    config=model_config,
    use_schema_constraints=False,
    fence_output=True,
    debug=False
)

Or even:

model_config = ModelConfig(
    model_id="gpt-4o-mini",
    provider=OpenAILanguageModel,
    provider_kwargs={
        "base_url": "http://localhost:9001/v1",
        "api_key": environ["OPENAI_API_KEY"]
    }
)
model = factory.create_model(model_config)
result = extract(
    prompt_description=settings.specs.header.prompt,
    examples=settings.specs.header.examples,
    text_or_documents=content,
    model=model,
    use_schema_constraints=False,
    fence_output=True,
    debug=False
)

Thoughts? I can create a PR for this if it sounds good.

[aksg87]

Hey @JustStas, thanks for the PR offer!

Given that we just landed a provider registry + plugin system (#97), I think the ideal path for Azure OpenAI and Bedrock would be as external provider plugins. We actually added OpenAI as a stopgap given high demand, but even that might move to a plugin eventually - right now I want to keep the core focused on extraction features rather than managing the provider ecosystem.

Would you be up for creating langextract-azure-openai and/or langextract-bedrock as separate packages? Others like LiteLLM would also work great as a plugin.

Once you publish and a few folks validate it works well, we'll add a community providers section to feature it. This way the community can iterate quickly on provider support while we focus on improving the core extraction capabilities.

@mariano great idea on the config and model parameters - moved it to #106!