Add support for Batch API of LLMs like OpenAI and Anthropic

[mfsi-suryar]

Checked

• I searched existing ideas and did not find a similar one
• I added a very descriptive title
• I've clearly described the feature request and motivation for it

Feature request

Large Language Models like OpenAI and Anthropic supports Batch request of prompts using their APIs with enhanced token limit and reduced cost. Currently those are not getting supported in current version of Langchain.

Motivation

I am a fullstack LLM apps developer and my current project which is utilizing langchain needed support for the Batch APIs of OpenAI.

Proposal (If applicable)

We should define adapters and standard intefaces for the Batch APIs within langchain_core and leave the implementation to the packages such as langchain_openai, langchain_anthropic and also for others as well.

[hect1c]

Bumping this as I added a discussion here, looking for the same thing for Google's Batch request langchain-ai/langchain-google#599.

Are there any plans to support this sometime soon or on the roadmap? I'd be happy to help look into this if some guidance is provided as to where is the best place to start or pieces of code to look at that we can leverage to get something working.

[garethcurtis]

I've successfully integrated Langchain to my test code but my main production code would definitely use the batch processing from Anthropic because of the significant saving and because I'm analysing several hundred images on a daily basis potentially.

I'd still like to use Langchain but after reading the above, I will just hack it to use Langsmith logging instead.

[codeonprod]

@garethcurtis
But to hack langsmith to track batch responses, langsmith should also support processing of the same data.

[garethcurtis]

Hack was the wrong word. I meant use @Traceable instead of Langchain. I agree it would be nice to have Langchain support batch processing.

[BlakeQG]

I can't believe it. Many people have this request many times in multiple posts, the Langchain still doesn't support batch API yet?!

[aubford]

I was also surprised there's no native support for batch APIs given how common they are in production LLM workflows. Seems like the tricky part is that batching essentially "breaks the chain" since you have to wait for the batch to complete before resuming any chain logic. Seems like this runs somewhat counter to LangChain's design philosophy of frictionless pluggability e.g. extending a high level interface like Runnable with use_batch_api so you could swap in batch API for any given chain isn't feasible.

One idea: extend BaseChatModel with to_batch_api(batch_name: str) and from_batch_api(batch_name: str) to explicitly mark batch boundaries. That would let you split and resume chains like:
prompt | retriever | llm.to_batch_api("batch_a")
...and later:
llm.from_batch_api("batch_a") | output_parser

Might be naive, but could be a path forward...

[skabbit]

Developer should understand, that using batch API is somewhat they "have to wait", so resuming chain only after batch is complete should expectable.

I assume that something easy like prompt | retriever | llm.with_batch_api() | output_parser is simple and obvious way to define batch API usage within the chain.

[aubford]

How are you suggesting that would be executed? Batch APIs can take up to 24hrs to complete. It's not practical to design with the expectation that developers are going to leave the process running/polling potentially overnight. Users could build an implementation for running on e.g. a cloud process but pretty sure that would be out of scope for LangChain itself.

[aubford]

It also occurred to me that there really should be an integration to provide the option to leverage Batch APIs for running LangSmith evaluations in the cloud. That actually would be a pluggable solution. Also could abstract frustrating things like batch expiration (I'm looking at you OpenAI). I figure people are more willing to pay for tools that save them money, right?