Skip to content

Commit a03e7cb

Browse files
markpollacktzolov
markpollack
authored and
tzolov
committed
This commit enhances the documentation for working with multiple chat models in Spring AI by:
- Add introduction explaining common scenarios where multiple chat models are useful. - Restructuring the content into clear subsections: - Multiple ChatClients with a Single Model Type - ChatClients for Different Model Types - Multiple OpenAI-Compatible API Endpoints Signed-off-by: Mark Pollack <mark.pollack@broadcom.com>
1 parent 87b680a commit a03e7cb

File tree

1 file changed

+155
-74
lines changed

1 file changed

+155
-74
lines changed

‎spring-ai-docs/src/main/antora/modules/ROOT/pages/api/chatclient.adoc

Lines changed: 155 additions & 74 deletions
Original file line numberDiff line numberDiff line change
@@ -47,23 +47,170 @@ class MyController {
4747
In this simple example, the user input sets the contents of the user message.
4848
The `call()` method sends a request to the AI model, and the `content()` method returns the AI model's response as a `String`.
4949

50-
=== Create a ChatClient programmatically
50+
=== Working with Multiple Chat Models
5151

52-
You can disable the `ChatClient.Builder` autoconfiguration by setting the property `spring.ai.chat.client.enabled=false`.
53-
This is useful if multiple chat models are used together.
54-
Then, create a `ChatClient.Builder` instance programmatically for every `ChatModel` you need:
52+
There are several scenarios where you might need to work with multiple chat models in a single application:
53+
54+
* Using different models for different types of tasks (e.g., a powerful model for complex reasoning and a faster, cheaper model for simpler tasks)
55+
* Implementing fallback mechanisms when one model service is unavailable
56+
* A/B testing different models or configurations
57+
* Providing users with a choice of models based on their preferences
58+
* Combining specialized models (one for code generation, another for creative content, etc.)
59+
60+
By default, Spring AI autoconfigures a single `ChatClient.Builder` bean. However, you may need to work with multiple chat models in your application. Here's how to handle this scenario:
61+
62+
In all cases, you need to disable the `ChatClient.Builder` autoconfiguration by setting the property `spring.ai.chat.client.enabled=false`.
63+
64+
This allows you to create multiple `ChatClient` instances manually.
65+
66+
==== Multiple ChatClients with a Single Model Type
67+
68+
This section covers a common use case where you need to create multiple ChatClient instances that all use the same underlying model type but with different configurations.
69+
70+
[source,java]
71+
----
72+
// Create ChatClient instances programmatically
73+
ChatModel myChatModel = ... // already autoconfigured by Spring Boot
74+
ChatClient chatClient = ChatClient.create(myChatModel);
75+
76+
// Or use the builder for more control
77+
ChatClient.Builder builder = ChatClient.builder(myChatModel);
78+
ChatClient customChatClient = builder
79+
.defaultSystemPrompt("You are a helpful assistant.")
80+
.build();
81+
----
82+
83+
==== ChatClients for Different Model Types
84+
85+
When working with multiple AI models, you can define separate `ChatClient` beans for each model:
86+
87+
[source,java]
88+
----
89+
import org.springframework.ai.chat.ChatClient;
90+
import org.springframework.context.annotation.Bean;
91+
import org.springframework.context.annotation.Configuration;
92+
93+
@Configuration
94+
public class ChatClientConfig {
95+
96+
@Bean
97+
public ChatClient openAiChatClient(OpenAiChatModel chatModel) {
98+
return ChatClient.create(chatModel);
99+
}
100+
101+
@Bean
102+
public ChatClient anthropicChatClient(AnthropicChatModel chatModel) {
103+
return ChatClient.create(chatModel);
104+
}
105+
}
106+
----
107+
108+
You can then inject these beans into your application components using the `@Qualifier` annotation:
55109

56110
[source,java]
57111
----
58-
ChatModel myChatModel = ... // usually autowired
59112
60-
ChatClient.Builder builder = ChatClient.builder(this.myChatModel);
113+
@Configuration
114+
public class ChatClientExample {
115+
116+
@Bean
117+
CommandLineRunner cli(
118+
@Qualifier("openAiChatClient") ChatClient openAiChatClient,
119+
@Qualifier("anthropicChatClient") ChatClient anthropicChatClient) {
120+
121+
return args -> {
122+
var scanner = new Scanner(System.in);
123+
ChatClient chat;
124+
125+
// Model selection
126+
System.out.println("\nSelect your AI model:");
127+
System.out.println("1. OpenAI");
128+
System.out.println("2. Anthropic");
129+
System.out.print("Enter your choice (1 or 2): ");
130+
131+
String choice = scanner.nextLine().trim();
132+
133+
if (choice.equals("1")) {
134+
chat = openAiChatClient;
135+
System.out.println("Using OpenAI model");
136+
} else {
137+
chat = anthropicChatClient;
138+
System.out.println("Using Anthropic model");
139+
}
140+
141+
// Use the selected chat client
142+
System.out.print("\nEnter your question: ");
143+
String input = scanner.nextLine();
144+
String response = chat.prompt(input).call().content();
145+
System.out.println("ASSISTANT: " + response);
146+
147+
scanner.close();
148+
};
149+
}
150+
}
151+
----
152+
153+
==== Multiple OpenAI-Compatible API Endpoints
61154

62-
// or create a ChatClient with the default builder settings:
155+
The `OpenAiApi` and `OpenAiChatModel` classes provide a `mutate()` method that allows you to create variations of existing instances with different properties. This is particularly useful when you need to work with multiple OpenAI-compatible APIs.
63156

64-
ChatClient chatClient = ChatClient.create(this.myChatModel);
157+
[source,java]
65158
----
66159
160+
@Service
161+
public class MultiModelService {
162+
163+
private static final Logger logger = LoggerFactory.getLogger(MultiModelService.class);
164+
165+
@Autowired
166+
private OpenAiChatModel baseChatModel;
167+
168+
@Autowired
169+
private OpenAiApi baseOpenAiApi;
170+
171+
public void multiClientFlow() {
172+
try {
173+
// Derive a new OpenAiApi for Groq (Llama3)
174+
OpenAiApi groqApi = baseOpenAiApi.mutate()
175+
.baseUrl("https://api.groq.com/openai")
176+
.apiKey(System.getenv("GROQ_API_KEY"))
177+
.build();
178+
179+
// Derive a new OpenAiApi for OpenAI GPT-4
180+
OpenAiApi gpt4Api = baseOpenAiApi.mutate()
181+
.baseUrl("https://api.openai.com")
182+
.apiKey(System.getenv("OPENAI_API_KEY"))
183+
.build();
184+
185+
// Derive a new OpenAiChatModel for Groq
186+
OpenAiChatModel groqModel = baseChatModel.mutate()
187+
.openAiApi(groqApi)
188+
.defaultOptions(OpenAiChatOptions.builder().model("llama3-70b-8192").temperature(0.5).build())
189+
.build();
190+
191+
// Derive a new OpenAiChatModel for GPT-4
192+
OpenAiChatModel gpt4Model = baseChatModel.mutate()
193+
.openAiApi(gpt4Api)
194+
.defaultOptions(OpenAiChatOptions.builder().model("gpt-4").temperature(0.7).build())
195+
.build();
196+
197+
// Simple prompt for both models
198+
String prompt = "What is the capital of France?";
199+
200+
String groqResponse = ChatClient.builder(groqModel).build().prompt(prompt).call().content();
201+
String gpt4Response = ChatClient.builder(gpt4Model).build().prompt(prompt).call().content();
202+
203+
logger.info("Groq (Llama3) response: {}", groqResponse);
204+
logger.info("OpenAI GPT-4 response: {}", gpt4Response);
205+
}
206+
catch (Exception e) {
207+
logger.error("Error in multi-client flow", e);
208+
}
209+
}
210+
}
211+
----
212+
213+
67214
== ChatClient Fluent API
68215

69216
The `ChatClient` fluent API allows you to create a prompt in three distinct ways using an overloaded `prompt` method to initiate the fluent API:
@@ -407,72 +554,6 @@ In this configuration, the `MessageChatMemoryAdvisor` will be executed first, ad
407554

408555
xref:ROOT:api/retrieval-augmented-generation.adoc#_questionansweradvisor[Learn about Question Answer Advisor]
409556

410-
TIP: Refer to the xref:api/chat-memory.adoc[Chat Memory] documentation for more information on how to use the `ChatMemory` interface to manage conversation history in combination with the advisors.
411-
412-
The following advisor implementations use the `ChatMemory` interface to advice the prompt with conversation history which differ in the details of how the memory is added to the prompt
413-
414-
* `MessageChatMemoryAdvisor` : Memory is retrieved and added as a collection of messages to the prompt
415-
* `PromptChatMemoryAdvisor` : Memory is retrieved and added into the prompt's system text.
416-
* `VectorStoreChatMemoryAdvisor` : The constructor `VectorStoreChatMemoryAdvisor(VectorStore vectorStore, String defaultConversationId, int chatHistoryWindowSize, int order)` This constructor allows you to:
417-
418-
. Specify the VectorStore instance used for managing and querying documents.
419-
. Set a default conversation ID to be used if none is provided in the context.
420-
. Define the window size for chat history retrieval in terms of token size.
421-
. Provide system text advice used for the chat advisor system.
422-
. Set the order of precedence for this advisor in the chain.
423-
424-
425-
The `VectorStoreChatMemoryAdvisor.builder()` method lets you specify the default conversation ID, the chat history window size, and the order of the chat history to be retrieved.
426-
427-
A sample `@Service` implementation that uses several advisors is shown below.
428-
429-
[source,java]
430-
----
431-
import static org.springframework.ai.chat.memory.ChatMemory.CHAT_MEMORY_CONVERSATION_ID_KEY;
432-
import static org.springframework.ai.chat.client.advisor.vectorstore.VectorStoreChatMemoryAdvisor.CHAT_MEMORY_RETRIEVE_SIZE_KEY;
433-
434-
@Service
435-
public class CustomerSupportAssistant {
436-
437-
private final ChatClient chatClient;
438-
439-
public CustomerSupportAssistant(ChatClient.Builder builder, VectorStore vectorStore, ChatMemory chatMemory) {
440-
441-
this.chatClient = builder
442-
.defaultSystem("""
443-
You are a customer chat support agent of an airline named "Funnair". Respond in a friendly,
444-
helpful, and joyful manner.
445-
446-
Before providing information about a booking or cancelling a booking, you MUST always
447-
get the following information from the user: booking number, customer first name and last name.
448-
449-
Before changing a booking you MUST ensure it is permitted by the terms.
450-
451-
If there is a charge for the change, you MUST ask the user to consent before proceeding.
452-
""")
453-
.defaultAdvisors(
454-
new MessageChatMemoryAdvisor(chatMemory), // CHAT MEMORY
455-
new QuestionAnswerAdvisor(vectorStore), // RAG
456-
new SimpleLoggerAdvisor())
457-
.defaultFunctions("getBookingDetails", "changeBooking", "cancelBooking") // FUNCTION CALLING
458-
.build();
459-
}
460-
461-
public Flux<String> chat(String chatId, String userMessageContent) {
462-
463-
return this.chatClient.prompt()
464-
.user(userMessageContent)
465-
.advisors(a -> a
466-
.param(CHAT_MEMORY_CONVERSATION_ID_KEY, chatId)
467-
.param(CHAT_MEMORY_RETRIEVE_SIZE_KEY, 100))
468-
.stream().content();
469-
}
470-
471-
}
472-
----
473-
474-
xref:ROOT:api/retrieval-augmented-generation.adoc#_questionansweradvisor[Learn about Question Answer Advisor]
475-
476557
=== Retrieval Augmented Generation
477558

478559
Refer to the xref:ROOT:api/retrieval-augmented-generation.adoc[Retrieval Augmented Generation] guide.

0 commit comments

Comments
 (0)