You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
# Optionally you can specify additional keyword arguments for the model, e.g. temperature.
53
+
# Optionally you can specify additional keyword arguments for the model, e.g. temperature and default_headers.
53
54
temperature=0.1,
55
+
default_headers={"route": "v1/chat/completions"}, # default route for chat models
54
56
)
55
57
56
58
Completion Models
57
59
=================
58
60
59
-
Completion models takes a text string and input and returns a string with completions. To use completion models, your model should be deployed with the completion endpoint (``/v1/completions``). The following example shows how you can use the ``OCIModelDeploymentVLLM`` class for model deployed with vLLM container. If you deployed the model with TGI container, you can use ``OCIModelDeploymentTGI`` similarly.
61
+
Completion models takes a text string and input and returns a string with completions. To use completion models, your model should be deployed with the completion endpoint (``/v1/completions``).
60
62
61
63
.. code-block:: python3
62
64
63
-
from ads.llm import OCIModelDeploymentVLLM
65
+
from ads.llm import OCIModelDeploymentLLM
64
66
65
-
llm = OCIModelDeploymentVLLM(
66
-
model="odsc-llm",
67
+
llm = OCIModelDeploymentLLM(
68
+
model="odsc-llm", # default model name if deployed on AQUA
# Optionally you can specify additional keyword arguments for the model.
69
71
max_tokens=32,
72
+
default_headers={"route": "v1/completions"}, # default route for completion models
70
73
)
71
74
72
75
# Invoke the LLM. The completion will be a string.
@@ -87,18 +90,19 @@ Completion models takes a text string and input and returns a string with comple
87
90
Chat Models
88
91
===========
89
92
90
-
Chat models takes `chat messages <https://python.langchain.com/docs/concepts/#messages>`_ as inputs and returns additional chat message (usually `AIMessage <https://python.langchain.com/docs/concepts/#aimessage>`_) as output. To use chat models, your models must be deployed with chat completion endpoint (``/v1/chat/completions``). The following example shows how you can use the ``ChatOCIModelDeploymentVLLM`` class for model deployed with vLLM container. If you deployed the model with TGI container, you can use ``ChatOCIModelDeploymentTGI`` similarly.
93
+
Chat models takes `chat messages <https://python.langchain.com/docs/concepts/#messages>`_ as inputs and returns additional chat message (usually `AIMessage <https://python.langchain.com/docs/concepts/#aimessage>`_) as output. To use chat models, your models must be deployed with chat completion endpoint (``/v1/chat/completions``).
91
94
92
95
.. code-block:: python3
93
96
94
97
from langchain_core.messages import HumanMessage, SystemMessage
95
-
from ads.llm import ChatOCIModelDeploymentVLLM
98
+
from ads.llm import ChatOCIModelDeployment
96
99
97
-
llm = ChatOCIModelDeploymentVLLM(
98
-
model="odsc-llm",
99
-
endpoint=f"<oci_model_deployment_url>>/predict",
100
+
llm = ChatOCIModelDeployment(
101
+
model="odsc-llm", # default model name if deployed on AQUA
102
+
endpoint=f"<oci_model_deployment_url>/predict",
100
103
# Optionally you can specify additional keyword arguments for the model.
101
104
max_tokens=32,
105
+
default_headers={"route": "v1/chat/completions"}, # default route for chat models
102
106
)
103
107
104
108
messages = [
@@ -133,7 +137,7 @@ The vLLM container support `tool/function calling <https://docs.vllm.ai/en/lates
133
137
from ads.llm import ChatOCIModelDeploymentVLLM, ChatTemplates
134
138
135
139
llm = ChatOCIModelDeploymentVLLM(
136
-
model="odsc-llm",
140
+
model="odsc-llm", # default model name if deployed on AQUA
Copy file name to clipboardExpand all lines: docs/source/user_guide/model_registration/large_language_model.rst
+10Lines changed: 10 additions & 0 deletions
Original file line number
Diff line number
Diff line change
@@ -2,6 +2,16 @@
2
2
Large Language Model
3
3
====================
4
4
5
+
.. admonition:: Introducing AI Quick Actions
6
+
:class: note
7
+
8
+
Oracle Data Science now has launched `AI Quick Actions <https://docs.oracle.com/en-us/iaas/data-science/using/ai-quick-actions.htm>`_, which makes it easy for you to browse foundation models, and deploy, fine-tune, and evaluate them inside Data Science notebooks.
9
+
10
+
.. admonition:: Use AI Quick Actions
11
+
:class: note
12
+
13
+
The following page is obsolete and please use `AI Quick Actions <https://docs.oracle.com/en-us/iaas/data-science/using/ai-quick-actions.htm>`_ instead.
14
+
5
15
Oracle ADS (Accelerated Data Science) opens the gateway to harnessing the full potential of the Large Language models
6
16
within Oracle Cloud Infrastructure (OCI). `Meta <https://ai.meta.com/resources/models-and-libraries/llama-downloads/>`_'s
7
17
latest offering, `Llama 2 <https://ai.meta.com/llama/>`_, introduces a collection of pre-trained and
0 commit comments