diff --git a/site/en/gemini-api/docs/function-calling/python.ipynb b/site/en/gemini-api/docs/function-calling/python.ipynb index 46614db26..ac332de0e 100644 --- a/site/en/gemini-api/docs/function-calling/python.ipynb +++ b/site/en/gemini-api/docs/function-calling/python.ipynb @@ -11,7 +11,7 @@ }, { "cell_type": "code", - "execution_count": 1, + "execution_count": null, "metadata": { "cellView": "form", "id": "906e07f6e562" @@ -65,7 +65,8 @@ "id": "df1767a3d1cc" }, "source": [ - "You can provide Gemini models with descriptions of functions. The model may ask you to call a function and send back the result to help the model handle your query." + "Use function calling to define custom functions and pass them to Gemini. The model does not directly invoke these functions, but instead generates structured data output that specifies the function name and suggested arguments. This output enables the calling of external APIs, and the resulting API output can then be incorporated back into the model, allowing for more comprehensive query responses. Function calling empowers LLMs to interact with real-time information and various services, such as databases, customer\n", + "relationship management systems, and document repositories, enhancing their ability to provide relevant and contextual answers. You can provide Gemini models with descriptions of functions. The model may ask you to call a function and send back the result to help the model handle your query." ] }, { @@ -119,7 +120,7 @@ }, { "cell_type": "code", - "execution_count": 3, + "execution_count": null, "metadata": { "id": "TS9l5igubpHO" }, @@ -130,7 +131,7 @@ "import time\n", "\n", "import google.generativeai as genai\n", - "\n", + "import google.ai.generativelanguage as glm\n", "\n", "from IPython import display\n", "from IPython.display import Markdown\n", @@ -176,7 +177,7 @@ }, { "cell_type": "code", - "execution_count": 22, + "execution_count": null, "metadata": { "id": "ab9ASynfcIZn" }, @@ -201,23 +202,16 @@ "id": "3f383614ec30" }, "source": [ - "## Function Basics" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "b82c1aecb657" - }, - "source": [ - "You can pass a list of functions to the `tools` argument when creating a `genai.GenerativeModel`.\n", + "## Basics of function calling\n", "\n", - "> Important: The SDK converts the function's argument's type annotations to a format the API understands. The API only supports a limited selection of argument types, and this automatic conversion only supports a subset of that: `int | float | bool | str | list | dict`" + "To use function calling, pass a list of functions to the `tools` parameter when creating a [`GenerativeModel`](https://ai.google.dev/api/python/google/generativeai/GenerativeModel). The model uses the function name, docstring, parameters, and parameter type annotations to decide if it needs the function to best answer a prompt.\n", + "\n", + "> Important: The SDK converts function parameter type annotations to a format the API understands (`glm.FunctionDeclaration`). The API only supports a limited selection of parameter types, and the Python SDK's automatic conversion only supports a subset of that: `AllowedTypes = int | float | bool | str | list['AllowedTypes'] | dict`" ] }, { "cell_type": "code", - "execution_count": 5, + "execution_count": null, "metadata": { "id": "42b27b02d2f5" }, @@ -255,12 +249,12 @@ "id": "d5fd91032a1e" }, "source": [ - "The recomended way to use function calling is through the chat interface. The main reason is that `FunctionCalls` fit nicely into chat's multi-turn structure." + "It is recommended to use function calls through the chat interface. This is because function calls naturally fit in to [multi-turn chats](https://ai.google.dev/api/python/google/generativeai/GenerativeModel#multi-turn) as they capture the back-and-forth interaction between the user and model. The Python SDK's [`ChatSession`](https://ai.google.dev/api/python/google/generativeai/ChatSession) is a great interface for chats because it handles the conversation history for you, and using the parameter `enable_automatic_function_calling` simplifies function calling even further:" ] }, { "cell_type": "code", - "execution_count": 6, + "execution_count": null, "metadata": { "id": "d3b91c855257" }, @@ -282,7 +276,7 @@ }, { "cell_type": "code", - "execution_count": 7, + "execution_count": null, "metadata": { "id": "81d8def3d865" }, @@ -305,7 +299,7 @@ }, { "cell_type": "code", - "execution_count": 8, + "execution_count": null, "metadata": { "id": "951c0f83f72e" }, @@ -328,20 +322,30 @@ { "cell_type": "markdown", "metadata": { - "id": "7731e35f2383" + "id": "J0bgvvIs3I9J" }, "source": [ - "If you look in the `ChatSession.history` you can see the sequence of events:\n", + "Examine the chat history to see the flow of the conversation and how function calls are integrated within it.\n", + "\n", + "The `ChatSession.history` property stores a chronological record of the conversation between the user and the Gemini model. Each turn in the conversation is represented by a [`glm.Content`](https://ai.google.dev/api/python/google/ai/generativelanguage/Content) object, which contains the following information:\n", + "\n", + "* **Role**: Identifies whether the content originated from the \"user\" or the \"model\".\n", + "* **Parts**: A list of [`glm.Part`](https://ai.google.dev/api/python/google/ai/generativelanguage/Part) objects that represent individual components of the message. With a text-only model, these parts can be:\n", + " * **Text**: Plain text messages.\n", + " * **Function Call** ([`glm.FunctionCall`](https://ai.google.dev/api/python/google/ai/generativelanguage/FunctionCall)): A request from the model to execute a specific function with provided arguments.\n", + " * **Function Response** ([`glm.FunctionResponse`](https://ai.google.dev/api/python/google/ai/generativelanguage/FunctionResponse)): The result returned by the user after executing the requested function.\n", "\n", - "1. You sent the question.\n", - "2. The model replied with a `glm.FunctionCall`.\n", - "3. The `genai.ChatSession` executed the function locally and sent the model back a `glm.FunctionResponse`.\n", - "4. The model used the function output in its answer." + " In the previous example with the mittens calculation, the history shows the following sequence:\n", + "\n", + "1. **User**: Asks the question about the total number of mittens.\n", + "1. **Model**: Determines that the multiply function is helpful and sends a FunctionCall request to the user.\n", + "1. **User**: The `ChatSession` automatically executes the function (due to `enable_automatic_function_calling` being set) and sends back a `FunctionResponse` with the calculated result.\n", + "1. **Model**: Uses the function's output to formulate the final answer and presents it as a text response." ] }, { "cell_type": "code", - "execution_count": 9, + "execution_count": null, "metadata": { "id": "9f7eff1e8e60" }, @@ -397,7 +401,152 @@ "While this was all handled automatically, if you need more control, you can:\n", "\n", "- Leave the default `enable_automatic_function_calling=False` and process the `glm.FunctionCall` responses yourself.\n", - "- Or use `GenerativeModel.generate_content`, where you also need to manage the chat history. " + "- Or use `GenerativeModel.generate_content`, where you also need to manage the chat history." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "qiOShqKn1Bh_" + }, + "source": [ + "## Parallel function calling\n", + "\n", + "In addition to basic function calling described above, you can also call multiple functions in a single turn. This section shows an example for how you can use parallel function calling." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "PvHIHmFdTg_c" + }, + "source": [ + "Define the tools." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "89QPizVHTeJa" + }, + "outputs": [], + "source": [ + "def power_disco_ball(power: bool) -> bool:\n", + " \"\"\"Powers the spinning disco ball.\"\"\"\n", + " print(f\"Disco ball is {'spinning!' if power else 'stopped.'}\")\n", + " return True\n", + "\n", + "\n", + "def start_music(energetic: bool, loud: bool, bpm: int) -> str:\n", + " \"\"\"Play some music matching the specified parameters.\n", + "\n", + " Args:\n", + " energetic: Whether the music is energetic or not.\n", + " loud: Whether the music is loud or not.\n", + " bpm: The beats per minute of the music.\n", + "\n", + " Returns: The name of the song being played.\n", + " \"\"\"\n", + " print(f\"Starting music! {energetic=} {loud=}, {bpm=}\")\n", + " return \"Never gonna give you up.\"\n", + "\n", + "\n", + "def dim_lights(brightness: float) -> bool:\n", + " \"\"\"Dim the lights.\n", + "\n", + " Args:\n", + " brightness: The brightness of the lights, 0.0 is off, 1.0 is full.\n", + " \"\"\"\n", + " print(f\"Lights are now set to {brightness:.0%}\")\n", + " return True" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "zlrmXN7fxQi0" + }, + "source": [ + "Now call the model with an instruction that could use all of the specified tools." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "21ecYHLgIsCl" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "power_disco_ball(power=True)\n", + "start_music(energetic=True, loud=True, bpm=120.0)\n", + "dim_lights(brightness=0.3)\n" + ] + } + ], + "source": [ + "# Set the model up with tools.\n", + "house_fns = [power_disco_ball, start_music, dim_lights]\n", + "\n", + "model = genai.GenerativeModel(model_name=\"gemini-1.5-pro-latest\", tools=house_fns)\n", + "\n", + "# Call the API.\n", + "chat = model.start_chat()\n", + "response = chat.send_message(\"Turn this place into a party!\")\n", + "\n", + "# Print out each of the function calls requested from this single call.\n", + "for part in response.parts:\n", + " if fn := part.function_call:\n", + " args = \", \".join(f\"{key}={val}\" for key, val in fn.args.items())\n", + " print(f\"{fn.name}({args})\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "t6iYpty7yZct" + }, + "source": [ + "Each of the printed results reflects a single function call that the model has requested. To send the results back, include the responses in the same order as they were requested." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "L7RxoiR3foBR" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Let's get this party started! I've turned on the disco ball, started playing some upbeat music, and dimmed the lights. 🎢✨ Get ready to dance! πŸ•ΊπŸ’ƒ \n", + "\n", + "\n" + ] + } + ], + "source": [ + "# Simulate the responses from the specified tools.\n", + "responses = {\n", + " \"power_disco_ball\": True,\n", + " \"start_music\": \"Never gonna give you up.\",\n", + " \"dim_lights\": True,\n", + "}\n", + "\n", + "# Build the response parts.\n", + "response_parts = [\n", + " glm.Part(function_response=glm.FunctionResponse(name=fn, response={\"result\": val}))\n", + " for fn, val in responses.items()\n", + "]\n", + "\n", + "response = chat.send_message(response_parts)\n", + "print(response.text)" ] }, { @@ -426,7 +575,7 @@ }, { "cell_type": "code", - "execution_count": 10, + "execution_count": null, "metadata": { "id": "S53E0EE8TBUF" }, @@ -446,7 +595,7 @@ }, { "cell_type": "code", - "execution_count": 11, + "execution_count": null, "metadata": { "id": "e36166b2c1b6" }, @@ -515,7 +664,7 @@ }, { "cell_type": "code", - "execution_count": 12, + "execution_count": null, "metadata": { "id": "qCwHM4WbC4wb" }, @@ -549,7 +698,7 @@ }, { "cell_type": "code", - "execution_count": 13, + "execution_count": null, "metadata": { "id": "5f2804046c94" }, @@ -567,7 +716,7 @@ }, { "cell_type": "code", - "execution_count": 14, + "execution_count": null, "metadata": { "id": "4cefe2c3c808" }, @@ -613,12 +762,12 @@ "id": "jS6ruiTp6VBf" }, "source": [ - "Either way, you pass a representation of a `glm.Tool` or list of tools to " + "Either way, you pass a representation of a `glm.Tool` or list of tools to" ] }, { "cell_type": "code", - "execution_count": 15, + "execution_count": null, "metadata": { "id": "xwhWG22cIIDU" }, @@ -638,12 +787,12 @@ "id": "517ca06297bb" }, "source": [ - "Like before the model returns a `glm.FunctionCall` invoking the calculator's `multiply` function: " + "Like before the model returns a `glm.FunctionCall` invoking the calculator's `multiply` function:" ] }, { "cell_type": "code", - "execution_count": 16, + "execution_count": null, "metadata": { "id": "xhey4QA0DTJf" }, @@ -698,7 +847,7 @@ }, { "cell_type": "code", - "execution_count": 17, + "execution_count": null, "metadata": { "id": "88758eebfd5c" }, @@ -733,7 +882,7 @@ }, { "cell_type": "code", - "execution_count": 18, + "execution_count": null, "metadata": { "id": "f3c67066411e" }, @@ -755,7 +904,7 @@ "source": [ "## Summary\n", "\n", - "Basic function calling is supported in the SDK. Remember that it is easier to manage using chat-mode, because of the natural back and forth structure. You're in charge of actually calling the functions and sending results back to the model so it can produce a text-response. " + "Basic function calling is supported in the SDK. Remember that it is easier to manage using chat-mode, because of the natural back and forth structure. You're in charge of actually calling the functions and sending results back to the model so it can produce a text-response." ] } ], diff --git a/site/en/gemini-api/docs/prompting_with_media.ipynb b/site/en/gemini-api/docs/prompting_with_media.ipynb index 567082162..c04102f8e 100644 --- a/site/en/gemini-api/docs/prompting_with_media.ipynb +++ b/site/en/gemini-api/docs/prompting_with_media.ipynb @@ -64,24 +64,19 @@ "id": "3c5e92a74e64" }, "source": [ - "The Gemini API supports prompting with text, image, and audio data, also known as *multimodal* prompting. You can include text, image,\n", - "and audio in your prompts. For small images, you can point the Gemini model\n", - "directly to a local file when providing a prompt. For larger images, videos\n", - "(sequences of image frames), and audio, upload the files with the [File\n", - "API](https://ai.google.dev/api/rest/v1beta/files) before including them in\n", + "The Gemini API supports *multimodal* prompting with text, image, and audio data. For small files, you can point the Gemini model\n", + "directly to a local file when providing a prompt. Upload larger files with the\n", + "[File API](https://ai.google.dev/api/rest/v1beta/files) before including them in\n", "prompts.\n", "\n", "The File API lets you store up to 20GB of files per project, with each file not\n", "exceeding 2GB in size. Files are stored for 48 hours and can be accessed with\n", - "your API key for generation within that time period. It is available at no cost in all regions where the [Gemini API is\n", + "your API key for generation within that time period and cannot be downloaded from the API. It is available at no cost in all regions where the [Gemini API is\n", "available](https://ai.google.dev/available_regions).\n", "\n", - "For information on valid file formats (MIME types) and supported models, see [Supported file formats](#supported_file_formats).\n", + "The File API handles inputs that can be used to generate content with [`model.generateContent`](https://ai.google.dev/api/rest/v1/models/generateContent) or [`model.streamGenerateContent`](https://ai.google.dev/api/rest/v1/models/streamGenerateContent). For information on valid file formats (MIME types) and supported models, see [Supported file formats](#supported_file_formats).\n", "\n", - "Note: Videos must be converted into image frames before uploading to the File\n", - "API.\n", - "\n", - "This guide shows how to use the File API to upload a media file and include it in a `GenerateContent` call to the Gemini API. For more information, see the [code\n", + "This guide shows how to use the File API to upload media files and include them in a `GenerateContent` call to the Gemini API. For more information, see the [code\n", "samples](https://github.com/google-gemini/gemini-api-cookbook/tree/main/quickstarts/file-api).\n" ] }, @@ -113,7 +108,25 @@ "metadata": { "id": "mN8x8DPgu9Kv" }, - "outputs": [], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Found existing installation: google-ai-generativelanguage 0.6.2\n", + "Uninstalling google-ai-generativelanguage-0.6.2:\n", + " Successfully uninstalled google-ai-generativelanguage-0.6.2\n", + "Found existing installation: google-generativeai 0.5.2\n", + "Uninstalling google-generativeai-0.5.2:\n", + " Successfully uninstalled google-generativeai-0.5.2\n", + " Installing build dependencies ... \u001b[?25l\u001b[?25hdone\n", + " Getting requirements to build wheel ... \u001b[?25l\u001b[?25hdone\n", + " Preparing metadata (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n", + "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m677.8/677.8 kB\u001b[0m \u001b[31m12.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[?25h Building wheel for google-generativeai (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n" + ] + } + ], "source": [ "!pip install -q -U google-generativeai" ] @@ -174,42 +187,48 @@ "id": "c-z4zsCUlaru" }, "source": [ - "## Upload a file to the File API\n", - "\n", - "The File API lets you upload a variety of multimodal MIME types, including images and audio formats. The File API handles inputs that can be used to generate content with [`model.generateContent`](https://ai.google.dev/api/rest/v1/models/generateContent) or [`model.streamGenerateContent`](https://ai.google.dev/api/rest/v1/models/streamGenerateContent).\n", - "\n", - "The File API accepts files under 2GB in size and can store up to 20GB of files per project. Files last for 2 days and cannot be downloaded from the API." + "## Prompting with images\n" ] }, { "cell_type": "markdown", "metadata": { - "id": "o1K81yn9mFBo" + "id": "rsdNkDszLBmQ" }, "source": [ - "First, you will prepare a sample image to upload to the API.\n", - "\n", - "To upload your own file, see the [Appendix section](#uploading_files_to_colab)." + "### Upload an image file to the File API" ] }, { - "cell_type": "code", - "execution_count": null, + "cell_type": "markdown", "metadata": { - "id": "lC6sS6DnmGmi" + "id": "o1K81yn9mFBo" }, - "outputs": [], "source": [ - "!curl -o image.jpg https://storage.googleapis.com/generativeai-downloads/images/jetpack.jpg" + "In this tutorial, you upload a sample image to the API and use it to generate content.\n", + "\n", + "Refer to the [Appendix section](#uploading_files_to_colab) to learn how to upload your own file." ] }, { - "cell_type": "markdown", + "cell_type": "code", + "execution_count": null, "metadata": { - "id": "rI84z01ZmSyF" + "id": "lC6sS6DnmGmi" }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + " % Total % Received % Xferd Average Speed Time Time Time Current\n", + " Dload Upload Total Spent Left Speed\n", + "100 349k 100 349k 0 0 1290k 0 --:--:-- --:--:-- --:--:-- 1293k\n" + ] + } + ], "source": [ - "Next, you'll upload that file to the File API." + "!curl -o image.jpg https://storage.googleapis.com/generativeai-downloads/images/jetpack.jpg" ] }, { @@ -223,11 +242,12 @@ "name": "stdout", "output_type": "stream", "text": [ - "Uploaded file 'Sample drawing' as: https://generativelanguage.googleapis.com/v1beta/files/ui00j5zfuqe0\n" + "Uploaded file 'Sample drawing' as: https://generativelanguage.googleapis.com/v1beta/files/o42pw1xtkpej\n" ] } ], "source": [ + "# Upload the file\n", "sample_file = genai.upload_file(path=\"image.jpg\",\n", " display_name=\"Sample drawing\")\n", "\n", @@ -242,7 +262,7 @@ "source": [ "The `response` shows that the File API stored the specified `display_name` for the uploaded file and a `uri` to reference the file in Gemini API calls. Use `response` to track how uploaded files are mapped to URIs.\n", "\n", - "Depending on your use cases, you could store the URIs in structures such as a `dict` or a database." + "Depending on your use case, you can also store the URIs in structures such as a `dict` or a database." ] }, { @@ -251,11 +271,11 @@ "id": "ds5iJlaembWe" }, "source": [ - "## Get file\n", + "### Get file\n", "\n", - "After uploading the file, you can verify the API has successfully received the files by calling `files.get`.\n", + "After uploading the file, verify that the File API has successfully received it by calling `files.get`.\n", "\n", - "It lets you get the file metadata that have been uploaded to the File API that are associated with the Cloud project your API key belongs to. Only the `name` (and by extension, the `uri`) are unique. Only use the `displayName` to identify files if you manage uniqueness yourself." + "`files.get` lets you get the file metadata that have been uploaded to the File API that are associated with the Cloud project your API key belongs to. Only the `name` (and by extension, the `uri`) are unique. Only use the `displayName` to identify files if you manage uniqueness yourself." ] }, { @@ -264,7 +284,15 @@ "metadata": { "id": "kLFsVLFHOWSV" }, - "outputs": [], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Retrieved file 'Sample drawing' as: https://generativelanguage.googleapis.com/v1beta/files/o42pw1xtkpej\n" + ] + } + ], "source": [ "file = genai.get_file(name=sample_file.name)\n", "print(f\"Retrieved file '{file.display_name}' as: {sample_file.uri}\")" @@ -276,9 +304,9 @@ "id": "EPPOECHzsIGJ" }, "source": [ - "## Generate content\n", + "### Generate content\n", "\n", - "After uploading the file, you can make `GenerateContent` requests that reference the File API URI. In this example, you create prompt that starts with a text followed by the uploaded image." + "After uploading the file, you can make `GenerateContent` requests that reference the File API URI. In this example, you create a prompt that starts with a text followed by the uploaded image." ] }, { @@ -287,7 +315,19 @@ "metadata": { "id": "ZYVFqmLkl5nE" }, - "outputs": [], + "outputs": [ + { + "data": { + "text/markdown": ">Scrawled on this lined paper is the blueprint for every overworked student's dream: the Jetpack Backpack. A deceptively normal-looking backpack, perhaps a tad bulky to fit an 18\" laptop, hides a secret. Retractable boosters, fueled by clean steam power, lie dormant at its base, ready to whisk you away from the mundane on a 15-minute flight of fancy. No more frantic dashes to class, no more groaning under the weight of textbooks – just the wind in your hair and the freedom of the skies, courtesy of padded straps and ingenious engineering. \n", + "text/plain": [ + "" + ] + }, + "execution_count": 8, + "metadata": {}, + "output_type": "execute_result" + } + ], "source": [ "# Set the model to Gemini 1.5 Pro.\n", "model = genai.GenerativeModel(model_name=\"models/gemini-1.5-pro-latest\")\n", @@ -303,7 +343,7 @@ "id": "IrPDYdQSKTg4" }, "source": [ - "## Delete files\n", + "### Delete files\n", "\n", "Files are automatically deleted after 2 days. You can also manually delete them using `files.delete()`." ] @@ -314,12 +354,192 @@ "metadata": { "id": "d4eO8ZXoKdZf" }, - "outputs": [], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Deleted Sample drawing.\n" + ] + } + ], "source": [ "genai.delete_file(sample_file.name)\n", "print(f'Deleted {sample_file.display_name}.')" ] }, + { + "cell_type": "markdown", + "metadata": { + "id": "TaUZc1mvLkHY" + }, + "source": [ + "## Prompting with videos" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "MNvhBdoDFnTC" + }, + "source": [ + "### Upload a video file to the File API\n", + "\n", + "The File API accepts video file formats directly. This example uses the short film \"Big Buck Bunny\".\n", + "\n", + "> \"Big Buck Bunny\" is (c) copyright 2008, Blender Foundation / www.bigbuckbunny.org and [licensed](https://peach.blender.org/about/) under the [Creative Commons Attribution 3.0](http://creativecommons.org/licenses/by/3.0/) License.\n", + "\n", + "Refer to the [Appendix section](#uploading_files_to_colab) to learn how to upload your own file." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "V4XeFdX1rxaE" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "--2024-05-14 00:48:30-- https://download.blender.org/peach/bigbuckbunny_movies/BigBuckBunny_320x180.mp4\n", + "Resolving download.blender.org (download.blender.org)... 104.22.65.163, 172.67.14.163, 104.22.64.163, ...\n", + "Connecting to download.blender.org (download.blender.org)|104.22.65.163|:443... connected.\n", + "HTTP request sent, awaiting response... 200 OK\n", + "Length: 64657027 (62M) [video/mp4]\n", + "Saving to: β€˜BigBuckBunny_320x180.mp4’\n", + "\n", + "BigBuckBunny_320x18 100%[===================>] 61.66M 11.4MB/s in 4.7s \n", + "\n", + "2024-05-14 00:48:35 (13.2 MB/s) - β€˜BigBuckBunny_320x180.mp4’ saved [64657027/64657027]\n", + "\n" + ] + } + ], + "source": [ + "!wget https://download.blender.org/peach/bigbuckbunny_movies/BigBuckBunny_320x180.mp4" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "_HzrDdp2Q1Cu" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Uploading file...\n", + "Completed upload: https://generativelanguage.googleapis.com/v1beta/files/gb4jshu3zzdy\n" + ] + } + ], + "source": [ + "video_file_name = \"BigBuckBunny_320x180.mp4\"\n", + "\n", + "print(f\"Uploading file...\")\n", + "video_file = genai.upload_file(path=video_file_name)\n", + "print(f\"Completed upload: {video_file.uri}\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "oOZmTUb4FWOa" + }, + "source": [ + "### Get file\n", + "\n", + "Verify the API has successfully received the files by calling the `files.get` method.\n", + "\n", + "NOTE: Video files have a `State` field in the File API. When a video is uploaded, it will be in `PROCESSING` state until it is ready for inference. Only `ACTIVE` files can be used for model inference." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "SHMVCWHkFhJW" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + ".." + ] + } + ], + "source": [ + "import time\n", + "\n", + "while video_file.state.name == \"PROCESSING\":\n", + " print('.', end='')\n", + " time.sleep(10)\n", + " video_file = genai.get_file(video_file.name)\n", + "\n", + "if video_file.state.name == \"FAILED\":\n", + " raise ValueError(video_file.state.name)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "zS5NmQeXLqeS" + }, + "source": [ + "### Generate content\n", + "\n", + "After the video has been uploaded, you can make `GenerateContent` requests that reference the File API URI." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "ypZuGQ-2LqeS" + }, + "outputs": [], + "source": [ + "# Create the prompt.\n", + "prompt = \"Describe this video.\"\n", + "\n", + "# Set the model to Gemini 1.5 Pro.\n", + "model = genai.GenerativeModel(model_name=\"models/gemini-1.5-pro-latest\")\n", + "\n", + "# Make the LLM request.\n", + "print(\"Making LLM inference request...\")\n", + "response = model.generate_content([video_file, prompt],\n", + " request_options={\"timeout\": 600})\n", + "print(response.text)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "diCy9BgjLqeS" + }, + "source": [ + "### Delete files\n", + "\n", + "Files are automatically deleted after 2 days or you can manually delete them using `files.delete()`." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "YYyi5PrKLqeb" + }, + "outputs": [], + "source": [ + "genai.delete_file(video_file.name)\n", + "print(f'Deleted file {video_file.uri}')" + ] + }, { "cell_type": "markdown", "metadata": { @@ -328,7 +548,7 @@ "source": [ "## Supported file formats\n", "\n", - "Gemini models support prompting with multiple file formats. This section explains considerations in using general media formats for prompting, specifically image, audio, and video files. You can use media files for prompting only with specific model versions, as shown in the following table.\n", + "Gemini models support prompting with multiple file formats. This section explains considerations in using general media formats for prompting, specifically image, audio, video, and plain text files. You can use media files for prompting only with specific model versions, as shown in the following table.\n", "\n", "\n", " \n", @@ -337,6 +557,7 @@ " \n", " \n", " \n", + " \n", " \n", " \n", " \n", @@ -345,12 +566,14 @@ " \n", " \n", " \n", + " \n", " \n", " \n", " \n", " \n", " \n", " \n", + " \n", " \n", " \n", "
ImagesAudioVideoPlain text
βœ” (3600 max image files)βœ”βœ”βœ”
Gemini Pro Visionβœ” (16 max image files)βœ”
\n", @@ -385,15 +608,48 @@ "\n", "### Video formats\n", "\n", - "You can use video data for prompting with the `gemini-1.5-pro` model. However, video file formats are not supported as direct inputs by the Gemini API. You can use video data as prompt input by breaking down the video into a series of still frame images and a separate audio file. This approach lets you manage the amount of data, and the level of detail provided by the video, by choosing how many frames per second are included in your prompt from the video file.\n", - "\n", - "Note: Video files added to a prompt as constituent parts, audio file and image frames, are considered as separate prompt data inputs by the model. For this reason, requests or questions that specify the time when *both* an audio snippet and video frames appear in the source video may not produce useful results." + "You can use video data for prompting with the `gemini-1.5-pro` model.\n", + "\n", + "- Video data is supported in the following common audio format [MIME types](https://developers.google.com/drive/api/guides/ref-export-formats):\n", + " - video/mp4\n", + " - video/mpeg\n", + " - video/mov\n", + " - video/avi\n", + " - video/x-flv\n", + " - video/mpg\n", + " - video/webm\n", + " - video/wmv\n", + " - video/3gpp\n", + "\n", + "- The File API service currently samples videos into images at 1 frame per second (FPS) and may be subject to change to provide the best inference quality. Individual images take up 258 tokens **regardless** of resolution and quality.\n", + "\n", + "### Plain text formats\n", + "\n", + "The File API supports uploading plain text files with the following [MIME types](https://developers.google.com/drive/api/guides/ref-export-formats):\n", + "- text/plain\n", + "- text/html\n", + "- text/css\n", + "- text/javascript\n", + "- application/x-javascript\n", + "- text/x-typescript\n", + "- application/x-typescript\n", + "- text/csv\n", + "- text/markdown\n", + "- text/x-python\n", + "- application/x-python-code\n", + "- application/json\n", + "- text/xml\n", + "- application/rtf\n", + "- text/rtf\n", + "\n", + "For plain text files with a MIME type not on the list, you can try specifying\n", + "one of the above MIME types manually." ] }, { "cell_type": "markdown", "metadata": { - "id": "uploading_files_to_colab" + "id": "rIoNRWn0nwUy" }, "source": [ "## Appendix: Uploading files to Colab\n", @@ -411,7 +667,7 @@ "cell_type": "code", "execution_count": null, "metadata": { - "id": "2QGBpboMmxHG" + "id": "VqAwyEa3nxaZ" }, "outputs": [], "source": [ diff --git a/site/en/gemini-api/tutorials/extract_structured_data.ipynb b/site/en/gemini-api/tutorials/extract_structured_data.ipynb index 214dcbca3..3b155a210 100644 --- a/site/en/gemini-api/tutorials/extract_structured_data.ipynb +++ b/site/en/gemini-api/tutorials/extract_structured_data.ipynb @@ -90,12 +90,13 @@ }, { "cell_type": "code", - "execution_count": 3, + "execution_count": 1, "metadata": { "id": "TS9l5igubpHO" }, "outputs": [], "source": [ + "import json\n", "import pathlib\n", "import textwrap\n", "\n", @@ -127,7 +128,7 @@ }, { "cell_type": "code", - "execution_count": 4, + "execution_count": 2, "metadata": { "id": "ab9ASynfcIZn" }, @@ -139,11 +140,9 @@ "\n", " # Or use `os.getenv('API_KEY')` to fetch an environment variable.\n", " GOOGLE_API_KEY=userdata.get('GOOGLE_API_KEY')\n", + " genai.configure(api_key=GOOGLE_API_KEY)\n", "except ImportError:\n", - " import os\n", - " GOOGLE_API_KEY = os.environ['GOOGLE_API_KEY']\n", - "\n", - "genai.configure(api_key=GOOGLE_API_KEY)" + " pass\n" ] }, { @@ -160,7 +159,7 @@ }, { "cell_type": "code", - "execution_count": 5, + "execution_count": 3, "metadata": { "id": "0THz95wOL4pL" }, @@ -184,7 +183,7 @@ }, { "cell_type": "code", - "execution_count": 6, + "execution_count": 4, "metadata": { "id": "yMnxJqubg759" }, @@ -226,7 +225,7 @@ "" ] }, - "execution_count": 6, + "execution_count": 4, "metadata": {}, "output_type": "execute_result" } @@ -255,7 +254,7 @@ }, { "cell_type": "code", - "execution_count": 7, + "execution_count": 5, "metadata": { "id": "eStTMD6VL4pM" }, @@ -288,21 +287,18 @@ }, { "cell_type": "code", - "execution_count": 8, + "execution_count": 6, "metadata": { "id": "B0b5zHI3uEBm" }, "outputs": [ { "data": { - "application/vnd.google.colaboratory.intrinsic+json": { - "type": "string" - }, "text/plain": [ - "'{\"people\": [\\n {\\n \"name\": \"Anya\",\\n \"description\": \"A young girl who lives in the town of Willow Creek with her parents, Elise and Edward. She possesses a magical backpack that was handed down to her from her grandmother.\",\\n \"start_place_name\": \"Willow Creek\",\\n \"end_place_name\": \"Willow Creek\"\\n },\\n {\\n \"name\": \"Elise\",\\n \"description\": \"Anya\\'s kind-hearted mother\",\\n \"start_place_name\": \"Willow Creek\",\\n \"end_place_name\": \"Willow Creek\"\\n },\\n {\\n \"name\": \"Edward\",\\n \"description\": \"Anya\\'s wise-bearded father\",\\n \"start_place_name\": \"Willow Creek\",\\n \"end_place_name\": \"Willow Creek\"\\n },\\n {\\n \"name\": \"Samuel\",\\n \"description\": \"Anya\\'s best friend, a curious and adventurous boy with a mischievous grin.\",\\n \"start_place_name\": \"Willow Creek\",\\n \"end_place_name\": \"Willow Creek\"\\n },\\n {\\n \"name\": \"Monster\",\\n \"description\": \"A massive beast with sharp teeth, glowing red eyes, and claws that could crush a human with ease.\",\\n \"start_place_name\": \"Forest\",\\n \"end_place_name\": \"Forest\"\\n }\\n], \"places\": [\\n {\\n \"name\": \"Willow Creek\",\\n \"description\": \"A quaint town nestled amidst rolling hills and whispering willows.\"\\n },\\n {\\n \"name\": \"Forest\",\\n \"description\": \"A shadowy place with rustling undergrowth and whispering trees.\"\\n },\\n {\\n \"name\": \"Schoolhouse\",\\n \"description\": \"The only school in the town of Willow Creek.\"\\n },\\n {\\n \"name\": \"Anya\\'s home\",\\n \"description\": \"A modest cottage with a creaky wooden door.\"\\n }\\n], \"things\": [\\n {\\n \"name\": \"Magic backpack\",\\n \"description\": \"A magical backpack that was handed down to Anya from her grandmother. Its soft, emerald-green fabric shimmered with an ethereal glow, and its leather straps held secrets that only Anya knew.\",\\n \"start_place_name\": \"Anya\\'s home\",\\n \"end_place_name\": \"Forest\"\\n },\\n {\\n \"name\": \"Shimmering sword\",\\n \"description\": \"A sword that shimmered in the sunlight and could strike with blinding light.\",\\n \"start_place_name\": \"Magic backpack\",\\n \"end_place_name\": \"Forest\"\\n },\\n {\\n \"name\": \"Book of ancient spells\",\\n \"description\": \"A book that contained ancient spells.\",\\n \"start_place_name\": \"Magic backpack\",\\n \"end_place_name\": \"Forest\"\\n },\\n {\\n \"name\": \"Tiny compass\",\\n \"description\": \"A compass that always pointed north.\",\\n \"start_place_name\": \"Magic backpack\",\\n \"end_place_name\": \"Forest\"\\n },\\n {\\n \"name\": \"Magical key\",\\n \"description\": \"A key that could open any lock.\",\\n \"start_place_name\": \"Magic backpack\",\\n \"end_place_name\": \"Forest\"\\n },\\n {\\n \"name\": \"Shattered crystals\",\\n \"description\": \"The remains of the monster after it was defeated by Anya\\'s magic backpack.\",\\n \"start_place_name\": \"Forest\",\\n \"end_place_name\": \"Forest\"\\n }\\n], \"relationships\": [\\n {\\n \"person_1_name\": \"Anya\",\\n \"person_2_name\": \"Elise\",\\n \"relationship\": \"mother-daughter\"\\n },\\n {\\n \"person_1_name\": \"Anya\",\\n \"person_2_name\": \"Edward\",\\n \"relationship\": \"father-daughter\"\\n },\\n {\\n \"person_1_name\": \"Anya\",\\n \"person_2_name\": \"Samuel\",\\n \"relationship\": \"best friends\"\\n }\\n]}'" + "'{\"people\":[{\"name\":\"Anya\",\"description\":\"A young girl with a magical backpack\",\"start_place_name\":\"Willow Creek\",\"end_place_name\":\"Willow Creek\"},{\"name\":\"Elise\",\"description\":\"Anya\\'s kind-hearted mother\",\"start_place_name\":\"Willow Creek\",\"end_place_name\":\"Willow Creek\"},{\"name\":\"Edward\",\"description\":\"Anya\\'s wise-bearded father\",\"start_place_name\":\"Willow Creek\",\"end_place_name\":\"Willow Creek\"},{\"name\":\"Samuel\",\"description\":\"Anya\\'s curious and adventurous best friend\",\"start_place_name\":\"Willow Creek\",\"end_place_name\":\"Willow Creek\"}],\"places\":[{\"name\":\"Willow Creek\",\"description\":\"A quaint town nestled amidst rolling hills and whispering willows\"},{\"name\":\"Anya\\'s cottage\",\"description\":\"A modest cottage in Willow Creek\"},{\"name\":\"Schoolhouse\",\"description\":\"The only schoolhouse in Willow Creek\"},{\"name\":\"Forest\",\"description\":\"A shadowy forest near Willow Creek\"}],\"things\":[{\"name\":\"Magical backpack\",\"description\":\"A shimmering emerald-green backpack with enchanted contents\",\"start_place_name\":\"Anya\\'s cottage\",\"end_place_name\":\"Forest\"},{\"name\":\"Shimmering sword\",\"description\":\"A sword found inside the backpack\",\"start_place_name\":\"Anya\\'s cottage\",\"end_place_name\":\"Forest\"},{\"name\":\"Book of ancient spells\",\"description\":\"A book found inside the backpack\",\"start_place_name\":\"Anya\\'s cottage\",\"end_place_name\":\"Forest\"},{\"name\":\"Tiny compass\",\"description\":\"A compass found inside the backpack\",\"start_place_name\":\"Anya\\'s cottage\",\"end_place_name\":\"Forest\"},{\"name\":\"Magical key\",\"description\":\"A key found inside the backpack\",\"start_place_name\":\"Anya\\'s cottage\",\"end_place_name\":\"Forest\"}],\"relationships\":[{\"person_1_name\":\"Anya\",\"person_2_name\":\"Elise\",\"relationship\":\"Mother-daughter\"},{\"person_1_name\":\"Anya\",\"person_2_name\":\"Edward\",\"relationship\":\"Father-daughter\"},{\"person_1_name\":\"Anya\",\"person_2_name\":\"Samuel\",\"relationship\":\"Best friends\"}]}\\n'" ] }, - "execution_count": 8, + "execution_count": 6, "metadata": {}, "output_type": "execute_result" } @@ -322,7 +318,7 @@ }, { "cell_type": "code", - "execution_count": 9, + "execution_count": 7, "metadata": { "id": "xSdj50czL4pM" }, @@ -335,7 +331,7 @@ " \"people\": [\n", " {\n", " \"name\": \"Anya\",\n", - " \"description\": \"A young girl who lives in the town of Willow Creek with her parents, Elise and Edward. She possesses a magical backpack that was handed down to her from her grandmother.\",\n", + " \"description\": \"A young girl with a magical backpack\",\n", " \"start_place_name\": \"Willow Creek\",\n", " \"end_place_name\": \"Willow Creek\"\n", " },\n", @@ -353,70 +349,58 @@ " },\n", " {\n", " \"name\": \"Samuel\",\n", - " \"description\": \"Anya's best friend, a curious and adventurous boy with a mischievous grin.\",\n", + " \"description\": \"Anya's curious and adventurous best friend\",\n", " \"start_place_name\": \"Willow Creek\",\n", " \"end_place_name\": \"Willow Creek\"\n", - " },\n", - " {\n", - " \"name\": \"Monster\",\n", - " \"description\": \"A massive beast with sharp teeth, glowing red eyes, and claws that could crush a human with ease.\",\n", - " \"start_place_name\": \"Forest\",\n", - " \"end_place_name\": \"Forest\"\n", " }\n", " ],\n", " \"places\": [\n", " {\n", " \"name\": \"Willow Creek\",\n", - " \"description\": \"A quaint town nestled amidst rolling hills and whispering willows.\"\n", + " \"description\": \"A quaint town nestled amidst rolling hills and whispering willows\"\n", " },\n", " {\n", - " \"name\": \"Forest\",\n", - " \"description\": \"A shadowy place with rustling undergrowth and whispering trees.\"\n", + " \"name\": \"Anya's cottage\",\n", + " \"description\": \"A modest cottage in Willow Creek\"\n", " },\n", " {\n", " \"name\": \"Schoolhouse\",\n", - " \"description\": \"The only school in the town of Willow Creek.\"\n", + " \"description\": \"The only schoolhouse in Willow Creek\"\n", " },\n", " {\n", - " \"name\": \"Anya's home\",\n", - " \"description\": \"A modest cottage with a creaky wooden door.\"\n", + " \"name\": \"Forest\",\n", + " \"description\": \"A shadowy forest near Willow Creek\"\n", " }\n", " ],\n", " \"things\": [\n", " {\n", - " \"name\": \"Magic backpack\",\n", - " \"description\": \"A magical backpack that was handed down to Anya from her grandmother. Its soft, emerald-green fabric shimmered with an ethereal glow, and its leather straps held secrets that only Anya knew.\",\n", - " \"start_place_name\": \"Anya's home\",\n", + " \"name\": \"Magical backpack\",\n", + " \"description\": \"A shimmering emerald-green backpack with enchanted contents\",\n", + " \"start_place_name\": \"Anya's cottage\",\n", " \"end_place_name\": \"Forest\"\n", " },\n", " {\n", " \"name\": \"Shimmering sword\",\n", - " \"description\": \"A sword that shimmered in the sunlight and could strike with blinding light.\",\n", - " \"start_place_name\": \"Magic backpack\",\n", + " \"description\": \"A sword found inside the backpack\",\n", + " \"start_place_name\": \"Anya's cottage\",\n", " \"end_place_name\": \"Forest\"\n", " },\n", " {\n", " \"name\": \"Book of ancient spells\",\n", - " \"description\": \"A book that contained ancient spells.\",\n", - " \"start_place_name\": \"Magic backpack\",\n", + " \"description\": \"A book found inside the backpack\",\n", + " \"start_place_name\": \"Anya's cottage\",\n", " \"end_place_name\": \"Forest\"\n", " },\n", " {\n", " \"name\": \"Tiny compass\",\n", - " \"description\": \"A compass that always pointed north.\",\n", - " \"start_place_name\": \"Magic backpack\",\n", + " \"description\": \"A compass found inside the backpack\",\n", + " \"start_place_name\": \"Anya's cottage\",\n", " \"end_place_name\": \"Forest\"\n", " },\n", " {\n", " \"name\": \"Magical key\",\n", - " \"description\": \"A key that could open any lock.\",\n", - " \"start_place_name\": \"Magic backpack\",\n", - " \"end_place_name\": \"Forest\"\n", - " },\n", - " {\n", - " \"name\": \"Shattered crystals\",\n", - " \"description\": \"The remains of the monster after it was defeated by Anya's magic backpack.\",\n", - " \"start_place_name\": \"Forest\",\n", + " \"description\": \"A key found inside the backpack\",\n", + " \"start_place_name\": \"Anya's cottage\",\n", " \"end_place_name\": \"Forest\"\n", " }\n", " ],\n", @@ -424,17 +408,17 @@ " {\n", " \"person_1_name\": \"Anya\",\n", " \"person_2_name\": \"Elise\",\n", - " \"relationship\": \"mother-daughter\"\n", + " \"relationship\": \"Mother-daughter\"\n", " },\n", " {\n", " \"person_1_name\": \"Anya\",\n", " \"person_2_name\": \"Edward\",\n", - " \"relationship\": \"father-daughter\"\n", + " \"relationship\": \"Father-daughter\"\n", " },\n", " {\n", " \"person_1_name\": \"Anya\",\n", " \"person_2_name\": \"Samuel\",\n", - " \"relationship\": \"best friends\"\n", + " \"relationship\": \"Best friends\"\n", " }\n", " ]\n", "}\n" @@ -442,8 +426,6 @@ } ], "source": [ - "import json\n", - "\n", "print(json.dumps(json.loads(response.text), indent=4))" ] }, @@ -489,45 +471,28 @@ }, { "cell_type": "code", - "execution_count": 10, + "execution_count": 8, "metadata": { - "id": "p2efqZA7BAzp" + "id": "d6293fed386a" }, "outputs": [], "source": [ - "person = glm.Schema(\n", - " type = glm.Type.OBJECT,\n", - " properties = {\n", - " 'name': glm.Schema(type=glm.Type.STRING),\n", - " 'description': glm.Schema(type=glm.Type.STRING),\n", - " 'start_place_name': glm.Schema(type=glm.Type.STRING),\n", - " 'end_place_name': glm.Schema(type=glm.Type.STRING)\n", - " },\n", - " required=['name', 'description', 'start_place_name', 'end_place_name']\n", - ")" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "HGV1wxx6BCJl" - }, - "source": [ - "Then define people as an `ARRAY` of `person` objects:" + "from typing_extensions import TypedDict" ] }, { "cell_type": "code", - "execution_count": 11, + "execution_count": 9, "metadata": { - "id": "Ur7kzpiA_Dqw" + "id": "e48ace2a3ded" }, "outputs": [], "source": [ - "people = glm.Schema(\n", - " type=glm.Type.ARRAY,\n", - " items=person\n", - ")" + "class Person(TypedDict):\n", + " name: str\n", + " description: str\n", + " start_place_name: str\n", + " end_place_name: str" ] }, { @@ -541,69 +506,43 @@ }, { "cell_type": "code", - "execution_count": 12, + "execution_count": 10, "metadata": { "id": "7wd3jTqj_bVi" }, "outputs": [], "source": [ - "place = glm.Schema(\n", - " type = glm.Type.OBJECT,\n", - " properties = {\n", - " 'name': glm.Schema(type=glm.Type.STRING),\n", - " 'description': glm.Schema(type=glm.Type.STRING),\n", - " }\n", - ")\n", - "\n", - "places = glm.Schema(\n", - " type=glm.Type.ARRAY,\n", - " items=place\n", - ")" + "class Place(TypedDict):\n", + " name:str\n", + " description:str" ] }, { "cell_type": "code", - "execution_count": 13, + "execution_count": 11, "metadata": { "id": "45cLwvCd_vg_" }, "outputs": [], "source": [ - "thing = glm.Schema(\n", - " type = glm.Type.OBJECT,\n", - " properties = {\n", - " 'name': glm.Schema(type=glm.Type.STRING),\n", - " 'description': glm.Schema(type=glm.Type.STRING),\n", - " }\n", - ")\n", - "\n", - "things = glm.Schema(\n", - " type=glm.Type.ARRAY,\n", - " items=thing\n", - ")" + "class Thing(TypedDict):\n", + " name:str\n", + " description:str\n", + " " ] }, { "cell_type": "code", - "execution_count": 14, + "execution_count": 12, "metadata": { "id": "8DdVSZJfADDY" }, "outputs": [], "source": [ - "relationship = glm.Schema(\n", - " type = glm.Type.OBJECT,\n", - " properties = {\n", - " 'person_1_name': glm.Schema(type=glm.Type.STRING),\n", - " 'person_2_name': glm.Schema(type=glm.Type.STRING),\n", - " 'relationship': glm.Schema(type=glm.Type.STRING),\n", - " }\n", - ")\n", - "\n", - "relationships = glm.Schema(\n", - " type=glm.Type.ARRAY,\n", - " items=relationship\n", - ")" + "class Relationship(TypedDict):\n", + " person_1_name: str\n", + " person_2_name: str\n", + " relationship: str" ] }, { @@ -617,27 +556,19 @@ }, { "cell_type": "code", - "execution_count": 15, + "execution_count": 13, "metadata": { - "id": "YQkiVCtsPbUy" + "id": "b1fe9734d8c3" }, "outputs": [], "source": [ - "add_to_database = glm.FunctionDeclaration(\n", - " name=\"add_to_database\",\n", - " description=textwrap.dedent(\"\"\"\\\n", - " Adds entities to the database.\n", - " \"\"\"),\n", - " parameters=glm.Schema(\n", - " type=glm.Type.OBJECT,\n", - " properties = {\n", - " 'people': people,\n", - " 'places': places,\n", - " 'things': things,\n", - " 'relationships': relationships\n", - " }\n", - " )\n", - ")" + "def add_to_database(\n", + " people: list[Person],\n", + " places: list[Place],\n", + " things: list[Thing],\n", + " relationships: list[Relationship]\n", + "):\n", + " pass" ] }, { @@ -653,13 +584,13 @@ }, { "cell_type": "code", - "execution_count": 16, + "execution_count": 14, "metadata": { "id": "5PGAPRDJP4Qx" }, "outputs": [], "source": [ - "model = model = genai.GenerativeModel(\n", + "model = genai.GenerativeModel(\n", " model_name='models/gemini-1.5-pro-latest',\n", " tools = [add_to_database])" ] @@ -682,12 +613,13 @@ "outputs": [], "source": [ "result = model.generate_content(f\"\"\"\n", - "Please add the people, places, things, and relationships from this story to the database:\n", - "\n", "{story}\n", + "\n", + "Please add the people, places, things, and relationships from this story to the database:\n", "\"\"\",\n", "# Force a function call\n", - "tool_config={'function_calling_config':'ANY'})" + "# tool_config={'function_calling_config':'ANY'}\n", + ")" ] }, { @@ -786,83 +718,67 @@ "{\n", " \"name\": \"add_to_database\",\n", " \"args\": {\n", - " \"things\": [\n", - " {\n", - " \"name\": \"Magical Backpack\",\n", - " \"description\": \"Anya's prized possession, the Magical Backpack, is no ordinary satchel. Its soft, emerald-green fabric shimmers with an ethereal glow, and its leather straps have secrets that only Anya knows. Within its capacious interior lay an enchanted world, filled with wonders that would ignite her imagination and change her life forever.\"\n", - " },\n", + " \"people\": [\n", " {\n", - " \"name\": \"Shimmering Sword\",\n", - " \"description\": \"Among the wonders in Anya's Magical Backpack, lies a shimmering sword. With a determined gleam in her eye, she retrieved the shimmering sword and charged towards the monster.\"\n", + " \"description\": \"A young girl with a magical backpack.\",\n", + " \"name\": \"Anya\"\n", " },\n", " {\n", - " \"description\": \"Residing within the Magical Backpack, the Book of Ancient Spells holds secrets untold.\",\n", - " \"name\": \"Book of Ancient Spells\"\n", + " \"description\": \"Anya\\\\'s kind-hearted mother.\",\n", + " \"name\": \"Elise\"\n", " },\n", " {\n", - " \"description\": \"Tucked away in the Magical Backpack is a tiny compass that always points north.\",\n", - " \"name\": \"Tiny Compass that Always Points North\"\n", + " \"description\": \"Anya\\\\'s wise-bearded father.\",\n", + " \"name\": \"Edward\"\n", " },\n", " {\n", - " \"description\": \"Hidden within the Magical Backpack is a magical key that can open any lock.\",\n", - " \"name\": \"Magical Key that Can Open Any Lock\"\n", + " \"description\": \"Anya\\\\'s curious and adventurous best friend.\",\n", + " \"name\": \"Samuel\"\n", " }\n", " ],\n", - " \"relationships\": [\n", - " {\n", - " \"relationship\": \"Mother-Daughter\",\n", - " \"person_1_name\": \"Anya\",\n", - " \"person_2_name\": \"Elise\"\n", - " },\n", + " \"things\": [\n", " {\n", - " \"person_2_name\": \"Edward\",\n", - " \"relationship\": \"Father-Daughter\",\n", - " \"person_1_name\": \"Anya\"\n", + " \"description\": \"A backpack with magical contents, including a shimmering sword, a book of ancient spells, a tiny compass, and a magical key.\",\n", + " \"name\": \"Magical backpack\"\n", " },\n", " {\n", - " \"person_2_name\": \"Samuel\",\n", - " \"person_1_name\": \"Anya\",\n", - " \"relationship\": \"Best Friends\"\n", + " \"description\": \"A sword that emits a blinding light when it strikes.\",\n", + " \"name\": \"Shimmering sword\"\n", " }\n", " ],\n", - " \"people\": [\n", + " \"places\": [\n", " {\n", - " \"name\": \"Anya\",\n", - " \"description\": \"Anya, the main character of the story, is a young girl with a magical backpack.\",\n", - " \"start_place_name\": \"Willow Creek\",\n", - " \"end_place_name\": \"Unknown\"\n", + " \"description\": \"A quaint town nestled amidst rolling hills and whispering willows.\",\n", + " \"name\": \"Willow Creek\"\n", " },\n", " {\n", - " \"name\": \"Elise\",\n", - " \"description\": \"Anya's mother, Elise is a kind-hearted woman.\",\n", - " \"end_place_name\": \"Unknown\",\n", - " \"start_place_name\": \"Willow Creek\"\n", + " \"description\": \"A modest cottage in Willow Creek.\",\n", + " \"name\": \"Anya\\\\'s cottage\"\n", " },\n", " {\n", - " \"start_place_name\": \"Willow Creek\",\n", - " \"end_place_name\": \"Unknown\",\n", - " \"name\": \"Edward\",\n", - " \"description\": \"Anya's father, Edward is a wise-bearded man.\"\n", + " \"description\": \"The town\\\\'s only schoolhouse.\",\n", + " \"name\": \"Schoolhouse\"\n", " },\n", " {\n", - " \"end_place_name\": \"Unknown\",\n", - " \"start_place_name\": \"Willow Creek\",\n", - " \"description\": \"Anya's best friend, Samuel is a curious and adventurous boy with a mischievous grin.\",\n", - " \"name\": \"Samuel\"\n", + " \"description\": \"A shadowy forest near Willow Creek where the monster lived.\",\n", + " \"name\": \"Forest\"\n", " }\n", " ],\n", - " \"places\": [\n", + " \"relationships\": [\n", " {\n", - " \"description\": \"The quaint town of Willow Creek is nestled amidst rolling hills and whispering willows.\",\n", - " \"name\": \"Willow Creek\"\n", + " \"person_1_name\": \"Anya\",\n", + " \"person_2_name\": \"Elise\",\n", + " \"relationship\": \"daughter\"\n", " },\n", " {\n", - " \"description\": \"The town's only schoolhouse.\",\n", - " \"name\": \"Schoolhouse\"\n", + " \"person_1_name\": \"Anya\",\n", + " \"person_2_name\": \"Edward\",\n", + " \"relationship\": \"daughter\"\n", " },\n", " {\n", - " \"description\": \"A shadowy place filled with secrets and dangers, the Forest is home to a terrifying monster.\",\n", - " \"name\": \"Forest\"\n", + " \"person_1_name\": \"Anya\",\n", + " \"person_2_name\": \"Samuel\",\n", + " \"relationship\": \"friend\"\n", " }\n", " ]\n", " }\n",