Skip to content

feat(ai): add support for grounding with google search #9068

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 9 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .changeset/five-kids-grow.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
---
'firebase': minor
'@firebase/ai': minor
---

Add support for Grounding with Google Search.
46 changes: 39 additions & 7 deletions common/api-review/ai.api.md
Original file line number Diff line number Diff line change
Expand Up @@ -485,6 +485,15 @@ export interface GoogleAIGenerateContentResponse {
usageMetadata?: UsageMetadata;
}

// @public
export interface GoogleSearch {
}

// @public
export interface GoogleSearchTool {
googleSearch: GoogleSearch;
}

// @public @deprecated (undocumented)
export interface GroundingAttribution {
// (undocumented)
Expand All @@ -497,16 +506,29 @@ export interface GroundingAttribution {
web?: WebAttribution;
}

// @public
export interface GroundingChunk {
web?: WebGroundingChunk;
}

// @public
export interface GroundingMetadata {
// @deprecated (undocumented)
groundingAttributions: GroundingAttribution[];
// (undocumented)
groundingChunks?: GroundingChunk[];
groundingSupports?: GroundingSupport[];
// @deprecated (undocumented)
retrievalQueries?: string[];
// (undocumented)
searchEntryPoint?: SearchEntrypoint;
webSearchQueries?: string[];
}

// @public
export interface GroundingSupport {
groundingChunkIndices?: number[];
segment?: Segment;
}

// @public
export enum HarmBlockMethod {
PROBABILITY = "PROBABILITY",
Expand Down Expand Up @@ -857,14 +879,17 @@ export enum SchemaType {
STRING = "string"
}

// @public (undocumented)
// @public
export interface SearchEntrypoint {
renderedContent?: string;
}

// @public
export interface Segment {
// (undocumented)
endIndex: number;
// (undocumented)
partIndex: number;
// (undocumented)
startIndex: number;
text: string;
}

// @public
Expand Down Expand Up @@ -901,7 +926,7 @@ export interface TextPart {
}

// @public
export type Tool = FunctionDeclarationsTool;
export type Tool = FunctionDeclarationsTool | GoogleSearchTool;

// @public
export interface ToolConfig {
Expand Down Expand Up @@ -961,5 +986,12 @@ export interface WebAttribution {
uri: string;
}

// @public
export interface WebGroundingChunk {
domain?: string;
title?: string;
uri?: string;
}


```
12 changes: 12 additions & 0 deletions docs-devsite/_toc.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -76,10 +76,18 @@ toc:
path: /docs/reference/js/ai.generativemodel.md
- title: GoogleAIBackend
path: /docs/reference/js/ai.googleaibackend.md
- title: GoogleSearch
path: /docs/reference/js/ai.googlesearch.md
- title: GoogleSearchTool
path: /docs/reference/js/ai.googlesearchtool.md
- title: GroundingAttribution
path: /docs/reference/js/ai.groundingattribution.md
- title: GroundingChunk
path: /docs/reference/js/ai.groundingchunk.md
- title: GroundingMetadata
path: /docs/reference/js/ai.groundingmetadata.md
- title: GroundingSupport
path: /docs/reference/js/ai.groundingsupport.md
- title: ImagenGCSImage
path: /docs/reference/js/ai.imagengcsimage.md
- title: ImagenGenerationConfig
Expand Down Expand Up @@ -130,6 +138,8 @@ toc:
path: /docs/reference/js/ai.schemarequest.md
- title: SchemaShared
path: /docs/reference/js/ai.schemashared.md
- title: SearchEntrypoint
path: /docs/reference/js/ai.searchentrypoint.md
- title: Segment
path: /docs/reference/js/ai.segment.md
- title: StartChatParams
Expand All @@ -150,6 +160,8 @@ toc:
path: /docs/reference/js/ai.videometadata.md
- title: WebAttribution
path: /docs/reference/js/ai.webattribution.md
- title: WebGroundingChunk
path: /docs/reference/js/ai.webgroundingchunk.md
- title: analytics
path: /docs/reference/js/analytics.md
section:
Expand Down
2 changes: 1 addition & 1 deletion docs-devsite/ai.functiondeclaration.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ Structured representation of a function declaration as defined by the [OpenAPI 3
<b>Signature:</b>

```typescript
export declare interface FunctionDeclaration
export interface FunctionDeclaration
```

## Properties
Expand Down
2 changes: 1 addition & 1 deletion docs-devsite/ai.functiondeclarationstool.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ A `FunctionDeclarationsTool` is a piece of code that enables the system to inter
<b>Signature:</b>

```typescript
export declare interface FunctionDeclarationsTool
export interface FunctionDeclarationsTool
```

## Properties
Expand Down
21 changes: 21 additions & 0 deletions docs-devsite/ai.googlesearch.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
Project: /docs/reference/js/_project.yaml
Book: /docs/reference/_book.yaml
page_type: reference

{% comment %}
DO NOT EDIT THIS FILE!
This is generated by the JS SDK team, and any local changes will be
overwritten. Changes should be made in the source code at
https://github.com/firebase/firebase-js-sdk
{% endcomment %}

# GoogleSearch interface
Specifies the Google Search configuration.

Currently, this is an empty object, but it's reserved for future configuration options.

<b>Signature:</b>

```typescript
export interface GoogleSearch
```
39 changes: 39 additions & 0 deletions docs-devsite/ai.googlesearchtool.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
Project: /docs/reference/js/_project.yaml
Book: /docs/reference/_book.yaml
page_type: reference

{% comment %}
DO NOT EDIT THIS FILE!
This is generated by the JS SDK team, and any local changes will be
overwritten. Changes should be made in the source code at
https://github.com/firebase/firebase-js-sdk
{% endcomment %}

# GoogleSearchTool interface
A tool that allows a Gemini model to connect to Google Search to access and incorporate up-to-date information from the web into its responses.

Important: If using Grounding with Google Search, you are required to comply with the "Grounding with Google Search" usage requirements for your chosen API provider: [Gemini Developer API](https://ai.google.dev/gemini-api/terms#grounding-with-google-search) or [Vertex AI Gemini API](https://cloud.google.com/terms/service-terms) (see Service Terms section within the Service Specific Terms).

<b>Signature:</b>

```typescript
export interface GoogleSearchTool
```

## Properties

| Property | Type | Description |
| --- | --- | --- |
| [googleSearch](./ai.googlesearchtool.md#googlesearchtoolgooglesearch) | [GoogleSearch](./ai.googlesearch.md#googlesearch_interface) | Specifies the Google Search configuration. Currently, this is an empty object, but it's reserved for future configuration options. Specifies the Google Search configuration. Currently, this is an empty object, but it's reserved for future configuration options.<!-- -->When using this feature, you are required to comply with the "Grounding with Google Search" usage requirements for your chosen API provider: [Gemini Developer API](https://ai.google.dev/gemini-api/terms#grounding-with-google-search) or [Vertex AI Gemini API](https://cloud.google.com/terms/service-terms) (see Service Terms section within the Service Specific Terms). |

## GoogleSearchTool.googleSearch

Specifies the Google Search configuration. Currently, this is an empty object, but it's reserved for future configuration options. Specifies the Google Search configuration. Currently, this is an empty object, but it's reserved for future configuration options.

When using this feature, you are required to comply with the "Grounding with Google Search" usage requirements for your chosen API provider: [Gemini Developer API](https://ai.google.dev/gemini-api/terms#grounding-with-google-search) or [Vertex AI Gemini API](https://cloud.google.com/terms/service-terms) (see Service Terms section within the Service Specific Terms).

<b>Signature:</b>

```typescript
googleSearch: GoogleSearch;
```
35 changes: 35 additions & 0 deletions docs-devsite/ai.groundingchunk.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
Project: /docs/reference/js/_project.yaml
Book: /docs/reference/_book.yaml
page_type: reference

{% comment %}
DO NOT EDIT THIS FILE!
This is generated by the JS SDK team, and any local changes will be
overwritten. Changes should be made in the source code at
https://github.com/firebase/firebase-js-sdk
{% endcomment %}

# GroundingChunk interface
Represents a chunk of retrieved data that supports a claim in the model's response. This is part of the grounding information provided when grounding is enabled.

<b>Signature:</b>

```typescript
export interface GroundingChunk
```

## Properties

| Property | Type | Description |
| --- | --- | --- |
| [web](./ai.groundingchunk.md#groundingchunkweb) | [WebGroundingChunk](./ai.webgroundingchunk.md#webgroundingchunk_interface) | Contains details if the grounding chunk is from a web source. |

## GroundingChunk.web

Contains details if the grounding chunk is from a web source.

<b>Signature:</b>

```typescript
web?: WebGroundingChunk;
```
49 changes: 47 additions & 2 deletions docs-devsite/ai.groundingmetadata.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,11 @@ https://github.com/firebase/firebase-js-sdk
{% endcomment %}

# GroundingMetadata interface
Metadata returned to client when grounding is enabled.
Metadata returned when grounding is enabled.

Currently, only Grounding with Google Search is supported (see [GoogleSearchTool](./ai.googlesearchtool.md#googlesearchtool_interface)<!-- -->).

Important: If using Grounding with Google Search, you are required to comply with the "Grounding with Google Search" usage requirements for your chosen API provider: [Gemini Developer API](https://ai.google.dev/gemini-api/terms#grounding-with-google-search) or [Vertex AI Gemini API](https://cloud.google.com/terms/service-terms) (see Service Terms section within the Service Specific Terms).

<b>Signature:</b>

Expand All @@ -23,13 +27,17 @@ export interface GroundingMetadata
| Property | Type | Description |
| --- | --- | --- |
| [groundingAttributions](./ai.groundingmetadata.md#groundingmetadatagroundingattributions) | [GroundingAttribution](./ai.groundingattribution.md#groundingattribution_interface)<!-- -->\[\] | |
| [groundingChunks](./ai.groundingmetadata.md#groundingmetadatagroundingchunks) | [GroundingChunk](./ai.groundingchunk.md#groundingchunk_interface)<!-- -->\[\] | A list of [GroundingChunk](./ai.groundingchunk.md#groundingchunk_interface) objects. Each chunk represents a piece of retrieved content (for example, from a web page). that the model used to ground its response. |
| [groundingSupports](./ai.groundingmetadata.md#groundingmetadatagroundingsupports) | [GroundingSupport](./ai.groundingsupport.md#groundingsupport_interface)<!-- -->\[\] | A list of [GroundingSupport](./ai.groundingsupport.md#groundingsupport_interface) objects. Each object details how specific segments of the model's response are supported by the <code>groundingChunks</code>. |
| [retrievalQueries](./ai.groundingmetadata.md#groundingmetadataretrievalqueries) | string\[\] | |
| [webSearchQueries](./ai.groundingmetadata.md#groundingmetadatawebsearchqueries) | string\[\] | |
| [searchEntryPoint](./ai.groundingmetadata.md#groundingmetadatasearchentrypoint) | [SearchEntrypoint](./ai.searchentrypoint.md#searchentrypoint_interface) | Google Search entry point for web searches. This contains an HTML/CSS snippet that must be embedded in an app to display a Google Search entry point for follow-up web searches related to a model's “Grounded Response”. |
| [webSearchQueries](./ai.groundingmetadata.md#groundingmetadatawebsearchqueries) | string\[\] | A list of web search queries that the model performed to gather the grounding information. These can be used to allow users to explore the search results themselves. |

## GroundingMetadata.groundingAttributions

> Warning: This API is now obsolete.
>
> Use [GroundingChunk](./ai.groundingchunk.md#groundingchunk_interface) instead.
>

<b>Signature:</b>
Expand All @@ -38,16 +46,53 @@ export interface GroundingMetadata
groundingAttributions: GroundingAttribution[];
```

## GroundingMetadata.groundingChunks

A list of [GroundingChunk](./ai.groundingchunk.md#groundingchunk_interface) objects. Each chunk represents a piece of retrieved content (for example, from a web page). that the model used to ground its response.

<b>Signature:</b>

```typescript
groundingChunks?: GroundingChunk[];
```

## GroundingMetadata.groundingSupports

A list of [GroundingSupport](./ai.groundingsupport.md#groundingsupport_interface) objects. Each object details how specific segments of the model's response are supported by the `groundingChunks`<!-- -->.

<b>Signature:</b>

```typescript
groundingSupports?: GroundingSupport[];
```

## GroundingMetadata.retrievalQueries

> Warning: This API is now obsolete.
>
> Use [GroundingSupport](./ai.groundingsupport.md#groundingsupport_interface) instead.
>

<b>Signature:</b>

```typescript
retrievalQueries?: string[];
```

## GroundingMetadata.searchEntryPoint

Google Search entry point for web searches. This contains an HTML/CSS snippet that must be embedded in an app to display a Google Search entry point for follow-up web searches related to a model's “Grounded Response”.

<b>Signature:</b>

```typescript
searchEntryPoint?: SearchEntrypoint;
```

## GroundingMetadata.webSearchQueries

A list of web search queries that the model performed to gather the grounding information. These can be used to allow users to explore the search results themselves.

<b>Signature:</b>

```typescript
Expand Down
46 changes: 46 additions & 0 deletions docs-devsite/ai.groundingsupport.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
Project: /docs/reference/js/_project.yaml
Book: /docs/reference/_book.yaml
page_type: reference

{% comment %}
DO NOT EDIT THIS FILE!
This is generated by the JS SDK team, and any local changes will be
overwritten. Changes should be made in the source code at
https://github.com/firebase/firebase-js-sdk
{% endcomment %}

# GroundingSupport interface
Provides information about how a specific segment of the model's response is supported by the retrieved grounding chunks.

<b>Signature:</b>

```typescript
export interface GroundingSupport
```

## Properties

| Property | Type | Description |
| --- | --- | --- |
| [groundingChunkIndices](./ai.groundingsupport.md#groundingsupportgroundingchunkindices) | number\[\] | A list of indices that refer to specific [GroundingChunk](./ai.groundingchunk.md#groundingchunk_interface) objects within the [GroundingMetadata.groundingChunks](./ai.groundingmetadata.md#groundingmetadatagroundingchunks) array. These referenced chunks are the sources that support the claim made in the associated <code>segment</code> of the response. For example, an array <code>[1, 3, 4]</code> means that <code>groundingChunks[1]</code>, <code>groundingChunks[3]</code>, and <code>groundingChunks[4]</code> are the retrieved content supporting this part of the response. |
| [segment](./ai.groundingsupport.md#groundingsupportsegment) | [Segment](./ai.segment.md#segment_interface) | Specifies the segment of the model's response content that this grounding support pertains to. |

## GroundingSupport.groundingChunkIndices

A list of indices that refer to specific [GroundingChunk](./ai.groundingchunk.md#groundingchunk_interface) objects within the [GroundingMetadata.groundingChunks](./ai.groundingmetadata.md#groundingmetadatagroundingchunks) array. These referenced chunks are the sources that support the claim made in the associated `segment` of the response. For example, an array `[1, 3, 4]` means that `groundingChunks[1]`<!-- -->, `groundingChunks[3]`<!-- -->, and `groundingChunks[4]` are the retrieved content supporting this part of the response.

<b>Signature:</b>

```typescript
groundingChunkIndices?: number[];
```

## GroundingSupport.segment

Specifies the segment of the model's response content that this grounding support pertains to.

<b>Signature:</b>

```typescript
segment?: Segment;
```
Loading
Loading