feat: support speech to speech by nova-sonic (#39)

Author: zhu-xiaowei

README.md

@@ -16,21 +16,23 @@
 SwiftChat is a fast and responsive AI chat application developed with [React Native](https://reactnative.dev/) and
 powered by [Amazon Bedrock](https://aws.amazon.com/bedrock/), with compatibility extending to other model providers such
 as Ollama, DeepSeek, OpenAI and OpenAI Compatible. With its minimalist design philosophy and robust privacy protection,
-it delivers real-time streaming conversations and AI image generation capabilities across Android, iOS, and macOS
-platforms.
+it delivers real-time streaming conversations, AI image generation and voice conversation assistant capabilities
+across Android, iOS, and macOS platforms.
 
 ![](assets/promo.avif)
 
 ### What's New 🔥
 
+- 🚀 Support Speech to Speech By Amazon Nova Sonic on Apple Platform. Check [How to Use](#amazon-nova-sonic) for
+  more details. (From v2.3.0).
+- Support Request Latency and token response speed display (From v2.3.0).
+- Change to new bubble format UI for user question (From v2.3.0).
 - Support for OpenAI Compatible models. You can now
   use [easy-model-deployer](https://github.com/aws-samples/easy-model-deployer),
   OpenRouter, or any OpenAI-compatible model provider via SwiftChat. Please
   check [Configure OpenAI Compatible](#openai-compatible) section for more details(From v2.2.0).
-- Support for quick model switching (From v2.2.0).
-- Support regeneration of AI responses (From v2.2.0).
 
-**Key Features:**
+### Key Features
 
 - Real-time streaming chat with AI
 - Rich Markdown Support: Tables, Code Blocks, LaTeX and More
@@ -45,7 +47,44 @@ platforms.
   and [OpenAI Compatible](#openai-compatible) Models)
 - Fully Customizable System Prompt Assistant
 
-**Supported Features For Amazon Nova series**
+### Amazon Nova Series Features
+
+#### Amazon Nova Sonic Speech to Speech Model
+
+**Usage Guide**
+
+1. Amazon Nova Sonic model is supported starting from v2.3.0. If you have deployed it before, You Need to:
+    * [Update CloudFormation](#upgrade-cloudformation) Stack
+    * [Update API](#upgrade-api)
+    * [Upgrade your App](#-quick-download) to v2.3.0 or later
+
+   If you have not Deployed your CloudFormation Stack please
+   finish [Getting Started with Amazon Bedrock](#getting-started-with-amazon-bedrock) section.
+2. Switch the **Region** to `us-east-1` in the settings page and select the `Nova Sonic` under **Chat Model**.
+3. Return to Chat page, select a system prompt or directly click the microphone icon to start your conversation.
+
+**Features for Speech to Speech**
+
+1. Built-in spoken language practice for words and sentences, as well as storytelling scenarios. You can also add
+   **Custom System Prompts** for voice chatting in different scenarios.
+2. Support **Barge In** by default, Also you can disable in system prompt.
+3. Support selecting voices in the settings page, including American/British English, and options for male and female voices.
+4. Support **Echo Cancellation**, You can talk directly to the device without wearing headphones.
+5. Support **Voice Waveform** to display volume level.
+
+**General Talk**
+
+https://github.com/user-attachments/assets/d3028312-c420-476c-88c2-ba870015f3c4
+
+**Learn Sentences**
+
+https://github.com/user-attachments/assets/ebf21b12-9c93-4d2e-a109-1d6484019838
+
+**Telling Story on Mac (With barge in feature)**
+
+https://github.com/user-attachments/assets/c70fc2b4-8960-4a5e-b4f8-420fcd5eafd4
+
+#### Other Features
 
 - Record 30-second videos directly on Android and iOS for Nova analysis
 - Upload large videos (1080p/4K) beyond 8MB with auto compression
@@ -57,7 +96,7 @@ platforms.
 #### YouTube Video
 
 [<img src="./assets/youtube.avif">](https://www.youtube.com/watch?v=rey05WzfEbM)
-> The content in the video is an early version. For UI, architecture, and inconsistencies, please refer to the current 
+> The content in the video is an early version. For UI, architecture, and inconsistencies, please refer to the current
 > documentation.
 
 **Comprehensive Multimodal Analysis**: Text, Image, Document and Video
@@ -111,7 +150,7 @@ this [example](https://github.com/awslabs/aws-lambda-web-adapter/tree/main/examp
 Ensure you have access to Amazon Bedrock foundation models. SwiftChat default settings are:
 
 - Region: `us-west-2`
-- Text Model: `Amazon Nova Pro`
+- Chat Model: `Amazon Nova Pro`
 - Image Model: `Stable Diffusion 3.5 Large`
 
 If you are using the image generation feature, please make sure you have enabled access to the `Amazon Nova Lite` model.
@@ -195,7 +234,7 @@ Congratulations 🎉 Your SwiftChat App is ready to use!
     ```bash
     http://localhost:11434
     ```
-3. Once the correct Server URL is entered, you can select your desired Ollama models from the **Text Model** dropdown
+3. Once the correct Server URL is entered, you can select your desired Ollama models from the **Chat Model** dropdown
    list.
 
 </details>
@@ -207,7 +246,7 @@ Congratulations 🎉 Your SwiftChat App is ready to use!
 
 1. Go to the **Settings Page** and select the **DeepSeek** tab.
 2. Input your DeepSeek API Key.
-3. Choose DeepSeek models from the **Text Model** dropdown list. Currently, the following DeepSeek models are supported:
+3. Choose DeepSeek models from the **Chat Model** dropdown list. Currently, the following DeepSeek models are supported:
     - `DeepSeek-V3`
     - `DeepSeek-R1`
 
@@ -220,9 +259,12 @@ Congratulations 🎉 Your SwiftChat App is ready to use!
 
 1. Navigate to the **Settings Page** and select the **OpenAI** tab.
 2. Enter your OpenAI API Key.
-3. Select OpenAI models from the **Text Model** dropdown list. The following OpenAI models are currently supported:
+3. Select OpenAI models from the **Chat Model** dropdown list. The following OpenAI models are currently supported:
     - `GPT-4o`
     - `GPT-4o mini`
+    - `GPT-4.1`
+    - `GPT-4.1 mini`
+    - `GPT-4.1 nano`
 
 Additionally, if you have deployed the [ClickStream Server](#step-2-deploy-stack-and-get-your-api-url), you can enable
 the **Use Proxy** option to forward your requests.
@@ -239,7 +281,7 @@ the **Use Proxy** option to forward your requests.
     - `Base URL` of your model provider
     - `API Key` of your model provider
     - `Model ID` of the models you want to use (separate multiple models with commas)
-3. Select one of your models from the **Text Model** dropdown list.
+3. Select one of your models from the **Chat Model** dropdown list.
 
 </details>
 
@@ -379,6 +421,26 @@ the [release notes](https://github.com/aws-samples/swift-chat/releases) to see i
 - **For Lambda**: Click and open [Lambda Services](https://console.aws.amazon.com/lambda/home#/functions), find and open
   your Lambda which start with `SwiftChatLambda-xxx`, click the **Deploy new image** button and click Save.
 
+### Upgrade CloudFormation
+
+1. Click and open [CloudFormation](https://console.aws.amazon.com/cloudformation), switch to the region which you
+   have deployed the **SwiftChatAPI** stack.
+2. Select the **SwiftChatAPI** Stack, click **Update stack** -> **Make a direct update**
+3. On the **Update stack** Page, select **Replace existing template** under the **Amazon S3 URL**, then input the
+   following template url.
+
+   For App Runner
+    ```
+    https://aws-gcr-solutions.s3.amazonaws.com/swift-chat/latest/SwiftChatAppRunner.template
+    ``` 
+   For Lambda
+    ```
+    https://aws-gcr-solutions.s3.amazonaws.com/swift-chat/latest/SwiftChatLambda.template
+    ``` 
+4. Click the **Next** button and continue click **Next** button. On the **Configure stack options** page,
+   check `I acknowledge that AWS CloudFormation might create IAM resources.` then click **Next** and *Submit* button to
+   update your CloudFormation Template.
+
 ## Security
 
 See [CONTRIBUTING](CONTRIBUTING.md#security-issue-notifications) for more information.

README_CN.md

@@ -15,18 +15,19 @@
 
 SwiftChat 是一款快速响应的 AI 聊天应用，采用 [React Native](https://reactnative.dev/)
 开发，并依托 [Amazon Bedrock](https://aws.amazon.com/bedrock/) 提供强大支持，同时兼容 Ollama、DeepSeek、OpenAI 和 OpenAI API 兼容的其他模型供应商。
-凭借其极简设计理念与坚实的隐私保护措施，该应用在 Android、iOS 和 macOS 平台上实现了实时流式对话及 AI 图像生成功能。
+凭借其极简设计理念与坚实的隐私保护措施，该应用在 Android、iOS 和 macOS 平台上实现了实时流式对话、AI 图像生成和语音对话助手功能。
 
 ![](assets/promo.avif)
 
 ### 新功能 🔥
 
+- 🚀 在 Apple 平台上支持 Amazon Nova Sonic 语音对话功能。查看 [使用方法](#amazon-nova-系列功能) 了解更多详情。（自 v2.3.0 起）。
+- 支持请求延迟和 token 响应速度显示（自 v2.3.0 起）。
+- 用户问题展示为新的气泡 UI 格式（自 v2.3.0 起）。
 - 支持 OpenAI Compatible 模型。您现在可以通过 SwiftChat 使用 [easy-model-deployer](https://github.com/aws-samples/easy-model-deployer)、
   OpenRouter 或任何 OpenAI API 兼容的模型。更多详情请查看 [配置 OpenAI Compatible](#openai-compatible) 部分（自 v2.2.0 起）。
-- 支持快速切换模型（自 v2.2.0 起）。
-- 支持 AI 内容的重新生成（自 v2.2.0 起）。
 
-**主要特点:**
+### 主要特点
 
 - 与 AI 进行实时流式聊天
 - 支持丰富的 Markdown 渲染：表格、代码块、LaTeX 公式等
@@ -42,7 +43,42 @@ SwiftChat 是一款快速响应的 AI 聊天应用，采用 [React Native](https
   和 [OpenAI Compatible](#openai-compatible) 模型)
 - 支持完全自定义的系统提示词助手
 
-**Amazon Nova 系列功能支持**
+### Amazon Nova 系列功能
+
+#### Amazon Nova Sonic 语音对话模型
+
+**使用指南**
+
+1. Amazon Nova Sonic 模型从 v2.3.0 开始支持。如果您之前已经部署过，您需要：
+    * [更新 CloudFormation](#升级-cloudformation) 堆栈
+    * [更新 API](#升级-api)
+    * [升级您的应用](#-快速下载) 到 v2.3.0 或更高版本
+
+   如果您尚未部署 CloudFormation 堆栈，请完成 [Amazon Bedrock 入门](#入门指南---使用-amazon-bedrock-上的模型) 部分。
+2. 在设置页面将 **区域** 切换到 `us-east-1`，并在 **Chat Model** 下选择 `Nova Sonic`。
+3. 返回聊天页面，选择系统提示词或直接点击麦克风图标开始对话。
+
+**语音对话功能**
+
+1. 内置单词和句子的口语练习，以及讲故事场景。您还可以添加 **自定义系统提示词** 用于不同场景的语音聊天。
+2. 支持在设置页面中选择声音类型，支持美式/英式英语，以及男声和女声的选择。
+3. 默认支持 **插话功能**，您也可以在系统提示词中禁用。
+4. 支持 **回声消除**，您可以直接对着设备说话而无需佩戴耳机。
+5. 支持 **语音波形** 显示音量级别。
+
+**日常对话**
+
+https://github.com/user-attachments/assets/d3028312-c420-476c-88c2-ba870015f3c4
+
+**学习句子**
+
+https://github.com/user-attachments/assets/ebf21b12-9c93-4d2e-a109-1d6484019838
+
+**Mac 上讲故事（打断功能展示）**
+
+https://github.com/user-attachments/assets/c70fc2b4-8960-4a5e-b4f8-420fcd5eafd4
+
+#### 其他功能
 
 - 支持直接在安卓和 iOS 设备上录制最长 30 秒的视频供 Nova 分析
 - 支持自动压缩上传超过 8MB 的高清视频（1080p/4K）
@@ -176,7 +212,7 @@ SwiftChat 是一款快速响应的 AI 聊天应用，采用 [React Native](https
     ```bash
     http://localhost:11434
     ```
-3. 输入正确的服务器 URL 后，您可以从 **文本模型** 下拉列表中选择所需的 Ollama 模型。
+3. 输入正确的服务器 URL 后，您可以从 **Chat Model** 下拉列表中选择所需的 Ollama 模型。
 
 </details>
 
@@ -187,7 +223,7 @@ SwiftChat 是一款快速响应的 AI 聊天应用，采用 [React Native](https
 
 1. 进入 **设置页面**，选择 **DeepSeek** 标签。
 2. 输入您的 DeepSeek API 密钥。
-3. 从 **文本模型** 下拉列表中选择 DeepSeek 模型。目前支持以下 DeepSeek 模型：
+3. 从 **Chat Model** 下拉列表中选择 DeepSeek 模型。目前支持以下 DeepSeek 模型：
     - `DeepSeek-V3`
     - `DeepSeek-R1`
 
@@ -200,9 +236,12 @@ SwiftChat 是一款快速响应的 AI 聊天应用，采用 [React Native](https
 
 1. 进入 **设置页面**，选择 **OpenAI** 标签。
 2. 输入您的 OpenAI API 密钥。
-3. 从 **文本模型** 下拉列表中选择 OpenAI 模型。目前支持以下 OpenAI 模型：
+3. 从 **Chat Model** 下拉列表中选择 OpenAI 模型。目前支持以下 OpenAI 模型：
     - `GPT-4o`
     - `GPT-4o mini`
+    - `GPT-4.1`
+    - `GPT-4.1 mini`
+    - `GPT-4.1 nano`
 
 此外，如果您已部署 [ClickStream Server](#第-2-步-部署堆栈并获取-api-url)，可以启用 **Use Proxy** 选项以转发您的请求。
 
@@ -218,7 +257,7 @@ SwiftChat 是一款快速响应的 AI 聊天应用，采用 [React Native](https
     - 模型提供商的 `Base URL`
     - 模型提供商的 `API Key`
     - 您想使用的 `Model ID`（多个模型用英文逗号分隔）
-3. 从 **文本模型** 下拉列表中选择您的一个模型。
+3. 从 **Chat Model** 下拉列表中选择您的一个模型。
 
 </details>
 
@@ -357,6 +396,23 @@ npm run ios
 - **对于 Lambda**：点击并打开 [Lambda Services](https://console.aws.amazon.com/lambda/home#/functions) 页面，找到并打开
   以 `SwiftChatLambda-xxx` 开头的 Lambda 函数，点击 **部署新镜像** 按钮并点击保存。
 
+### 升级 CloudFormation
+
+1. 点击并打开 [CloudFormation](https://console.aws.amazon.com/cloudformation)，切换到您已部署 **SwiftChatAPI** 堆栈的区域。
+2. 选择 **SwiftChatAPI** 堆栈，点击 **更新堆栈** -> **进行直接更新**
+3. 在 **更新堆栈** 页面上，在 **Amazon S3 URL** 下选择 **替换现有模板**，然后输入以下模板 URL。
+
+   对于 App Runner
+    ```
+    https://aws-gcr-solutions.s3.amazonaws.com/swift-chat/latest/SwiftChatAppRunner.template
+    ```
+   对于 Lambda
+    ```
+    https://aws-gcr-solutions.s3.amazonaws.com/swift-chat/latest/SwiftChatLambda.template
+    ```
+4. 点击 **下一步** 按钮并继续点击 **下一步** 按钮。在 **配置堆栈选项** 页面上，
+   勾选 `我确认，AWS CloudFormation 可能会创建 IAM 资源。` 然后点击 **下一步** 和 **提交** 按钮来更新您的 CloudFormation 模板。
+
 ## 安全
 
 更多信息请参见 [CONTRIBUTING](CONTRIBUTING.md#security-issue-notifications)。

react-native/Gemfile.lock

@@ -5,11 +5,12 @@ GEM
       base64
       nkf
       rexml
-    activesupport (7.0.8.3)
+    activesupport (6.1.7.10)
       concurrent-ruby (~> 1.0, >= 1.0.2)
       i18n (>= 1.6, < 2)
       minitest (>= 5.1)
       tzinfo (~> 2.0)
+      zeitwerk (~> 2.3)
     addressable (2.8.6)
       public_suffix (>= 2.0.2, < 6.0)
     algoliasearch (1.27.5)
@@ -88,13 +89,14 @@ GEM
       colored2 (~> 3.1)
       nanaimo (~> 0.3.0)
       rexml (>= 3.3.6, < 4.0)
+    zeitwerk (2.6.18)
 
 PLATFORMS
   ruby
 
 DEPENDENCIES
   activesupport (>= 6.1.7.5, < 7.1.0)
-  cocoapods (>= 1.13, < 1.15)
+  cocoapods (>= 1.13, < 1.17)
 
 RUBY VERSION
    ruby 3.2.2p53

react-native/ios/Modules/VoiceChat/VoiceChatModule.m

@@ -0,0 +1,31 @@
+//
+//  VoiceChatModule.m
+//  SwiftChat
+//
+//  Created on 2025/4/10.
+//
+
+#import <React/RCTBridgeModule.h>
+#import <React/RCTEventEmitter.h>
+
+@interface RCT_EXTERN_MODULE(VoiceChatModule, RCTEventEmitter)
+
+RCT_EXTERN_METHOD(initialize:(NSDictionary *)config
+                  withResolver:(RCTPromiseResolveBlock)resolve
+                  withRejecter:(RCTPromiseRejectBlock)reject)
+
+RCT_EXTERN_METHOD(startConversation:(NSString *)systemPrompt
+                  withVoiceId:(NSString *)voiceId
+                  withAllowInterruption:(BOOL *)voiceId
+                  withResolver:(RCTPromiseResolveBlock)resolve
+                  withRejecter:(RCTPromiseRejectBlock)reject)
+
+
+RCT_EXTERN_METHOD(endConversation:(RCTPromiseResolveBlock)resolve
+                  withRejecter:(RCTPromiseRejectBlock)reject)
+
+RCT_EXTERN_METHOD(updateCredentials:(NSDictionary *)config
+                  withResolver:(RCTPromiseResolveBlock)resolve
+                  withRejecter:(RCTPromiseRejectBlock)reject)
+
+@end