lip-syncing for live2d models #583
Conversation
- new character card editing dialog component - add edit button in character card details - implement character card update logic - add related internationalization texts
✅ Deploy Preview for airi-vtuber ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
❌ Deploy Preview for airi-docs failed. Why did it fail? →
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Summary of Changes
Hello @303-SeeOther, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!
This pull request significantly enhances the application by introducing dynamic lip-syncing for Live2D models, making character interactions more immersive. Concurrently, it empowers users with the ability to edit their character cards, providing greater control over character customization and details.
Highlights
- Live2D Lip-Syncing Implementation: Initial implementation of lip-syncing for Live2D models, enabling dynamic mouth movements based on speech. This feature currently supports vowel-based recognition and lays the groundwork for future custom configuration uploads.
- Character Card Editing Feature: A new character card editing functionality has been added, allowing users to modify existing character cards through a dedicated dialog. This includes editing various card properties such as identity, behavior, and settings.
Using Gemini Code Assist
The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.
Invoking Gemini
You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.
| Feature | Command | Description |
|---|---|---|
| Code Review | /gemini review |
Performs a code review for the current pull request in its current state. |
| Pull Request Summary | /gemini summary |
Provides a summary of the current pull request in its current state. |
| Comment | @gemini-code-assist | Responds in comments when explicitly tagged, both in issue comments and review comments. |
| Help | /gemini help |
Displays a list of available commands. |
Customization
To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.
Limitations & Feedback
Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.
You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.
Footnotes
-
Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩
There was a problem hiding this comment.
Code Review
This pull request introduces lip-syncing functionality for Live2D models and adds a character card editing feature. The review focuses on the added CardEditDialog.vue component and modifications to Model.vue for lip-syncing, ensuring code correctness, efficiency, and adherence to the provided style guide. The review also checks for potential issues related to error handling, resource management, and UI clarity.
| const motionSync = new MotionSync(internalModel) | ||
| motionSync.loadDefaultMotionSync() | ||
| // 保存motionSync实例供音频播放使用 | ||
| motionSyncInstance = motionSync |
There was a problem hiding this comment.
It's important to handle potential errors during MotionSync initialization. Wrap the MotionSync instantiation and loadDefaultMotionSync call in a try...catch block to gracefully handle any exceptions that may occur. This will prevent the component from crashing if there's an issue with the MotionSync library or the model itself.
try {
const motionSync = new MotionSync(internalModel)
motionSync.loadDefaultMotionSync()
// 保存motionSync实例供音频播放使用
motionSyncInstance = motionSync
} catch (error) {
console.error("Error initializing MotionSync:", error);
}
| audioQueue.clear() | ||
| ttsQueue.clear() |
There was a problem hiding this comment.
It's important to handle potential errors when clearing the queues. Wrap the audioQueue.clear() and ttsQueue.clear() calls in try...catch blocks to prevent the cleanup process from being interrupted if an error occurs during queue clearing. This will ensure that all resources are properly released, even if there's an issue with one of the queues.
try {
audioQueue.clear()
} catch (error) {
console.error("Error clearing audioQueue:", error);
}
try {
ttsQueue.clear()
} catch (error) {
console.error("Error clearing ttsQueue:", error);
}
| <Button | ||
| variant="secondary" | ||
| icon="i-solar:clapperboard-edit-line-duotone" | ||
| :label="t('settings.pages.card.edit.edit')" | ||
| @click="isEditDialogOpen = true" | ||
| /> |
There was a problem hiding this comment.
Consider extracting this button into a separate component to improve reusability and maintainability. This would also simplify the CardDetailDialog.vue component and make it easier to read.
If the button is only used in this component, this is less important, but if it's used in multiple places, it would be beneficial to extract it.
| function componentCleanUp() { | ||
| // 取消所有动画帧请求 | ||
| cancelAnimationFrame(dropShadowAnimationId.value) | ||
| if (volumeUpdateId) { | ||
| cancelAnimationFrame(volumeUpdateId) | ||
| volumeUpdateId = null | ||
| } | ||
| // 清理模型相关资源 | ||
| model.value && pixiApp.value?.stage.removeChild(model.value) | ||
| // 清理TTS相关资源 | ||
| lipSyncStarted.value = false | ||
| nowSpeaking.value = false | ||
| // 停止并断开当前音频源 | ||
| if (currentAudioSource) { | ||
| try { | ||
| currentAudioSource.stop() | ||
| currentAudioSource.disconnect() | ||
| } | ||
| catch (error) { | ||
| console.warn('Error during audio source cleanup:', error) | ||
| } | ||
| currentAudioSource = null | ||
| } | ||
| // 断开音频分析器连接 | ||
| if (audioAnalyser.value) { | ||
| audioAnalyser.value.disconnect() | ||
| // 注意：不要在这里设置audioAnalyser.value = null，因为组件可能会被重新激活 | ||
| } | ||
| // 清理motionSync实例 | ||
| if (motionSyncInstance) { | ||
| try { | ||
| motionSyncInstance.reset() | ||
| } | ||
| catch (error) { | ||
| console.error('MotionSync reset failed during cleanup:', error) | ||
| } | ||
| motionSyncInstance = null | ||
| } | ||
| // 清空队列 | ||
| audioQueue.clear() | ||
| ttsQueue.clear() | ||
| } | ||
| onUnmounted(() => { | ||
| componentCleanUp() |
There was a problem hiding this comment.
This cleanup function is quite comprehensive, which is good. However, it would be beneficial to add comments explaining the purpose of each section (e.g., "Cleanup model-related resources", "Cleanup TTS-related resources"). This will make it easier for other developers to understand the code and maintain it in the future.
| const updateCard = (id: string, card: AiriCard | Card | ccv3.CharacterCardV3) => { | ||
| if (cards.value.has(id)) { | ||
| const _existingCard = cards.value.get(id)! | ||
| const updatedCard = newAiriCard(card) | ||
|
|
||
| // Preserve the existing ID | ||
| cards.value.set(id, updatedCard) | ||
| return true | ||
| } | ||
| return false | ||
| } |
There was a problem hiding this comment.
Consider adding a check to ensure that the card argument is not null or undefined before attempting to update the card. This will prevent potential errors if an invalid card object is passed to the updateCard function.
| const updateCard = (id: string, card: AiriCard | Card | ccv3.CharacterCardV3) => { | |
| if (cards.value.has(id)) { | |
| const _existingCard = cards.value.get(id)! | |
| const updatedCard = newAiriCard(card) | |
| // Preserve the existing ID | |
| cards.value.set(id, updatedCard) | |
| return true | |
| } | |
| return false | |
| } | |
| const updateCard = (id: string, card: AiriCard | Card | ccv3.CharacterCardV3) => { | |
| if (!card) { | |
| console.warn("Attempted to update card with null or undefined card object."); | |
| return false; | |
| } | |
| if (cards.value.has(id)) { |
| */ | ||
| key: T, | ||
| transform?: (input: string) => string, | ||
| ) { |
There was a problem hiding this comment.
| const lipSyncStarted = ref(false) | ||
| let currentAudioSource: AudioBufferSourceNode | null = null | ||
| const audioQueue = createQueue<{ audioBuffer: AudioBuffer, text: string }>({ |
There was a problem hiding this comment.
If those were all deleted, how can the character connect and interact with TTS service?
There was a problem hiding this comment.
Thank you very much for your attention. I have migrated the code related to interaction with the TTS server to packages/stage-ui/src/components/Scenes/Live2D/Model.vue. After testing, it can interact with the TTS server normally.
3 participants
Description
The function of lip-syncing for live2d models has been initially implemented, and a character card editing feature has been added.
Linked Issues
Additional Context
Currently, the default lip shape voice synchronization configuration only includes vowel lip shape recognition. In the future, we will add support for users to upload their own lip shape voice synchronization configuration files.