Voice Assistant System for Home Assitant | Seeed Studio Wiki

[giscus[bot]]

Voice Assistant System for Home Assitant | Seeed Studio Wiki

Voice Assistant System for Home Assitant

https://wiki.seeedstudio.com/respeaker_lite_ha/

[Gangoke]

How to get esphome media_player working?

[jerryyip]

Thanks to @formatBCE, you can find a wonderful esphome yaml (with media_player of course) in his repo

[jastBytes]

My speaker seems to reboot constantly and giving a crackling sound when it does. It connects to HA and disconnects again and again. Is that a knwon problem? The logs in ESPHome don't show any hint whats the issue.

[jastBytes]

I found the error log but don't know how to fix:

[12:55:30][D][voice_assistant:510]: State changed from IDLE to START_MICROPHONE
[12:55:30][D][voice_assistant:516]: Desired state set to START_PIPELINE
[12:55:30][D][light:036]: 'ReSpeakerv3' Setting:
[12:55:30][D][voice_assistant:222]: Starting Microphone
[12:55:30]Guru Meditation Error: Core  1 panic'ed (StoreProhibited). Exception was unhandled.
[12:55:30]
[12:55:30]Core  1 register dump:
[12:55:30]PC      : 0x400570af  PS      : 0x00060a30  A0      : 0x82011fc1  A1      : 0x3fcedec0  
[12:55:30]A2      : 0xa5a5a5a5  A3      : 0x00000000  A4      : 0x00000400  A5      : 0xa5a5a5a5  
[12:55:30]A6      : 0x3fcaf900  A7      : 0x00000000  A8      : 0x00000040  A9      : 0x3fcede40  
[12:55:30]A10     : 0x3fcee30c  A11     : 0x00000005  A12     : 0x8037c3d8  A13     : 0x3fc98270  
[12:55:30]A14     : 0x00000000  A15     : 0x00000000  SAR     : 0x00000004  EXCCAUSE: 0x0000001d  
[12:55:30]EXCVADDR: 0xa5a5a5a5  LBEG    : 0x40056f08  LEND    : 0x40056f12  LCOUNT  : 0x00000000  
[12:55:30]
[12:55:30]
[12:55:30]Backtrace: 0x400570ac:0x3fcedec0 0x42011fbe:0x3fceded0 0x42012d78:0x3fcedef0 0x42081b99:0x3fcedfa0 0x42081be5:0x3fcedfc0 0x42016ad0:0x3fcedfe0 0x420195f6:0x3fcee010 0x4200b3f2:0x3fcee030
[12:55:30]
[12:55:30]
[12:55:30]
[12:55:30]
[12:55:30]ELF file SHA256: 582573910116efa7
[12:55:30]
[12:55:30]Rebooting...

[jerryyip]

Could you please put the backtrace and your ELF file to https://maximeborges.github.io/esp-stacktrace-decoder/ to decode the stack? And are you using the same yaml file as our wiki?

[jastBytes]

It was a problem with ESPHome. Updating it helped. It works now.

[alpaygani]

Even though I followed all the steps, I cannot see my "respeaker lite" device with web.esphome.
I don't have a device showing as "USB/Jtag" in the device list.
I am using

• Windows 11 (64bit)
• Chrome browser
  I installed "respeaker_lite_i2s_dfu_firmware_v1.0.7.bin" firmware on Respeaker Lite via esp-idf. I also installed the (Winusb V6...) drivers via Zadgi. I tried all of this on 3 different computers. Still I can't see my device in the comport section.
  Please help on this matter.
  Thank you very much.

[jerryyip]

The respeaker_lite_i2s_dfu_firmware_v1.0.7.bin firmware is for the XMOS chip, not the ESP32S3. It should be installed with dfu-util tool.
Both XMOS chip and XIAO ESP32S3 need to flash/update their firmware.
This guide is just for XMOS chip firmware updates: https://github.com/respeaker/ReSpeaker_Lite/blob/master/xmos_firmwares/dfu_guide.md
And these steps are for XIAO ESP32S3 ESPhome firmware installation:
https://wiki.seeedstudio.com/respeaker_lite_ha/#add-xiao-esp32s3-to-esphome
Thanks to the community we have some perfect video guides too:
https://www.youtube.com/watch?v=XjUeJh2Ok3o&t=1s

[thematrixdev]

If I want to use the 3.5mm jack to output audio, rather than the jumper, do I need to modify anything?

[formatBCE]

It works automatically, just insert jack

[Richard-Regan]

Hi,
When I run the downloaded YAML, I get errors as shown below:
NFO ESPHome 2024.12.1
INFO Reading configuration /config/esphome/respeaker-s3.yaml...
INFO Generating C++ source...
INFO Updating https://github.com/espressif/esp-protocols.git@mdns-v1.3.2
INFO Compiling app...
Processing esp32s3 (board: esp32-s3-devkitc-1; framework: espidf; platform: https://github.com/pioarduino/platform-espressif32.git#51.03.07)

HARDWARE: ESP32S3 240MHz, 320KB RAM, 8MB Flash

• framework-espidf @ 3.50105.0 (5.1.5)
• tool-cmake @ 3.21.3
• tool-esptoolpy @ 4.8.1
• tool-mklittlefs @ 3.2.0
• tool-ninja @ 1.10.2
• tool-riscv32-esp-elf-gdb @ 12.1.0+20221002
• tool-xtensa-esp-elf-gdb @ 12.1.0+20221002
• toolchain-esp32ulp @ 2.35.0-20220830
• toolchain-riscv32-esp @ 12.2.0+20230208
• toolchain-xtensa-esp32s3 @ 12.2.0+20230208
  Reading CMake configuration...
  Generating assembly for certificate bundle...
  Warning! Flash memory size mismatch detected. Expected 4MB, found 2MB!
  Please select a proper value in your sdkconfig.defaults or via the menuconfig target!
  No dependencies
  Compiling .pioenvs/esp32s3/src/esphome/components/api/api_connection.cpp.o
  Compiling .pioenvs/esp32s3/src/esphome/components/api/api_frame_helper.cpp.o
  Compiling .pioenvs/esp32s3/src/esphome/components/api/api_pb2.cpp.o
  Compiling .pioenvs/esp32s3/src/esphome/components/api/api_pb2_service.cpp.o
  Compiling .pioenvs/esp32s3/src/esphome/components/api/api_server.cpp.o
  Compiling .pioenvs/esp32s3/src/esphome/components/api/list_entities.cpp.o
  Compiling .pioenvs/esp32s3/src/esphome/components/api/proto.cpp.o
  Compiling .pioenvs/esp32s3/src/esphome/components/api/subscribe_state.cpp.o
  Compiling .pioenvs/esp32s3/src/esphome/components/api/user_services.cpp.o
  Compiling .pioenvs/esp32s3/src/esphome/components/captive_portal/captive_portal.cpp.o
  Compiling .pioenvs/esp32s3/src/esphome/components/esp32/core.cpp.o
  Compiling .pioenvs/esp32s3/src/esphome/components/esp32/gpio.cpp.o
  Compiling .pioenvs/esp32s3/src/esphome/components/esp32/preferences.cpp.o
  Compiling .pioenvs/esp32s3/src/esphome/components/esp32_rmt_led_strip/led_strip.cpp.o
  In file included from src/esphome/components/esp32_rmt_led_strip/led_strip.h:12,
  from src/esphome/components/esp32_rmt_led_strip/led_strip.cpp:2:
  /data/cache/platformio/packages/framework-espidf/components/driver/deprecated/driver/rmt.h:18:2: warning: #warning "The legacy RMT driver is deprecated, please use driver/rmt_tx.h and/or driver/rmt_rx.h" [-Wcpp]
  18 | #warning "The legacy RMT driver is deprecated, please use driver/rmt_tx.h and/or driver/rmt_rx.h"
  | ^~~~~~~
  Compiling .pioenvs/esp32s3/src/esphome/components/i2s_audio_xiao/i2s_audio_xiao.cpp.o
  In file included from src/esphome/components/i2s_audio_xiao/i2s_audio_xiao.h:5,
  from src/esphome/components/i2s_audio_xiao/i2s_audio_xiao.cpp:1:
  /data/cache/platformio/packages/framework-espidf/components/driver/deprecated/driver/i2s.h:27:2: warning: #warning "This set of I2S APIs has been deprecated, please include 'driver/i2s_std.h', 'driver/i2s_pdm.h' or 'driver/i2s_tdm.h' instead. if you want to keep using the old APIs and ignore this warning, you can enable 'Suppress leagcy driver deprecated warning' option under 'I2S Configuration' menu in Kconfig" [-Wcpp]
  27 | #warning "This set of I2S APIs has been deprecated,
  | ^~~~~~~
  Compiling .pioenvs/esp32s3/src/esphome/components/i2s_audio_xiao/microphone/i2s_audio_microphone.cpp.o
  In file included from src/esphome/components/i2s_audio_xiao/microphone/../i2s_audio_xiao.h:5,
  from src/esphome/components/i2s_audio_xiao/microphone/i2s_audio_microphone.h:5,
  from src/esphome/components/i2s_audio_xiao/microphone/i2s_audio_microphone.cpp:1:
  /data/cache/platformio/packages/framework-espidf/components/driver/deprecated/driver/i2s.h:27:2: warning: #warning "This set of I2S APIs has been deprecated, please include 'driver/i2s_std.h', 'driver/i2s_pdm.h' or 'driver/i2s_tdm.h' instead. if you want to keep using the old APIs and ignore this warning, you can enable 'Suppress leagcy driver deprecated warning' option under 'I2S Configuration' menu in Kconfig" [-Wcpp]
  27 | #warning "This set of I2S APIs has been deprecated,
  | ^~~~~~~
  src/esphome/components/i2s_audio_xiao/i2s_audio_xiao.cpp: In member function 'virtual void esphome::i2s_audio_xiao::I2SAudioComponent::setup()':
  src/esphome/components/i2s_audio_xiao/i2s_audio_xiao.cpp:15:24: error: 'I2S_NUM_MAX' was not declared in this scope; did you mean 'I2S_NUM_1'?
  15 | if (next_port_num >= I2S_NUM_MAX) {
  | ^~~~~~~~~~~
  | I2S_NUM_1
  Compiling .pioenvs/esp32s3/src/esphome/components/i2s_audio_xiao/speaker/i2s_audio_speaker.cpp.o
  In file included from src/esphome/components/i2s_audio_xiao/speaker/../i2s_audio_xiao.h:5,
  from src/esphome/components/i2s_audio_xiao/speaker/i2s_audio_speaker.h:5,
  from src/esphome/components/i2s_audio_xiao/speaker/i2s_audio_speaker.cpp:1:
  /data/cache/platformio/packages/framework-espidf/components/driver/deprecated/driver/i2s.h:27:2: warning: #warning "This set of I2S APIs has been deprecated, please include 'driver/i2s_std.h', 'driver/i2s_pdm.h' or 'driver/i2s_tdm.h' instead. if you want to keep using the old APIs and ignore this warning, you can enable 'Suppress leagcy driver deprecated warning' option under 'I2S Configuration' menu in Kconfig" [-Wcpp]
  27 | #warning "This set of I2S APIs has been deprecated,
  | ^~~~~~~
  *** [.pioenvs/esp32s3/src/esphome/components/i2s_audio_xiao/i2s_audio_xiao.cpp.o] Error 1
  src/esphome/components/i2s_audio_xiao/speaker/i2s_audio_speaker.cpp: In static member function 'static void esphome::i2s_audio_xiao::I2SAudioSpeaker::player_task(void*)':
  src/esphome/components/i2s_audio_xiao/speaker/i2s_audio_speaker.cpp:54:24: error: 'I2S_MCLK_MULTIPLE_DEFAULT' was not declared in this scope; did you mean 'I2S_MCLK_MULTIPLE_512'?
  54 | .mclk_multiple = I2S_MCLK_MULTIPLE_DEFAULT,
  | ^~~~~~~~~~~~~~~~~~~~~~~~~
  | I2S_MCLK_MULTIPLE_512
  *** [.pioenvs/esp32s3/src/esphome/components/i2s_audio_xiao/speaker/i2s_audio_speaker.cpp.o] Error 1
  ========================= [FAILED] Took 75.61 seconds =========================

Note: I am using ESPHome Builder, instead of ESPHome, as ESPHome was not available.

[vanislman]

Thanks so much for your help and for those links. I now followed the instructions in your second link
https://github.com/formatBCE/Respeaker-Lite-ESPHome-integration
and got much further:

• I re-installed the 48K XMOS firmware and used the one on your page, just in case it was different from the Seed Studio one
• I installed the 48K .yaml file (after waiting forever for it to compile on my RPI4)
• I downloaded the new ESP firmware after compiling it
  Now the Respeaker Lite boots up properly, connects to Wifi, and I could configure it in HA.
  Everything seems to work, except the WAKEWORD:
• The HA setup page for the respeaker shows all standard entities, except the "Wakeword" drop-down is grayed out
• When I run the "Setup voice assistant" wizard, the blue LED blinks and I hear the welcome sound.
• When I click on "change wake word" the system doesn't show any wake words

Below some screenshots.

Any idea what I'm still missing?

(Oh, I didn't do any of the soldering you suggested for the buttons. I don't care about the USR button, and the MUTE button seems to work out-of-the-box)

[formatBCE]

Yeah, there's some trouble with wake word choosing. I fixed it in Koala, but didn't look into it in regular Respeaker sat. If you want different word, just set it as the only one in YAML and recompile (sorry to hear you have to do it on Pi, it's pain). By default it's using the best one, Ok Nabu. And it should work without wizard.

On the soldering - I recommend doing it. Software mute won't work if you don't, and respeaker_lite component isn't initializing properly if mute button not available. But you can remove related code if you don't want it to be there.
USR button is used to start pipeline without wake word, pause music etc.

[vanislman]

Thanks for the fast response again. I just tested with "ok nabu" - and it works!
No need to change it for now, I just want to do some testing and get some experience with Voice.
Thanks for all your great work and support !

[vanislman]

Hello @formatBCE , can you please tell me how to switch the default wake word in YAML to "Hey Jarvis"? I looked through the file but can't figure out how to change it.

[formatBCE]

Hello @formatBCE , can you please tell me how to switch the default wake word in YAML to "Hey Jarvis"? I looked through the file but can't figure out how to change it.

Looks like on latest ESPHome wizard works, and drop-down with wake words is enabled too..
If it's not - just remove "ok Nabu" code section from Microwakeword config, leave ok Jarvis on first place there, and rebuild.

[thematrixdev]

I am trying to add a media_player to play the start-up "beep" sound before listening.

media_player:
  - platform: i2s_audio_xiao
    id: mp3_player
    dac_type: external
    i2s_dout_pin: GPIO43
    mode: stereo

The IDE warns me This feature is only available with frameworks ['arduino'].

How to fix or is there any workaround?

Thanks.

[formatBCE]

Yes, it might. Case is, that GPIO pin, that is used for USR button, isn't in correct state (pulled up to 3.3v), if button isn't connected to it. So it sends "on" state, or flickering, all the time.

Did you solder USR pad to D2 and MUTE to D3 on Respeaker board, as it's stated in README?

If yes, check the connection. Reflow it.
If no - either do it, or delete all code related to user_button and mute_button from config, as not only it won't work, but will misbehave.

[vanislman]

You were right again ;-) I soldered the 2 buttons as suggested, and now the LED behaves as I think it's supposed to. And USR button performs the actions you said earlier.
Now I discovered one other issue: I can't get the wake word "Hey Jarvis" to work, it just doesn't respond. I tried many times, ran the setup assistant... no go. Hey Mycroft and OK Nabu work just fine, and I can switch between them.

[formatBCE]

Nice!

Do you have Hey Jarvis in the list? If so, but it doesn't hear you, try rebuilding with lower probability cut-off (check Nabu wake word for reference). All models except Nabu are built on synthetic datasets, and don't really work well, especially with some accents. I just use Nabu, everyone in the family got used to it.. :)

[vanislman]

I set the probability cut-off to 0.8 for Hey Jarvis. But still no way to use this wake word. I tried really hard to hide my german accent, no luck.
Both Nabu and Mycroft work fine, and I can switch between them. I can also see Hey Jarvis on the drop-down list and in the setup assistant. But it's just not recognized.
It seems to me that the system doesn't really have the code or look for Jarvis properly.
But don't worry about it, I can absolutely live with the 2 choices I have.

[formatBCE]

@vanislman but that's strange .. I use it with no problem at all... :(

[masman69]

Hello, I would like give you an update. It turns out that the media player issue was a misconfiguration in my homeassistant instance. Media Player works fine. My only issue is the User button which still does not work correctly. Thanks, Michael

…

On Mon, Dec 30, 2024, at 8:21 PM, Andrii Mitnovych wrote: I will check tho, what may cause the problem. Everything else is okay? — Reply to this email directly, view it on GitHub <#1609 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHPV6GXRL543YRHP6WDUBPD2IHWR3AVCNFSM6AAAAABNDE4BFCVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCNZQGEYTANQ>. You are receiving this because you commented.Message ID: ***@***.***>

[formatBCE]

Please check these lines:
https://github.com/formatBCE/Respeaker-Lite-ESPHome-integration/blob/main/esphome-respeaker-lite.yaml#L290-L292

Male sure it's the same in your config. It should be inverted.

[andrewv42280]

Your steps need updated. starting to regret buying this module. Error in your YAML at like 760

external_components: [source /config/esphome/andrews-speaker.yaml:760]

• source:
  type: git
  url: https://github.com/esphome/home-assistant-voice-pe
  ref: dev
  components:

  Could not find init.py file for component aic3204. Please check the component is defined by this source (search path: /data/external_components/3bf38338/esphome/components/aic3204/init.py).

  • aic3204
  • audio_dac
  • media_player
  • micro_wake_word
  • microphone
  • nabu
  • voice_assistant

[Vivida1]

Same problem here, an updated documentation would help.

[formatBCE]

Same problem here, an updated documentation would help.

Use https://github.com/formatBCE/Respeaker-Lite-ESPHome-integration

[nfs0619]

Thank you for your feedback! We are aware of the documentation issue and have arranged for updates. We sincerely apologize for any inconvenience this may have caused. If you encounter any other issues, please feel free to contact our technical support team at techsupport@seeed.io. We are here to assist you!

[andrewv42280]

I will say, I have tried several projects like this and this is the best. I even tried my own and just did not go far enough in depth like the respeaker lite did. Thank you this is a great product. I will more than likely be buying more of these.

[nfs0619]

Thank you for your support! If you encounter any issues in the future, feel free to reach out to us.

[lottox17]

Hello, i have a question before buying the ReSpeaker Lite Voice Assistant Kit : Do we agree that i can use it to cast some music on the ouput 3,5mm Jack port ? i would like to create a connected multiroom speaker with an old speaker using the ReSpeaker and its output jack port.

Thank you

[formatBCE]

IDK why wouldn't you have both?
I use my Samsung speaker with great bass for music, and Koala (which is pretty similar to VPE) for assist. Just get used to different devices for music playback and assistant responses. :)

[lottox17]

I am not sure to understand your answer. I don't know Koala. In your previous message, you explained me there is no open-source solution open-source combining Multiroom + voice control using an old speaker on 3.5mm Jack. Do you have any documentation on Koala? Thank you for your help

[formatBCE]

I meant this: https://github.com/formatBCE/Koala-Satellite

Basically having separate unit for voice communication, announcements etc. And have something with better speaker for music.

[lottox17]

Ok, i understand better. You don't have any voice control issue if you have music in the same time on your samsung speaker ?

[formatBCE]

It depends on how far the satellite is from the music speaker. Mine are on different sides of the room, and i rarely have issues. When TV is loud though, it can introduce "speech interference", and because there's no "beam forming" or "voice printing", it's hard to give commands when someone is talking on background. But music is pretty okay.

[Skyman81]

I still don't understand why a Sonoff Basicr2 is needed....? For what purpose?

[nfs0619]

Hello, thanks for your question!
In this wiki, we use the Sonoff BasicR2 as an example to demonstrate how to use the ReSpeaker Lite Voice Assistant Kit to control smart devices through Home Assistant.
You can choose other suitable devices to achieve similar functionality, and Sonoff BasicR2 is not required.

[Skyman81]

So… it’s not required…

[iarj]

YAML file from this article can't be compiled anymore on 2025.5 I'll be really interested to have an updated version of this wiki

[formatBCE]

Here you go: https://github.com/formatBCE/Respeaker-Lite-ESPHome-integration