[Question] Play audio files simultaneously and stream UDP sound half duplex

[Theagainmen]

Hi all,

First of all thanks for this great library!

Question 1

I'm wondering if it's possible to play 2+ audio files simultaneously from for example the SD card of SPIFFS? Currently I've only found examples that are able to play 1 sound at a time.

Question 2

Is it possible to send and receive data over UDP? I've found 2 examples with UDP send and receive.

I understand and assume it's not possible to send and receive simultaneously, but it would be nice if it was possible to have both abilities on a single ESP32 and just switch to whatever is needed (either send or receive) on the go. An example setup would be:

Setup 1:
ESP32 with a microphone and a speaker

Setup 2:
ESP32 with a microphone and a speaker

When setup 1 starts talking into the microphone it sends the UDP data to setup 2. Once setup 1 is done talking and setup 2 starts talking it sends this over to setup 1. Basically a bidirectional half duplex (full would be nice, but not expected of course) call/walkie-talkie system over UDP.

Thanks for any help :).

[pschatzmann]

In order to play 2 sources at the same time you need to mix the two signals, so I suggest that you read the corresponding chapter in the wiki.

UDP is a lossy protocol and it might be possible to get things to work if you keep the sample rate very low. You might want to try it in combination with a CODEC.

However I would suggest that you also look into ESP-Now!

[Theagainmen]

Awesome, thanks for your reply, I'll look into the UDP part and ESP-Now later.

I'm now playing files from the SD card and was able to run the input mixer, I noticed that the Wiki mentions Note that this is however only working if the basic audio input is in PCM format and is the same across all inputs:, this means I can use the mixer with .wav files but no .mp3 files right? Or no files at all?

And if I can use .wav files how would that exactly work in code? I currently don't see how to edit the mixer code to take the file from the SD card as input.

[Theagainmen]

I still haven't really figured this out, basically I want to play sound effects through the ESP32, each button press will play a sound. So even when multiple buttons are pressed they need to play up to 2-5 audio sound effects.

[pschatzmann]

For mixing on the input side you actually would need to use PCM files.

[Theagainmen]

Thanks, I found a converter online and have converted some of my audio files to PCM files (wow these raw files are huge in size!).

I can't figure out how to replace these parts of the code, from the inputmixer example:

GeneratedSoundStream<int16_t> sound1(sineWave1);             // Stream generated from sine wave
GeneratedSoundStream<int16_t> sound2(sineWave2);             // Stream generated from sine wave

// [...]

  mixer.add(sound1);
  mixer.add(sound2);

so that the mixer actually uses my PCM files as input.

This is one of the things I've came up so far, I know this is wrong, but after quite some trial and error I must say that I'm unable to figure out how to replace the generated sound (for the input mixer) by the SD card PCM file. (original: inputmixer, decoders: wiki)

#include <SPI.h>
#include <SD.h>
#include "AudioTools.h"
#include "AudioLibs/AudioKit.h"
#include "AudioCodecs/Codec8Bit.h"


const int chipSelect=PIN_AUDIO_KIT_SD_CARD_CS;
AudioKitStream i2s; // final output of decoded stream
EncodedAudioStream decoder(&i2s, new Decoder8Bit()); // Decoding stream
File audioFile1;
File audioFile2;

// Custom mixer stuff
uint16_t sample_rate=32000;
uint8_t channels = 2;   
InputMixer<int16_t> mixer;
StreamCopy copier(i2s, mixer);                             // copies sound into i2s

void setup(){
  Serial.begin(115200);
  AudioLogger::instance().begin(Serial, AudioLogger::Info);  

  // setup audiokit before SD!
  auto config = i2s.defaultConfig(TX_MODE);
  config.sd_active = true;
  i2s.begin(config);

  // setup file
  SD.begin(chipSelect);
  audioFile1 = SD.open("/alarm.pcm");
  audioFile2 = SD.open("/gameover.pcm");

  mixer.add(audioFile1);
  mixer.add(audioFile2);

  // setup I2S based on sampling rate provided by decoder
  decoder.begin();

  // begin copy
  //copier.setCheckAvailableForWrite(false);
  //copier.begin(decoder, audioFile1);

}

void loop(){
  if (!copier.copy()) {
    stop();
  }
}

[pschatzmann]

I would have expected that something like the following should work. You can decrease the file size by moving to mono and decreasing the sample size:

#include <SPI.h>
#include <SD.h>
#include "AudioTools.h"
#include "AudioLibs/AudioKit.h"

const int chipSelect=PIN_AUDIO_KIT_SD_CARD_CS;
AudioKitStream i2s; // final output of decoded stream
File audioFile1;
File audioFile2;

// Custom mixer stuff
uint16_t sample_rate=32000;
uint8_t channels = 2;   
InputMixer<int16_t> mixer;
StreamCopy copier(i2s, mixer);                             // copies sound into i2s

void setup(){
  Serial.begin(115200);
  AudioLogger::instance().begin(Serial, AudioLogger::Info);  

  // setup audiokit before SD!
  auto config = i2s.defaultConfig(TX_MODE);
  config.sample_rate = sample_rate;
  config.channels = channels;
  config.sd_active = true;
  i2s.begin(config);

  // setup file
  SD.begin(chipSelect);
  audioFile1 = SD.open("/alarm.pcm");
  audioFile2 = SD.open("/gameover.pcm");

  mixer.add(audioFile1);
  mixer.add(audioFile2);

}

void loop(){
  copier.copy();
}

[Theagainmen]

Thanks for your reply once again, I'd love to try it and let you know if it worked, but for some reason I keep running into errors. Even with the default unchanged inputmixer example. I've been trying a lot of things and I really don't understand what I did different than a few days ago when everything was just working fine.

Error with the default unchanged inputmixer example:

In file included from .pio\libdeps\esp32dev\audio-tools\src/AudioLibs/AudioKit.h:3:0,
                 from src\streams-generator_inputmixer-audiokit.cpp:9:
.pio\libdeps\esp32dev\audiokit\src/AudioKitHAL.h: In member function 'i2s_config_t AudioKitConfig::i2sConfig()':
.pio\libdeps\esp32dev\audiokit\src/AudioKitHAL.h:129:34: warning: narrowing conversion of 'AudioKitConfig::sampleRate()' from 'uint32_t {aka unsigned int}' to 'int' inside { } [-Wnarrowing]
         .sample_rate = sampleRate(),
                                  ^
src\streams-generator_inputmixer-audiokit.cpp: At global scope:
src\streams-generator_inputmixer-audiokit.cpp:17:22: error: narrowing conversion of '1024' from 'int' to 'unsigned char' inside { } [-Wnarrowing]
 InputMixer<uint16_t> mixer;
                      ^
In file included from .pio\libdeps\esp32dev\audio-tools\src/AudioTools/AudioPrint.h:7:0,
                 from .pio\libdeps\esp32dev\audio-tools\src/AudioFilter/Equilizer.h:3,
                 from .pio\libdeps\esp32dev\audio-tools\src/AudioTools.h:16,
                 from src\streams-generator_inputmixer-audiokit.cpp:8:
.pio\libdeps\esp32dev\audio-tools\src/AudioTools/AudioStreams.h: In constructor 'audio_tools::InputMixer<T>::InputMixer() [with T = short unsigned int]':
.pio\libdeps\esp32dev\audio-tools\src/AudioTools/AudioStreams.h:1341:5: error: narrowing conversion of '1024' from 'int' to 'unsigned char' inside { } [-Wnarrowing]
     InputMixer() = default;
     ^
src\streams-generator_inputmixer-audiokit.cpp: At global scope:
src\streams-generator_inputmixer-audiokit.cpp:17:22: note: synthesized method 'audio_tools::InputMixer<T>::InputMixer() [with T = short unsigned int]' first required here
 InputMixer<uint16_t> mixer;
                      ^
*** [.pio\build\esp32dev\src\streams-generator_inputmixer-audiokit.cpp.o] Error 1

platformio.ini

[env:esp32dev]
platform = espressif32
board = esp32dev
framework = arduino

board_build.f_cpu = 240000000L      ; 240MHz
upload_speed = 921600           
monitor_speed = 115200


; From the Wiki https://github.com/pschatzmann/arduino-audio-tools/wiki/Working-with-PlatformIO, I have board lyrat_v4_3 = 1
lib_ldf_mode = deep+
monitor_filters = esp32_exception_decoder
build_flags =  -DCORE_DEBUG_LEVEL=5 -DAUDIOKIT_BOARD=1 -DUSE_HELIX

lib_deps = 
    https://github.com/pschatzmann/arduino-audio-tools
    https://github.com/pschatzmann/arduino-audiokit
    https://github.com/pschatzmann/arduino-libhelix

[pschatzmann]

Just look at the error message: InputMixer<uint16_t> is nonsense!
The audio data needs to be signed...

[Theagainmen]

Hmh, but it is signed!

InputMixer<int16_t> mixer;

I'm using an exact copy of the example:

#include "AudioTools.h"
#include "AudioLibs/AudioKit.h"

uint16_t sample_rate=32000;
uint8_t channels = 2;                                      // The stream will have 2 channels 
SineWaveGenerator<int16_t> sineWave1(32000);                // subclass of SoundGenerator with max amplitude of 32000
SineWaveGenerator<int16_t> sineWave2(32000);                // subclass of SoundGenerator with max amplitude of 32000
GeneratedSoundStream<int16_t> sound1(sineWave1);             // Stream generated from sine wave
GeneratedSoundStream<int16_t> sound2(sineWave2);             // Stream generated from sine wave
InputMixer<int16_t> mixer;
AudioKitStream out; 
StreamCopy copier(out, mixer);                             // copies sound into i2s

// Arduino Setup
void setup(void) {  
  // Open Serial 
  Serial.begin(115200);
  AudioLogger::instance().begin(Serial, AudioLogger::Info);

  mixer.add(sound1);
  mixer.add(sound2);

  // start I2S
  Serial.println("starting I2S...");
  auto config = out.defaultConfig(TX_MODE);
  config.sample_rate = sample_rate; 
  config.channels = channels;
  config.bits_per_sample = 16;
  out.begin(config);

  // Setup sine wave
  sineWave1.begin(channels, sample_rate, N_B4);
  sineWave2.begin(channels, sample_rate, N_E4);
  Serial.println("started...");
}

// Arduino loop - copy sound to out 
void loop() {
  copier.copy();
}

[pschatzmann]

But this does not correspond with what you have shown in the error message:
src\streams-generator_inputmixer-audiokit.cpp:17:22: error: narrowing conversion of '1024' from 'int' to 'unsigned char' inside { } [-Wnarrowing]
InputMixer<uint16_t> mixer;
^

[Theagainmen]

Ah my bad, I copy-pasted an error from where I was playing around and changed the datatype indeed, the error is however almost identical for the actual example one:

In file included from .pio\libdeps\esp32dev\audio-tools\src/AudioLibs/AudioKit.h:3:0,
                 from src\streams-generator_inputmixer-audiokit.cpp:9:
.pio\libdeps\esp32dev\audiokit\src/AudioKitHAL.h: In member function 'i2s_config_t AudioKitConfig::i2sConfig()':
.pio\libdeps\esp32dev\audiokit\src/AudioKitHAL.h:129:34: warning: narrowing conversion of 'AudioKitConfig::sampleRate()' from 'uint32_t {aka unsigned int}' to 'int' inside { } [-Wnarrowing]
         .sample_rate = sampleRate(),
                                  ^
src\streams-generator_inputmixer-audiokit.cpp: At global scope:
src\streams-generator_inputmixer-audiokit.cpp:17:21: error: narrowing conversion of '1024' from 'int' to 'unsigned char' inside { } [-Wnarrowing]
 InputMixer<int16_t> mixer;
                     ^
In file included from .pio\libdeps\esp32dev\audio-tools\src/AudioTools/AudioPrint.h:7:0,
                 from .pio\libdeps\esp32dev\audio-tools\src/AudioFilter/Equilizer.h:3,
                 from .pio\libdeps\esp32dev\audio-tools\src/AudioTools.h:16,
                 from src\streams-generator_inputmixer-audiokit.cpp:8:
.pio\libdeps\esp32dev\audio-tools\src/AudioTools/AudioStreams.h: In constructor 'audio_tools::InputMixer<T>::InputMixer() [with T = short int]':
.pio\libdeps\esp32dev\audio-tools\src/AudioTools/AudioStreams.h:1341:5: error: narrowing conversion of '1024' from 'int' to 'unsigned char' inside { } [-Wnarrowing]
     InputMixer() = default;
     ^
src\streams-generator_inputmixer-audiokit.cpp: At global scope:
src\streams-generator_inputmixer-audiokit.cpp:17:21: note: synthesized method 'audio_tools::InputMixer<T>::InputMixer() [with T = short int]' first required here
 InputMixer<int16_t> mixer;
                     ^
*** [.pio\build\esp32dev\src\streams-generator_inputmixer-audiokit.cpp.o] Error 1

[pschatzmann]

I can reproduce your issue in PlatformIO, but somehow it does not make any sense to me.
I do not see where the 1024 would come from...

[Theagainmen]

Same.. the error doesn't make sense to me and it seems to happen at random. After some playing around and re-compiling the code a few times while changing this line (in platformio.ini at lib_deps):

https://github.com/pschatzmann/arduino-audio-tools

to

https://github.com/pschatzmann/arduino-audio-tools#v0.9.2

seem to have fixed it. I've never seen this behaviour before.. does it also work for you when you change this line a few times and re-compile?

[pschatzmann]

It seems that the initializer_list in the Vector was causing this.
I have deactivated this functionality..

[Theagainmen]

Thanks a lot! The SD mixer example works great! It seems to only work at max with 2 files at the same time, with 3 simultaneously the audio becomes very bad quickly. Right now I'm changing the audio files on the-go with simply calling a new SD open:

void loop()
{
  copier.copy();

  unsigned long currentMillis = millis();

  if (currentMillis - previousMillis >= interval) {    // Re-play the sound every X seconds, audioFile2 remains unchanged
    previousMillis = currentMillis;
    audioFile1 = SD.open("/sound.pcm");
  }
}

Sometimes the changing on the go results in a loud short crack sound, the same sound you hear when you unplug or plug an AUX jack in a speaker that's already on. Am I doing this the right way, or is there a better/more efficient way to change the mixed audio files on the go?

[pschatzmann]

You can try to reduce the sample rate if you want to support more files.
In order to reduce the pops you can try to turn the volume down before doing any changes or try out the PoppingSoundRemover converter or change your files so that the start and end around 0

[veitk]

Hi, since i will do something similar: i guess placing the samples as header files as you did in the memory-stream will improve the performance ... am i right? i want to mix a few samples on button press as well for a game.

By the way: do you have a link to a good wav-to-header file converter? bitluni's is converting to signed not unsigned and only 8 bit, wav2sketch is producing integer not char arrays ...

[pschatzmann]

I think it is just a convenient way to make data available to a sketch w/o needing any additional hardware (e.g. sd drive).

I usually use Audacity to convert data: You can select the output format e.g. raw headerless, signed 8 bit or signed 16 bit.
The ESP32 can't handle 8 bit data directly, so you can use a Codec to convert 8 bit to 16 bit.

I am using xxd to convert the audio file to a c file, that can be compiled see Wiki. This produces a char[] which is needed by my framework. If you want to process/interpret the data, you can just cast to the converted format. So if you converted the data to signed 16 bit you can cast to (int16_t*)!

[pschatzmann]

ps I just had a look at the docu for wav2sketch: wav2sketch -16 is producing int16_t data, so if you want to process it in the audio tools just cast it to (uint8_t*). If you don't provide any option it produces 8 bit µ-law: so you will need to use the ulow decoder

[laserir]

Hey guys, did you figure it out how to play sounds from the SD card simultaneously? I have a similar setup, playing a longer background music and from time to time (triggered via mqtt/serial) short sounds must be mixed in. Any ideas? :)

PS: I'm using the AudioKit

[pschatzmann]

You can configure the InputMixer dynamically (with setWeight()); just set the weight before you play the short playing sound and when it has finished set it to 0 !

And if should also be easy to resolve this with an OutputMixer....

[vkjambit]

i didn't try it with a SD card, i had my two sounds in the flash memory. this worked pretty well.