Guidance (or actual code) for how to detect when speech has ended from an I2S input (or from the output)

[cmcooper1971]

Hi All,

Further to my Show and Tell post here - https://github.com/pschatzmann/arduino-audio-tools/discussions/1629

Presently, when the I2S Microphone is disabled, I have set it to automatically play a short static noise track (MP3). However, the users have asked if this can be re-designed to trigger when they stop speaking, as they do not wish to keep enabling and disabling the Microphone input.

For information, when a Storm Trooper stops speaking in the films, they have this static noise at the end, which is what I have tried to replicate.

Code which currently triggers the static.mp3 files as follows:

// Play chosen inputs

if (playerControlFlag == true && micControlFlag == false) {

	if (staticNoise == true && proSelect == 0) {

		File root = SD.open("/");

		audioFile = SD.open("/static.mp3");

		if (audioFile) {

			debugln("File opened successfully");
			enablePlayerMP3();

		}

		else {

			debugln("Failed to open file");

		}

		staticNoise == false;

	}

	size_t copied = mp3.copy();			// Start copy and check data byte size

	if (copied == 0  && playAudioFile == true) {

		// No more data to copy, end of file reached

		debugln("End of file reached.");

		disablePlayer();
		audioFile.close();
		playAudioFile = false;

	}

}

if (micControlFlag == true && playerControlFlag == false) {

	mic.copy();

}

Thanks in anticipation.

[pschatzmann]

I guess you talk about the "annoying feature" of the new ESP32 I2S API which provides some nose, if no audio data is provided.
So you either need to stop i2s or just continue to copy silence or data from the microphone to avoid that you run out of audio data...

You could determine if you have any audio input by measuring the volume...

Or you can roll back to a pre 3.0 ESP32 Arduino Core...

[cmcooper1971]

Sorry, not quite, that was solved with your help if you recall as I now stop the MP3WAV player after each track.

This is a new requirement, to play a audio clip from the movie, each time the user stops speaking through the I2S microphone. So I need to somehow detect speech, not noise and after a fixed delay or count, trigger the track to play?

[pschatzmann]

As mentioned above, the best and easiest way would be to just measure the volume.

You could try to build and train an AI model to distinguish between noise and speech, but that that would be quite involving...

[cmcooper1971]

Thanks Phil, I'll see what I come up with. If I solve it, I will post it here for others.

[cmcooper1971]

Hi Phil,

I am really struggling on how to implement the VolumeMeter class into my sketch. I thought I understood how you call streams and link them together, but what ever I try on this class, I get all sorts of issues.

These are my parameters as they are working right now, with my VolumeMeter efforts removed:

// Audio object

AudioInfo info(44100, 2, 16);

AudioInfo infoMic(44100, 2, 16);

// Audio streams in and out

I2SStream in;

I2SStream out;

ConverterFillLeftAndRight<int16_t> left_and_right(RightIsEmpty);

ConverterStream<int16_t> converter(in, left_and_right);

//AudioEffectStream effects(out);

EncodedAudioStream decoderWav(&out, new WAVDecoder());			// Decoding stream
EncodedAudioStream decoderMP3(&out, new MP3DecoderHelix());		// Decoding stream

StreamCopy wav;
StreamCopy mp3;
StreamCopy mic(out, converter);

This is my I2S Microphone function:

// Enable MIC & disable Player

void enableMIC() {

	// Stop player

	decoderWav.end();
	decoderMP3.end();
	mp3.end();
	in.end();
	out.end();

	// Start I2S in

	AudioLogger::instance().begin(Serial, AudioLogger::Warning);
	debugln("Starting I2S...");

	auto config_in = in.defaultConfig(RX_MODE);
	config_in.copyFrom(infoMic);
	config_in.i2s_format = I2S_STD_FORMAT;
	config_in.is_master = true;
	config_in.port_no = 0;
	config_in.pin_bck = I2S_BCLK_0;
	config_in.pin_ws = I2S_LRC_0;
	config_in.pin_data = I2S_DIN_0;
	in.begin(config_in);

	// Start I2S out

	auto config_out = out.defaultConfig(TX_MODE);
	config_out.copyFrom(infoMic);
	config_out.i2s_format = I2S_STD_FORMAT;
	config_out.is_master = true;
	config_out.port_no = 1;
	config_out.pin_bck = I2S_BCLK_1;
	config_out.pin_ws = I2S_LRC_1;
	config_out.pin_data = I2S_DOUT_1;
	config_out.pin_mck = I2S_MCLK_1;
	out.begin(config_out);

	// Start the microphone stream

	mic.begin(out, converter);

	debugln("Mic Enabled");
	digitalWrite(amberLED, LOW);
	playerControlFlagChanged = false;

	displayVolumeRegister(false);

} // Close function

And this is how it is called in the loop:

	if (micControlFlag == true && playerControlFlag == false) {

		mic.copy();

		// Just print the max amplitude

		//Serial.print("Volume: ");
		//Serial.print(vLevel.volume());
		//Serial.print(" left: ");
		//Serial.print(vLevel.volume(0));
		//Serial.print(" right: ");
		//Serial.println(vLevel.volume(1));

	}

However, what ever I try to get the I2S Microphone input or output to the VolumeMeter so I can run some basic logic to detect when speech has ended, I simply am not understanding it.

I have followed your various posts to edit the above, but alas no joy, such as:

https://github.com/pschatzmann/arduino-audio-tools/wiki/Determining-the-Volume

and

https://github.com/pschatzmann/arduino-audio-tools/wiki/Volume-Control

I hate to ask as I prefer to solve myself, especially when you have written so much documentation, but I'm not understanding something important!

[pschatzmann]

Just put the VolumeMeter into your input chain: e.g. I2SStream -> VolumeMeter -> ConverterStream

[cmcooper1971]

Thanks for replying this late!

I have, as follows, but the output shows 0, although the Microphone still works.

New parameters

VolumeMeter vMeter;

StreamCopy mic(vMeter, converter);

Enabled Mic function, updated as follows:

	// Start I2S in

	AudioLogger::instance().begin(Serial, AudioLogger::Warning);
	debugln("Starting I2S...");

	auto config_in = in.defaultConfig(RX_MODE);
	config_in.copyFrom(infoMic);
	config_in.i2s_format = I2S_STD_FORMAT;
	config_in.is_master = true;
	config_in.port_no = 0;
	config_in.pin_bck = I2S_BCLK_0;
	config_in.pin_ws = I2S_LRC_0;
	config_in.pin_data = I2S_DIN_0;
	in.begin(config_in);

	// Start I2S out

	auto config_out = out.defaultConfig(TX_MODE);
	config_out.copyFrom(infoMic);
	config_out.i2s_format = I2S_STD_FORMAT;
	config_out.is_master = true;
	config_out.port_no = 1;
	config_out.pin_bck = I2S_BCLK_1;
	config_out.pin_ws = I2S_LRC_1;
	config_out.pin_data = I2S_DOUT_1;
	config_out.pin_mck = I2S_MCLK_1;
	out.begin(config_out);

	// Start the VolumeMeter

	vMeter.begin(infoMic);

	// Start the microphone stream

	mic.begin(out, converter);

Loop updated for output, but all data comes back as 0

	if (micControlFlag == true && playerControlFlag == false) {

		mic.copy();

		// Print the max amplitude using VolumeMeter

		int16_t volume = vMeter.volume();
		Serial.print("Volume: ");
		Serial.println(volume);

		// Optionally, print left and right channel volumes

		int16_t leftVolume = vMeter.volume(0);
		int16_t rightVolume = vMeter.volume(1);
		Serial.print("Left Channel Volume: ");
		Serial.println(leftVolume);
		Serial.print("Right Channel Volume: ");
		Serial.println(rightVolume);

	}

[pschatzmann]

You define StreamCopy mic(vMeter, converter); and then you just ignore and overwrite this later with mic.begin(out, converter);

[cmcooper1971]

That's the bit I don't understand then, as I still want the output to the Amplifier. If I use the define, I do get the data, but nothing through to the amp.

Updated as follows:

	// Start the VolumeMeter

	vMeter.begin(infoMic);

	// Start the microphone stream

	mic.begin(vMeter, converter);

The loop now displays the serial data which clearly changes when I speak. But nothing to the amp like I said previously.

[pschatzmann]

I told you to create the following chain: I2SStream -> VolumeMeter -> ConverterStream -- copy > I2SStream
Instead you did this I2SStream -> ConverterStream -- copy > VolumeMeter

[cmcooper1971]

Yes you did.... That's why in my world, you are Yoda.

Updated and it now works as follows, for anyone else who doesn't listen to the master.

// Audio streams in and out

I2SStream in;

I2SStream out;

ConverterFillLeftAndRight<int16_t> left_and_right(RightIsEmpty);

VolumeMeter vMeter(in);

ConverterStream<int16_t> converter(vMeter, left_and_right);

EncodedAudioStream decoderWav(&out, new WAVDecoder());			// Decoding stream
EncodedAudioStream decoderMP3(&out, new MP3DecoderHelix());		// Decoding stream

StreamCopy wav;
StreamCopy mp3;
StreamCopy mic(out, converter);

Enable Mic function changes:

// Start the VolumeMeter

vMeter.begin(infoMic);

// Start the microphone stream

mic.begin(out, converter);