Automatic silence generation when I2S input clock/ws stops

[jamesbaber]

I have a fairly complex project but have distilled it down into the attached example code which has the same issue.

I have two I2S inputs, one from a WM8782 ADC and one from a CP2615 (USB->I2S interface).
The two inputs are being mixed together and output.
When the CP2615 is not streaming from the attached computer, it stops transmitting data (LRCLK/WS stops).
When my sketch stops receiving I2S data from one of the inputs, the output halts altogether which is not the desired behaviour as I still need to hear the signal from the WM8782.
There is no viable setting that makes the CP chip always output the LRCLK.
I have tried querying StreamCopy's .available() before my copy operations (it returns 3072) but it doesn't report any reduced data availability even right up to the point where .copy() hangs waiting for more I2S data to be received.

I can see functions for generating silence but I haven't found any examples where these are used and I'm not sure how to determine when silence needs to be generated, or even if that is a sensible solution to explore.

#include "AudioTools.h"

AudioInfo systemFormat(44100, 2, 16);
CsvOutput<int16_t> csvStream(Serial);

I2SStream CPInI2S; // Intermittent WS
I2SStream WMInI2S; // Constant signal

OutputMixer<int16_t> mixer(csvStream, 2);

StreamCopy copyCPToMixer(mixer, CPInI2S);
StreamCopy copyWMToMixer(mixer, WMInI2S);

void setup() {
  Serial.begin(500000);
  AudioLogger::instance().begin(Serial, AudioLogger::Warning);

  Serial.println("Setting up I2S from WM8782...");
  auto I2SWMCfg = WMInI2S.defaultConfig(RX_MODE);
  I2SWMCfg.copyFrom(systemFormat);
  I2SWMCfg.i2s_format = I2S_STD_FORMAT;
  I2SWMCfg.is_master = true;
  I2SWMCfg.use_apll = true;
  I2SWMCfg.port_no = 0;
  I2SWMCfg.pin_mck = 0;
  I2SWMCfg.pin_bck = 4;
  I2SWMCfg.pin_ws = 5;
  I2SWMCfg.pin_data = 18;
  WMInI2S.begin(I2SWMCfg);

  Serial.println("Setting up I2S from CP2615...");
  auto I2SCPCfg = CPInI2S.defaultConfig(RX_MODE);
  I2SCPCfg.copyFrom(systemFormat);
  I2SCPCfg.i2s_format = I2S_STD_FORMAT;
  I2SCPCfg.is_master = false;
  I2SCPCfg.use_apll = false;
  I2SCPCfg.port_no = 1;
  I2SCPCfg.pin_mck = 3;
  I2SCPCfg.pin_bck = 36;
  I2SCPCfg.pin_ws = 12;
  I2SCPCfg.pin_data = 39;
  CPInI2S.begin(I2SCPCfg);

  Serial.println("Setting up mixer...");
  mixer.begin();

  Serial.println("Setting csv stream...");
  csvStream.begin(systemFormat);

  Serial.println("Done.");
}

void loop() {
  copyCPToMixer.copy();
  copyWMToMixer.copy();
}

[pschatzmann]

To keep things simple, I am using blocking reads and writes for I2S and just provide a fixed value as result for available() and availableForWrite(). The ESP32 API does not provide any functionality to read the available bytes in the buffer.

I don't see any easy, portable way to resolve this: But you can try to use the following approach:

• I have added the methods setWaitTimeReadMs and setWaitTimeWriteMs to the I2S driver. So you can e.g. call i2s.driver()->setWaitTimeReadMs(value) to limit the wait time for the read or write methods (so they will not be blocking any more).
• Implement you own input stream wrapper class, e.g. based on the CallbackStream where you just return silence when there is no data returned any more.
• Alternatively you can reconfigure the mixer to ignore the input w/o any data

Please note that this is a kind of hack and will only work on the ESP32. Compiling this sketch on an other platform will give compile errors...

[pschatzmann]

Oh, I just noticed that you use an output mixer and my answer was assuming you used an input mixer. So this should get quite easy then

• setWaitTimeReadMs in setup
• call flushMixer() at the end of the loop to indicate that this is the end of the mixing step

[jamesbaber]

Thanks for your quick replies.

I can confirm that things are now non-blocking.
However, my copyWMToMixer.copy() is causing the following message:
[E] StreamCopy.h : 349 - write to target has failed after 22 retries! (1 bytes)
and the WM audio is still not making it through when the CP is inactive.

I've tried setWaitTimeReadMs values of 8, 512 and a few in between with the same result.

Here's the a log of a full loop (with a print before each of the three functions).

Copying CP to mixer
[D] StreamCopy.h : 97 - copy 
[D] StreamCopy.h : 177 - available: 3072
[I] StreamCopy.h : 143 - StreamCopy::copy  1 -> 0 -> 0 bytes - in 0 hops
[D] StreamCopy.h : 148 - size_t audio_tools::StreamCopyT<T>::copy() [with T = unsigned char; size_t = unsigned int]
Copying WM to mixer
[D] StreamCopy.h : 97 - copy 
[D] StreamCopy.h : 177 - available: 3072
[D] StreamCopy.h : 332 - write: 1
[D] AudioOutput.h : 363 - write idx 0: 1
[D] Buffers.h : 76 - writeArray 0 -> 0
[D] StreamCopy.h : 338 - write: 1 -> 0
[D] AudioOutput.h : 363 - write idx 1: 1
[D] Buffers.h : 76 - writeArray 0 -> 0
[D] AudioOutput.h : 393 - flush
[D] StreamCopy.h : 338 - write: 1 -> 0
[I] StreamCopy.h : 356 - try write  - 2 (open 1 bytes) 
[D] AudioOutput.h : 363 - write idx 0: 1
[D] Buffers.h : 76 - writeArray 0 -> 0
[D] StreamCopy.h : 338 - write: 1 -> 0
[I] StreamCopy.h : 356 - try write  - 3 (open 1 bytes) 
[D] AudioOutput.h : 363 - write idx 1: 1
[D] Buffers.h : 76 - writeArray 0 -> 0
[D] AudioOutput.h : 393 - flush
[D] StreamCopy.h : 338 - write: 1 -> 0
[I] StreamCopy.h : 356 - try write  - 4 (open 1 bytes) 
[D] AudioOutput.h : 363 - write idx 0: 1
[D] Buffers.h : 76 - writeArray 0 -> 0
[D] StreamCopy.h : 338 - write: 1 -> 0
[I] StreamCopy.h : 356 - try write  - 5 (open 1 bytes) 
[D] AudioOutput.h : 363 - write idx 1: 1
[D] Buffers.h : 76 - writeArray 0 -> 0
[D] AudioOutput.h : 393 - flush
[D] StreamCopy.h : 338 - write: 1 -> 0
[I] StreamCopy.h : 356 - try write  - 6 (open 1 bytes) 
[D] AudioOutput.h : 363 - write idx 0: 1
[D] Buffers.h : 76 - writeArray 0 -> 0
[D] StreamCopy.h : 338 - write: 1 -> 0
[I] StreamCopy.h : 356 - try write  - 7 (open 1 bytes) 
[D] AudioOutput.h : 363 - write idx 1: 1
[D] Buffers.h : 76 - writeArray 0 -> 0
[D] AudioOutput.h : 393 - flush
[D] StreamCopy.h : 338 - write: 1 -> 0
[I] StreamCopy.h : 356 - try write  - 8 (open 1 bytes) 
[D] AudioOutput.h : 363 - write idx 0: 1
[D] Buffers.h : 76 - writeArray 0 -> 0
[D] StreamCopy.h : 338 - write: 1 -> 0
[I] StreamCopy.h : 356 - try write  - 9 (open 1 bytes) 
[D] AudioOutput.h : 363 - write idx 1: 1
[D] Buffers.h : 76 - writeArray 0 -> 0
[D] AudioOutput.h : 393 - flush
[D] StreamCopy.h : 338 - write: 1 -> 0
[I] StreamCopy.h : 356 - try write  - 10 (open 1 bytes) 
[D] AudioOutput.h : 363 - write idx 0: 1
[D] Buffers.h : 76 - writeArray 0 -> 0
[D] StreamCopy.h : 338 - write: 1 -> 0
[I] StreamCopy.h : 356 - try write  - 11 (open 1 bytes) 
[D] AudioOutput.h : 363 - write idx 1: 1
[D] Buffers.h : 76 - writeArray 0 -> 0
[D] AudioOutput.h : 393 - flush
[D] StreamCopy.h : 338 - write: 1 -> 0
[I] StreamCopy.h : 356 - try write  - 12 (open 1 bytes) 
[D] AudioOutput.h : 363 - write idx 0: 1
[D] Buffers.h : 76 - writeArray 0 -> 0
[D] StreamCopy.h : 338 - write: 1 -> 0
[I] StreamCopy.h : 356 - try write  - 13 (open 1 bytes) 
[D] AudioOutput.h : 363 - write idx 1: 1
[D] Buffers.h : 76 - writeArray 0 -> 0
[D] AudioOutput.h : 393 - flush
[D] StreamCopy.h : 338 - write: 1 -> 0
[I] StreamCopy.h : 356 - try write  - 14 (open 1 bytes) 
[D] AudioOutput.h : 363 - write idx 0: 1
[D] Buffers.h : 76 - writeArray 0 -> 0
[D] StreamCopy.h : 338 - write: 1 -> 0
[I] StreamCopy.h : 356 - try write  - 15 (open 1 bytes) 
[D] AudioOutput.h : 363 - write idx 1: 1
[D] Buffers.h : 76 - writeArray 0 -> 0
[D] AudioOutput.h : 393 - flush
[D] StreamCopy.h : 338 - write: 1 -> 0
[I] StreamCopy.h : 356 - try write  - 16 (open 1 bytes) 
[D] AudioOutput.h : 363 - write idx 0: 1
[D] Buffers.h : 76 - writeArray 0 -> 0
[D] StreamCopy.h : 338 - write: 1 -> 0
[I] StreamCopy.h : 356 - try write  - 17 (open 1 bytes) 
[D] AudioOutput.h : 363 - write idx 1: 1
[D] Buffers.h : 76 - writeArray 0 -> 0
[D] AudioOutput.h : 393 - flush
[D] StreamCopy.h : 338 - write: 1 -> 0
[I] StreamCopy.h : 356 - try write  - 18 (open 1 bytes) 
[D] AudioOutput.h : 363 - write idx 0: 1
[D] Buffers.h : 76 - writeArray 0 -> 0
[D] StreamCopy.h : 338 - write: 1 -> 0
[I] StreamCopy.h : 356 - try write  - 19 (open 1 bytes) 
[D] AudioOutput.h : 363 - write idx 1: 1
[D] Buffers.h : 76 - writeArray 0 -> 0
[D] AudioOutput.h : 393 - flush
[D] StreamCopy.h : 338 - write: 1 -> 0
[I] StreamCopy.h : 356 - try write  - 20 (open 1 bytes) 
[D] AudioOutput.h : 363 - write idx 0: 1
[D] Buffers.h : 76 - writeArray 0 -> 0
[D] StreamCopy.h : 338 - write: 1 -> 0
[I] StreamCopy.h : 356 - try write  - 21 (open 1 bytes) 
[D] AudioOutput.h : 363 - write idx 1: 1
[D] Buffers.h : 76 - writeArray 0 -> 0
[D] AudioOutput.h : 393 - flush
[D] StreamCopy.h : 338 - write: 1 -> 0
[E] StreamCopy.h : 349 - write  to target has failed after 22 retries! (1 bytes)
[I] StreamCopy.h : 143 - StreamCopy::copy  1 -> 1 -> 0 bytes - in 22 hops
[D] StreamCopy.h : 148 - size_t audio_tools::StreamCopyT<T>::copy() [with T = unsigned char; size_t = unsigned int]
Flushing mixer
[D] AudioOutput.h : 393 - flush

[pschatzmann]

In this case just don't use the copier on the data that might not be returned. This way you will be informed about the missing data and you can act acordingly: You would just do a readBytes on the source and a write to the destination.
In any case it would be recommended to check the number of bytes of the copy() call on the reliable source and use this as input for the step described above.

[jamesbaber]

Thanks for the help so far.

Just wanted to provide an update as I haven't got this working properly yet but am making progress.
I've added a format converter to the minified version of project that I'm using for troubleshooting so that it sounds correct (when both sources are playing).

I now have:
CPInI2S.driver()->setWaitTimeReadMs(8);
and my own copy buffer uint8_t CPBuffer[1024];

And my loop is now:

size_t WMBytesCopied = copyWMToMixer.copy(); // copy reliable input

memset(CPBuffer, 0, WMBytesCopied); // clear copy buffer
size_t CPBytesRead = convertCP.readBytes(CPBuffer, WMBytesCopied); // read discontinuous input
CPVolCtrl.write(CPBuffer, WMBytesCopied); // write the same amount regardless of what is available
mixer.flushMixer();

With a CsvOutput it looks good and continues to stream indefinitely so I wanted to see if it sounds right...
But when I swap the CsvOutput for an A2DPStream, after the usual ~13 seconds of no-CP-signal, the audio becomes intermittent (instead of stopping entirely). I don't have any serial logging activity so that's not the cause.

I tried removing convertCP.readBytes. This makes the reliable input work indefinitely but obviously no sound comes from the CP anymore.
I'm not sure why reading 0 bytes from the FormatConverterStream causes the output to stall when my array is full of 0s regardless (which I have checked by printing them to the serial monitor manually).

I will continue to investigate myself but wanted to check if you had any ideas as I'm sinking a lot of time into troubleshooting this.

Thanks again,
James

[pschatzmann]

I think you need to make sure that the non reliable data source provides WMBytesCopied bytes if it is active...

[pschatzmann]

What is convertCP ?
You don't provide any information that would allow me to figure out what you are doing and it has nothing to do with the initial sketch that you posted

[jamesbaber]

Sorry, I was referencing this

I've added FormatConverterStreams in my main project, which I am reading from instead of the I2S directly.

[pschatzmann]

I am still not sure what you are trying to say: is the read from i2s blocking in spite of setting a timeout ?
Did you eventually change the ESP32 core version ?

[jamesbaber]

Yes exactly. The timeout makes it non-blocking, but only with the older version of audio-tools. Now I've updated, it blocks anyway.
I'm on ESP-IDF version 4.4 in both cases, if that's what you're asking?

[pschatzmann]

I am not aware of any change which could cause this:

• Would you have a suggestion for a reproducible test case ?
• Can you share the log in Debug mode ?

[pschatzmann]

Here is the related test case:

#include "AudioTools.h"

I2SStream i2s;
FormatConverterStream fc(i2s);
AudioInfo to{22000, 2, 16};
uint8_t tmp[1024];

void setup() {
  Serial.begin(114200);
  AudioLogger::instance().begin(Serial, AudioLogger::Info);

  auto cfg = i2s.defaultConfig(RX_MODE);
  cfg.is_master = false;
  i2s.driver()->setWaitTimeReadMs(8);
  i2s.begin(cfg);

  fc.setBuffered(false);
  fc.begin(cfg, to);
}

void loop() {
  auto rc = fc.readBytes(tmp, 1024);
  Serial.println(rc);
}

The FormatConverterStream was waiting for the requested data to be available. I changed the logic a bit to make it non blocking and committed a correction

ps.
Generally it is better (faster and more memory efficient) to apply the FormatConverterStream on the output side.

[jamesbaber]

That's amazing, thank you.
That commit has fixed the issue in my main project as well as in the test case I finished writing 5 minutes after you 😁