Problems when trying to csvstream data from i2s source

[FinnMeyer]

Hello,
I would like to capture the output of my inmp441 with my esp32 and send the data via serial to my pc. With the code i provided below this works in general but when I capture the data with my pc and turn it into a wav file it seems like im losing some data when increasing the sample rate. I already increased:
DEFAULT_BUFFER_SIZE
I2S_BUFFER_SIZE
I2S_BUFFER_COUNT
but i am still not able to stream all the data via serial at sampling rates even close to 44100. Is there some workaround to increase the sampling rate without losing data? Should I use the HexDumpStream instead of the CsvStream? Does it make sense to use a codec here for compression or am i missing something else?
Thank you!

#include "AudioTools.h"

I2SStream i2sStream; // Access I2S as stream
CsvStream<int16_t> out(Serial);                         // Output to Serial
AudioInfo from(22000, 1, 32);                                           
AudioInfo to(22000, 1, 16);                                           
FormatConverterStream conv(i2sStream);
StreamCopy copier(out, conv);                           // copies converted sound to out

void setup(){
  Serial.begin(500000);
  i2sStream.setAudioInfo(from);
  i2sStream.begin();
  conv.begin(from, to);
  out.begin(to);
}


void loop(){
    copier.copy();
}

[pschatzmann]

Today I was trying to do something similar: I tried to send the data directly in binary format and read it from a linux device.
I am currently a little bit confused because I would have expected that the Serial interface should handle binary data w/o any issues.

Here seems to be the challenge: with the data as binary 44100 bytes per second would require a bit rate of 882000.
You try to send it as ascii, which would require even hight rates.

I am not sure what rates are realistic...

What OS are you using ? It would be interresting to see if you run into the same issues if you just copy the data to Serial...

[FinnMeyer]

im on ubuntu 20.04. Increasing the baudrate helps. For the esp i was able to increase it to 2 000 000 which now at least makes it possible to record up to a sample rate of 20 000 Hz which sounds ok for human speech. The direct serial feed without the csvStream i am still trying to implement

[pschatzmann]

I tried to measure the performance...

Here is the Arduino Sketch:

#include "AudioTools.h"

AudioInfo info(8000,1,16);
SineWaveGenerator<int16_t> sineWave(32000);                // subclass of SoundGenerator with max amplitude of 32000
GeneratedSoundStream<int16_t> sound(sineWave);             // Stream generated from sine wave
StreamCopy copier(Serial, sound);                          // copies sound into serial

// Arduino Setup
void setup(void) {  
  AudioLogger::instance().begin(Serial2, AudioLogger::Error);

  // Open output 
  Serial.begin(500000);
  Serial.setTxBufferSize(1024);

  // Setup sine wave
  sineWave.begin(info, N_B4);
}

// Arduino loop - copy sound to out 
void loop() {
  copier.copy();
}

And on the PC I was receiving the data with a Python script

import serial
import numpy as np
import array
import time

sendSerial = serial.Serial ("/dev/ttyUSB5", 500000)
start_time = time.time()
total = 0;
#while True:
for count in range(100):
    byte_array = sendSerial.read(1024)
    data = array.array('h',byte_array)
    total = total + len(data);
    #print(len(data))
    # print(data)

end_time = time.time()
run_time = end_time-start_time
print("samples:", total, "time:", run_time, "=> sample_rate: ", total/run_time)

Here is the result I was getting on my 10 years old Macbook with Linux
samples: 51200 time: 2.036194324493408 => sample_rate: 25144.947799978876

[pschatzmann]

I was just looking at the received data, and I am not receiving a stable sine wave...

[FinnMeyer]

with your code i can sometimes get correct data and sometimes i dont. In both runs i had a sample rate of 100 000.

[FinnMeyer]

i took one of the 1024 byte arrays that was corrupted and started to chop the bytes from the beginning and after a few i got a perfect sin wave again so my guess is that Serial write or read is missing some bytes which results in the hole stream being corrupted?

[FinnMeyer]

For some reason adding a FormatConverter seems to fix the problem i dont know why but when adjusting the script to:

#include "AudioTools.h"

SineWaveGenerator<int16_t> sineWave(32000);                // subclass of SoundGenerator with max amplitude of 32000
GeneratedSoundStream<int16_t> sound(sineWave);             // Stream generated from sine wave
AudioInfo from(8000, 1, 32);                                           
AudioInfo to(8000, 1, 16);   
CsvStream<int16_t> out(Serial); 
FormatConverterStream conv(sound);    
StreamCopy copier(Serial, conv);                          // copies sound into serial

// Arduino Setup
void setup(void) {  
  AudioLogger::instance().begin(Serial2, AudioLogger::Debug);

  sineWave.begin(to, N_B4);
  Serial.begin(2000000);
  Serial.setTxBufferSize(1024);
  conv.begin(from, to);


}

// Arduino loop - copy sound to out 
void loop() {
  
  copier.copy();
}

i always get correct data.

[FinnMeyer]

With this code:

#include "AudioTools.h"
AudioInfo to(32000, 1, 16);
I2SStream i2sStream; // Access I2S as stream                                         
     
StreamCopy copier(Serial, i2sStream);                          // copies sound into serial

// Arduino Setup
void setup(void) {  
  AudioLogger::instance().begin(Serial2, AudioLogger::Debug);

  i2sStream.setAudioInfo(to);
  i2sStream.begin();
  Serial.begin(2000000);
  Serial.setTxBufferSize(1024);

}

// Arduino loop - copy sound to out 
void loop() {
  
  copier.copy();
}

i was able to use my mic until a sampling rate of 32000 Hz after that i run into this overlow behavior:

i dont really get why increasing my sample rate leads to these overflow behavior i would have assumed to get more of a random behavior. Beside these jumps the data looks "ok" (reasonable peaks when i speak into the mic) Any ideas where this comes from or how to fix? Thank you

[pschatzmann]

You should change the AudioLogger::instance().begin(Serial2, AudioLogger::Debug); to AudioLogger::instance().begin(Serial2, AudioLogger::Error); I would expect that the log level debug has quite some nasty effect on the quality...

[pschatzmann]

espressif/arduino-esp32#8037

[pschatzmann]

Here is my understanding what's happening: when sending binary data it is critical to read the data with the proper int16_t allignement. If you just start to read after mssing one byte you will get noise. The same is happening when you loose a byte in the middle of the processing: your proper signal just turns into noise and after loosing another byte turns into a proper signal again.

So it turns out sending binary audio data over serial will not work because of this.
I came up with the solution to base encode a single frame which turned out to be much more reliable (but also slower than binary).

I also tried to send mp3 data: the transmission started properly but after about 10 or 20 seconds I was running into dropped frames. I guess for the very same reason that the binary was not working.

Honestly, I am surprised that Serial (over USB) is so unreliable...

[FinnMeyer]

ok thank you for your efforts!