Tips on Opus

[jeffiel]

Hi All -

I'm trying to use opus to enable two way voice communication. I have the bare basic implementation of the codec with a sine wave for test, but am getting very garbled signal. Running on an ESP32. Code is from this blog post from Phil:

https://www.pschatzmann.ch/home/2022/05/06/audio-streaming-the-opus-codec/

Only modified to output to my amplifier. Tried various complexities, applications, same result. Very garbled sine wave. Removing encoder/decoder, I get a beautiful sine wave.

Is this just what opus does to a sine wave? Or tips on how I can get this sounding right? Thanks in advance!

#include "AudioTools.h"
#include "AudioCodecs/CodecOpus.h"

#define AMP_I2S_PORT 1      // i2s port for amp
#define AMP_I2S_DATA 27     // GPIO pin for amp data
#define AMP_I2S_SEL 12      // GPIO pin for amp word select
#define AMP_I2S_CLK 19      // GPIO pin for amp clock
#define I2S_MUTE_PIN -1                                           // init I2SStream w/ -1 as the MUTE_PIN which is important - ESP32 seems to crash otherwise


int sample_rate = 24000;
int channels = 2;  // The stream will have 2 channels

SineWaveGenerator<int16_t> sineWave( 32000);  // SoundGenerator max amplitude of 32000
GeneratedSoundStream<int16_t> sound( sineWave); // Stream generated from sine wave
I2SStream out(I2S_MUTE_PIN);                     // I2S Streamer to the amplifier hardware
OpusAudioEncoder enc;
EncodedAudioStream decoder(&out, new OpusAudioDecoder()); // encode and write 
EncodedAudioStream encoder(&decoder, &enc); // encode and write 
StreamCopy copier(encoder, sound);     

void setup() {
  Serial.begin(115200);
  AudioLogger::instance().begin(Serial, AudioLogger::Info);

  // configure AMP output
  auto config_out = out.defaultConfig(TX_MODE);
  config_out.sample_rate = sample_rate; 
  config_out.bits_per_sample = 16; 
  config_out.i2s_format = I2S_STD_FORMAT;
  config_out.channels = channels;
  config_out.is_master = true;
  config_out.port_no = AMP_I2S_PORT;
  config_out.pin_bck = AMP_I2S_CLK;
  config_out.pin_ws = AMP_I2S_SEL;
  config_out.pin_data = AMP_I2S_DATA;
  out.begin(config_out);

  // Setup sine wave
  auto cfgs = sineWave.defaultConfig();
  cfgs.sample_rate = sample_rate;
  cfgs.channels = channels;
  cfgs.bits_per_sample = 16;
  sineWave.begin(cfgs, N_B4);

  // Opus decoder needs to know the audio info
  decoder.begin(cfgs);

  // configure and start encoder
  enc.config().application = OPUS_APPLICATION_AUDIO;
  enc.config().complexity = 5;
  encoder.begin(cfgs);

  Serial.println("Test started...");
}

void loop() { 
  copier.copy();
}

[pschatzmann]

I am not sure but encoding and decoding are very expensive operations (which use extensive floating point operations) and maybe you are asking just a little too much from such a slow micro controller. I guess that was the reason why in my test case I used the reduced sample rate of 24000 and limited the channels to 1.

In order to confirm this: did you try to decrease (e.g. half) the sample rate ?

The esp32 has 2 processors, so if you really need to do both, you could consider to use one processor for encoding and the other for decoding.

[jeffiel]

Thank you Phil. I tried a variety of sample rates and channels and didn't see material differences. But I'll do another round of testing.

Any other parameters that simplify that I should play with, such as complexity, application, bit rate, etc. that come to mind?

I've been using G711 successfully but am now looking to increase audio quality with other codecs. Opus came to mind, as voice is my use case. LMK if you think I should explore any other codecs in your suite.

As an aside, I appreciate how responsive you are to your community. Thank you!

[jeffiel]

It was hard to design a test that separated CPU loads onto the 2 cores, so I went back to my working implementation that transports over the network. This effectively isolates CPU load, but does open up more variables. I do have a question though. With many encoder values, I get an assert failure at the same line in the opus codec. It seems that sample rates other than 48k or 24k trigger this, as well as setting application to VOIP seem to reliably trigger it. I'm wondering if you can shed some light on what it means. Here's the stack trace:

PC: 0x40100c16: celt_pitch_xcorr_c at /Users/jeffiel/Documents/Arduino/libraries/arduino-libopus/src/opus-1.3.1/celt/pitch.c line 252
EXCVADDR: 0x00000000

Decoding stack results
0x40100c13: celt_pitch_xcorr_c at /Users/jeffiel/Documents/Arduino/libraries/arduino-libopus/src/opus-1.3.1/celt/pitch.c line 252
0x4010f001: silk_burg_modified_c at /Users/jeffiel/Documents/Arduino/libraries/arduino-libopus/src/opus-1.3.1/silk/fixed/burg_modified_FIX.c line 98
0x4011366a: silk_find_LPC_FIX at /Users/jeffiel/Documents/Arduino/libraries/arduino-libopus/src/opus-1.3.1/silk/fixed/find_LPC_FIX.c line 63
0x40113fcc: silk_find_pred_coefs_FIX at /Users/jeffiel/Documents/Arduino/libraries/arduino-libopus/src/opus-1.3.1/silk/fixed/find_pred_coefs_FIX.c line 133
0x4010fbf8: silk_encode_frame_FIX at /Users/jeffiel/Documents/Arduino/libraries/arduino-libopus/src/opus-1.3.1/silk/fixed/encode_frame_FIX.c line 155
0x4010b849: silk_Encode at /Users/jeffiel/Documents/Arduino/libraries/arduino-libopus/src/opus-1.3.1/silk/enc_API.c line 499
0x4011923d: opus_encode_native at /Users/jeffiel/Documents/Arduino/libraries/arduino-libopus/src/opus-1.3.1/src/opus_encoder.c line 1846
0x40119b8b: opus_encode at /Users/jeffiel/Documents/Arduino/libraries/arduino-libopus/src/opus-1.3.1/src/opus_encoder.c line 2231
0x400d5221: audio_tools::OpusAudioEncoder::encodeFrame() at /Users/jeffiel/Documents/Arduino/libraries/arduino-audio-tools/src/AudioCodecs/CodecOpus.h line 311
0x400d5312: audio_tools::OpusAudioEncoder::write(void const*, unsigned int) at /Users/jeffiel/Documents/Arduino/libraries/arduino-audio-tools/src/AudioCodecs/CodecOpus.h line 298
0x400d3cf1: audio_tools::EncodedAudioStream::write(unsigned char const*, unsigned int) at /Users/jeffiel/Documents/Arduino/libraries/arduino-audio-tools/src/AudioCodecs/AudioEncoded.h line 342
0x400d386c: DownvertingStream::write(unsigned char const*, unsigned int) at /Users/jeffiel/code/wifi-talkie/WiFiTalkie-audiotools/SupplementalStreams.h line 61
0x400d5961: audio_tools::StreamCopyT ::write(unsigned int, unsigned int&) at /Users/jeffiel/Documents/Arduino/libraries/arduino-audio-tools/src/AudioTools/AudioCopy.h line 257
0x400d5a97: audio_tools::StreamCopyT ::copy() at /Users/jeffiel/Documents/Arduino/libraries/arduino-audio-tools/src/AudioTools/AudioCopy.h line 279
0x400d5b43: AudioSubsystem::loop() at /Users/jeffiel/Documents/Arduino/libraries/arduino-audio-tools/src/AudioTools/AudioCopy.h line 350
0x400d2e4a: loop() at /Users/jeffiel/code/wifi-talkie/WiFiTalkie-audiotools/WiFiTalkie-audiotools.ino line 132
0x4011facd: loopTask(void*) at /Users/jeffiel/Library/Arduino15/packages/esp32/hardware/esp32/2.0.5/cores/esp32/main.cpp line 50

Here's the line:

   celt_sig_assert((((unsigned char *)_x-(unsigned char *)NULL)&3)==0);

[pschatzmann]

Difficult to say but taking some wild guesses it might be one of the following:

• The post you refer to is just a hack that relies that a single opus packets is being processed with each copy step. This can not be used like this in real life and Opus needs a container: Are you sure that you are using the OpusOggEncoder and OpusOggDecoder classes ? However I am not sure if I was able to have this working properly!
• Your solution might not be thread save and might have some race conditions.

I realize now that an implementation using 2 cores is quite challenging and you would need to rely on the functionality provided by RTOS to pass the data is a save way (e.g. using https://github.com/pschatzmann/arduino-freertos-addons).

So I would suggest to test in 2 steps: Encoding and writing a file, reading from a file and decoding. But in any case you will need to use the codec together with a container.

[jeffiel]

Thanks Phil. I was wondering whether an Ogg container was necessary here so thanks for verifying that. I’ll play with the OggOpus library

[pschatzmann]

It's too long ago. But post your findings here that others can profit from your investigations