recive audio from multiple sources for audiokit

[josef2600]

thank you for your work. but i have multiple problems! it is very confusing when i am trying to do a new thing with your code. and i found some confusing problems. for example:
i have ESP32-A1S module with ES8388 codec. if i use your default setting or i use option 5 (that you sett for default) then i get ton of errors and it wont work. but if i choose option 7, it works. it is really frustrating. then in some of your examples for audiokit, instead of using audiokit , you are using audioinfo and copying it to output!? what? it doesn't work. i have to change it to copy it to audiokit.
lets go to my code problem. i cant fined one good sample or code to send audio to esp32. why? non of them works. i do not care about radio. i want to send an audio from vlc via any protocol to my esp32. the best result i got was with CodecMP3Helix , and that was playing 100ms and then gets confused what is it it is receiving! constantly changing bitrates and other settings and never gets it. and no, i have checked mp3 with all settings possible and bitrates . but no, nothings works.
anyway, i need to receive audio from multiple places. i figured this code to do it, but obviously , surprise! it doesn't work! any way, if you can check it, it would be great. thanks.

#include <WiFi.h>
#include "Wire.h"
#include <WiFiUdp.h>


//I2S buffer length at most 1024
#define AUDIOKIT_BOARD  7
#define ES8388_DEFAULT_INPUT_GAIN 5
#define AI_THINKER_ES8388_VOLUME_HACK 1


#include "AudioTools.h"
#include "AudioLibs/AudioKit.h" // comment out when not using AudioKit
#include "AudioKitHAL.h"
#include "AudioLibs/RTSP.h"
#include "AudioStreamer.h"
#include "RTSPServer.h"
#include "AudioCodecs/CodecMP3Helix.h"



#include <WiFi.h>
#include "Arduino.h"
#include "AudioTools.h"
#include "AudioCodecs/CodecMP3Helix.h"
#include "AudioLibs/AudioKit.h"
#include <ESPAsyncWebServer.h>

// Replace with your network credentials
const char* ssid = "josef_wifi";
const char* password = "josef_wifi";

// Create instances of your audio tools
MP3DecoderHelix gen;
AudioKitStream kit;  // Audio source

// Create an instance of the web server
AsyncWebServer server(80);

void setup()
{
  // Connect to Wi-Fi
  WiFi.begin(ssid, password);
  while (WiFi.status() != WL_CONNECTED)
  {
    delay(1000);
    Serial.println("Connecting to WiFi...");
  }
  Serial.println("Connected to WiFi");

  // Initialize the AudioKit
  kit.begin();

  // Set up the web server
  server.on("/audio", HTTP_POST, {
    // Get the audio data from the POST request
    AsyncWebParameter* p = request->getParam(0);
    if(p->isPost()){
      // The POST request contains audio data
      String audioData = p->value();

      // TODO: Convert the string data to a format that can be used by the audio generator

      // Use the audio generator to decode the data and output it through I2S
      gen.input(audioData);  // Set the input stream
      gen.begin();           // Start decoding
    }
    request->send(200);
  });

  // Start the server
  server.begin();
}



void loop()
{
  // Nothing to do here - everything is handled by the web server
}

i am trying to create a server that multiple devices can send audio to it and it reads the data and sends it to speaker. you can Set up the web server the way you like. it is easy. just receives the data from the link "/audio" and sends it to the decoder MP3DecoderHelix gen; . see ! easy!
just the ESPAsyncWebServer.h . you know how bad it is!
please don't brush it off, check one code that does the receiving via rtsp or http or udp or whatever, just not internet radio.

[pschatzmann]

Yes, this will not work: You obviously ignored all the documentation that can be found in the AudioKit Project!
Your AudioServer project also already exsists...

[josef2600]

thank you. but you did exactly the opposite ! i ask for a code that RECEIVES audio from multiple sources, NOT sending audio. you didn't read what i said! but i have read the AudioKit Project!

[pschatzmann]

So you did not see the text in the Configuration Section:
You must define your board and the default settings in the libraries/arduino-audiokit/src/AudioKitSettings.h file. Please note that the functionality will not work properly if this is done in the Sketch.

ps. you cant't store the audio data in a string on a Microcontroller: There is just not enough memory....

[josef2600]

thank you. about the config, yes i know i have to define it, hence why i choose the board number.and it works in sketch if it is sett before included library's, that is why i sett it before includes. what i meant was that is odd that option 5 5) ai_thinker (ES8388) 2957 3478 A149 doesn't work, at least for me. but 7 works. and it is the same board you have introduced in audiokit lib. ai_thinker (ES8388) 2957 is my module. i just don't have the bottoms.
thank you, best regards.

[pschatzmann]

That's why you can find the following in the Readme: Unfortunately AI Thinker created a big mess with their boards using different Audio chips and pin assingments, all of them using the same version number. You might need to make an educated guess by looking at the number after the version and if you have a 2957 board, you need to check all 3 options!

If you have a new combination, feel free to add a new board number...

[josef2600]

thank you. i know! it is china after all. but i did check it. i opened the case and checked the codec ic number.

[josef2600]

and about the defines in sketch, you have already did a great job in defining everything! i love it. for example you did it:

#ifndef DEFAULT_SAMPLE_RATE 
#  define DEFAULT_SAMPLE_RATE 44100
#endif

and for all definitions that are important, you have don it. the user just need to re-define something before the include, then it works fine. and i prefer not to touch your source code, since it is changing every day.

[pschatzmann]

Actually I recomend to do this in the config object as described here: https://github.com/pschatzmann/arduino-audio-tools/wiki/External-DAC

[josef2600]

do you have any opinion on how i can do the receive from multiple senders? well, technically, i have not been able to receive audio at all! i don't mean play from internet. i need to send audio from my PC or tablet with something like vlc, and play it with audio kit? preferably with a codec, like mp3. because its a hassle to convert mp3 to wave then stream it to esp32 audio kit.
i mean, like audio kit has a fix ip and others send audio to it. because my brain is going to explode! i cant figure it out why it doesn't work. i would really appreciate it.
p.s 👍

[pschatzmann]

I am not quite sure what you mean with multiple senders: Do you have multiple senders that send concurrently different audio to one single destination ?

[josef2600]

yes. lets imagine it is a building with 5 houses in it. each one has an iphone (not that shitty company!) to communicate with the front door. in front door is the esp32 audio kit for receiving the voice. if it is not possible to listen to multiple sources at the same time, if possible, give me a sample for just one! but it has to work for LAN network, not internet. it is very important. i remember there was one your examples that could check if the sender is sending or not, but i don't remember where it was! because it can check if any of them is active. they are deactivated after conversation is over. i can have their ip too. whatever is possible in a good matter!
the sending part, is for later! and there are many ways for it. so i am not on that yet!
thank you.

[josef2600]

just for fun!
https://www.youtube.com/watch?v=r6tH55syq0o

[pschatzmann]

Your comment about internet and LAN does not make any sense: both use the same communications protocols!

I think your multiple senders is not impossible, but quite challenging. The issue is that concurrent audio output is not handled by the operating system, but you would need to do this yourself with the help of a mixer. So I would expect, that part would be easier on a Rasperry PI (If you are looking for a small one, a zero would do)

For the AudioTools, just look at the communications examples: e.g. ip, udp, iban. For the iban, I did not test the receiving part: so there is no example, but it might be an interesting starting point because you already get the sending App and for your voice example it does not disturb that it's working with pcm.

Even your webserver post example might work as well, if you manage to get a server implementation that provides you the data as a stream...

[pschatzmann]

I just added a mp3 post example to my TinyHttp project...
Something similar should work with your server as well: you just need to implement a handler and from there you have access to the AsyncClient that you can use a copy source...

[josef2600]

thank you. i will test it.

[josef2600]

ok, i have this code:

/**
 * @file AudioKitSettings.h
 * @author Phil Schatzmann
 * @brief AUDIOKIT_BOARD selects a specic board:
 *  1) lyrat_v4_3
 *  2) lyrat_v4_2
 *  3) lyrat_mini_v1_1
 *  4) esp32_s2_kaluga_1_v1_2
 *  5) ai_thinker (ES8388) 2957 3478 A149
 *  6) ai_thinker (AC101) 2473 2762 2957
 *  7) ai_thinker (ES8388) 2957
 *  8) esp32_s3_box
 *  9) esp32_s3_box_lite
 *  10) generci_es8388
 *  11) generic_es8311
 *  12) generic (no defaults)
 * @version 0.1
 * @date 2021-12-12
 *
 * @copyright Copyright (c) 2021
 */



//I2S buffer length at most 1024
#define AUDIOKIT_BOARD  7
#define ES8388_DEFAULT_INPUT_GAIN 1 // 25
#define AI_THINKER_ES8388_VOLUME_HACK 1
#define COPY_DELAY_ON_NODATA 1


#define ARDUINO_LOOP_STACK_SIZE (3 * 1024)

#define DEFAULT_BUFFER_SIZE  512
#define DEFAULT_SAMPLE_RATE 16000 // 44100
#define DEFAULT_CHANNELS 2 // 2
#define DEFAULT_BITS_PER_SAMPLE 16
#define I2S_BUFFER_SIZE DEFAULT_BUFFER_SIZE
#define I2S_BUFFER_COUNT 3  //6
#define ANALOG_BUFFER_SIZE 512 // 1024
#define ANALOG_BUFFER_COUNT 6 // 6
#define CODEC_DELAY_MS 1 // 10
#define COPY_DELAY_ON_NODATA 1

//#define OPUS_DEC_MAX_BUFFER_SIZE 20*1024  //4*1024


#include <WiFi.h>


#include "AudioTools.h"
#include "AudioLibs/AudioKit.h"


//#include "AudioCodecs/CodecOpusOgg.h"
//#include "AudioCodecs/CodecL16.h"
//#include "AudioCodecs/CodecADPCMXQ.h"
//#include "AudioCodecs/CodecMP3Mini.h"
//#include "AudioCodecs/CodecMP3MAD.h"
#include "AudioCodecs/CodecMP3Helix.h"
//#include "AudioCodecs/CodecOpus.h"
//#include "AudioCodecs/CodecOpusOgg.h"


#define MAX_CLIENTS 3

uint16_t me_port = 7000;
WiFiServer server(me_port);
WiFiClient my_clients[MAX_CLIENTS];
AudioKitStream me_I2S;

EncodedAudioStream* decoders[MAX_CLIENTS];
//EncodedAudioStream decoders(&me_I2S, new MP3DecoderHelix());

//MeasuringStream* outTimed[MAX_CLIENTS];
StreamCopy* copiers[MAX_CLIENTS];

 // AudioKitStream me_I2S;

  //OpusAudioDecoder opus;

  //EncodedAudioStream dec(&me_I2S, new MP3DecoderHelix());
  //EncodedAudioStream dec(&me_I2S, new OpusAudioDecoder());
  //EncodedAudioStream dec(&me_I2S, new OpusOggDecoder());
  //StreamCopy copier(dec, me_client);

  const char *mssid = "josef";
  const char *mpassword = "josef";
  
  unsigned long lastCheckedTime = 0;


void connectWifi()
{
  // connect to WIFI
  WiFi.begin(mssid, mpassword);
  while (WiFi.status() != WL_CONNECTED)
  {
      delay(500);
      Serial.print(".");
  }
  Serial.println();
  Serial.println(WiFi. localIP());

  // Performance Hack              
  //esp_wifi_set_ps(WIFI_PS_NONE);
  WiFi.setSleep(false);
}

void setup()
{
  // ... your setup code ...
  //Serial.begin(115200);
  Serial.begin(230400);
  delay(500);
  AudioLogger::instance().begin(Serial, AudioLogger::Warning);//Warning  Info
  delay(100);

 // if (!WiFi.configg(local_IP, gateway, subnet, primaryDNS))
  //{
 //   Serial.println("STA Failed to configure");
 // }
  delay(500);

  
  connectWifi();


  Serial.println("");

  Serial.print("IP address: ");
  Serial.println(WiFi.localIP());


  // start server
  server.begin();


    auto configg = me_I2S.defaultConfig(TX_MODE);
    configg.sd_active = false;
    configg.buffer_size = DEFAULT_BUFFER_SIZE;
    configg.buffer_count = I2S_BUFFER_COUNT;

    configg.bits_per_sample = DEFAULT_BITS_PER_SAMPLE;
    configg.sample_rate = DEFAULT_SAMPLE_RATE; // 11025 - 8000 - 22050 - 24000

    me_I2S.setVolume(90);
    me_I2S.begin(configg);
    me_I2S.setVolume(90);
   // dec.begin();
   //decoders.begin();
   
  // start me_I2S and set up decoders and copiers for each my_clients
  for (int i = 0; i < MAX_CLIENTS; i++)
  {
    decoders[i] = new EncodedAudioStream(&me_I2S, new MP3DecoderHelix());
    //decoders[i] = new EncodedAudioStream(&me_I2S, new MP3DecoderMAD());
    //decoders[i] = new EncodedAudioStream(&me_I2S, new MP3DecoderMini());
    //decoders[i] = new EncodedAudioStream(&me_I2S, new ADPCMDecoderXQ());
    //decoders[i] = new EncodedAudioStream(&me_I2S, new DecoderL16());
    //decoders[i] = new EncodedAudioStream(&me_I2S, new OpusOggDecoder());
    decoders[i]->begin();
    
    //outTimed[i] = new MeasuringStream(*decoders[i]);
    copiers[i] = new StreamCopy(*decoders[i], my_clients[i]);
  }

  xTaskCreatePinnedToCore(task_my_handleClient, "task_my_handleClient", 45000, NULL, 5, NULL,1);
  
}



void task_my_handleClient(void * parameter)
{
  while (1)
  {
    for (uint8_t i = 0; i < MAX_CLIENTS; i++)
    {
      if (copiers[i]) copiers[i]->copy();
    }

    taskYIELD();
    //vTaskDelay(1);
  }
  vTaskDelete(NULL); // Delete this task
}



void loop()
{

  // check for new me_clients
  WiFiClient newClient = server.available();
  if (newClient)
  {
    for (int i = 0; i < MAX_CLIENTS; i++)
    {
      if (!my_clients[i] || !my_clients[i].connected())
      {
        if (my_clients[i]) my_clients[i].stop();
        //my_clients[i] = server.available();
        my_clients[i] = newClient;

        //copiers[i] = new StreamCopy(*decoders[i], my_clients[i]);
        if (copiers[i])
        {
          vTaskDelay(100);
          copiers[i]->end();
          delay(5);
          delete copiers[i];
          copiers[i] = nullptr;
          copiers[i] = new StreamCopy(*decoders[i], my_clients[i]);
          delay(5);
        }
        break;
      }
    }
  }

  // copy data for each connected client
  for (uint8_t i = 0; i < MAX_CLIENTS; i++)
  {
    if (my_clients[i] && my_clients[i].connected())
    {
     // copiers[i]->copy();
    }
    else if (my_clients[i])
    {
      my_clients[i].stop();
    }
  }

 // for (uint8_t i = 0; i < MAX_CLIENTS; i++)
 // {
 //   copiers[i]->copy();
 // }

  if ((lastCheckedTime + 3000) < millis())
  {
    lastCheckedTime = millis();
    Serial.print("Free heap:   ");
    Serial.println(ESP.getFreeHeap());
    //delay(1);
  }

  delay(300);
}

it is suppose to get up to 3 different client audio , and play it. if i connect 1 client, it is great, but for 2 or more, audio gets clipy and very bad. i don't understand where the clipping is coming from, where is the delay in it? its like it reads one client and then waits few ms then goes and read other client and so on. what is happening? i really need your help.
and in windows i am using curl for sending audio:

curl -X POST -H "Content-Type: audio/mpeg" --data-binary @D:\k\5.mp3 http://192.168.1.10:7000/audio
curl -X POST -H "Content-Type: audio/mpeg" --data-binary @D:\k\15.mp3 http://192.168.1.10:7000/audio
curl -X POST -H "Content-Type: audio/wav" --data-binary @D:\k\14.wav http://192.168.1.10:7000/audio
curl -X POST -H "Content-Type: audio/wav" --data-binary @D:\k\16.wav http://192.168.1.10:7000/audio
curl -X POST -H "Content-Type: audio/gsm" --data-binary @D:\k\17.gsm http://192.168.1.10:7000/audio
curl -X POST -H "Content-Type: audio/L16" --data-binary @D:\k\11.wav http://192.168.1.10:7000/audio/wav
curl -X POST -H "Content-Type: audio/L16" --data-binary @D:\k\16.wav http://192.168.1.10:7000/audio
curl -X POST -H "Content-Type: audio/mpeg" --data-binary @D:\k\18.mp3 http://192.168.1.10:7000/audio
curl -X POST -H "Content-Type: audio/ogg" --data-binary @D:\k\19.ogg http://192.168.1.10:7000/audio
curl -X POST -H "Content-Type: audio/ogg" --data-binary @D:\k\20.ogg http://192.168.1.10:7000/audio

https://curl.se/windows/microsoft.html

and it can only play mp3. i don't know where i am going bad. i have tested lot of decoders. but that really doesn't matter.
i love to be able to play GSM, but doesn't work. and if mp3 is less than 24khz, it wont play it either. i cant play wav either.
i am using ffmpeg for converting mp3 to gsm.
ffmpeg -i D:\k\4.mp3 -ar 8000 -ac 1 -ab 13k -f gsm D:\k\output.gsm

i am slamming my head to the wall! i am going mad! need help.
regards, Josef.

[pschatzmann]

I suggest that you compare the data that you send with the one that you receive. At least for the first few bytes

[josef2600]

you suspect data corruption ? but what about mp3 files? what is causing the jitter ? they are playing, but not right.

[pschatzmann]

maybe just some unexpected cr lf characters from the end of the the http header