FFT for spectrum analyzer

[venquessa]

Hi all!

(Hi Phil, we spoke briefly via email, "Paul" here.).

I am hoping to write a spectrum analyser for a "toe dipping" project with ESP32 Audio and a WS2812b matrix.

I did get the AudioKitFFT project sample up and running, but I was stumped on how to "choose" which "bands" I wanted for the analyser. Ideally, I thought I would want 32 bands for 32 columns of 8 LEDs.

Looking first at the code in the example and in the library behind it I thought I needed a "Window" function. That appeared to be a dead end. In my day job a Window function is a completely different thing :)

I was almost at the point where I would need to blow dust off my maths skills and go learning how FFTs work.

Thankfully I found this video: https://www.youtube.com/watch?v=Mgh2WblO5_c

Straight up described exactly my issue and the gap in communications and understanding around "the bands" or "bins" in the library code.

I haven't tried it yet, but the number of frequencies in the results is determined by the length of samples used.

total bins = Number of Samples / 2

However, half of the bins are unrelated to amplitude so, it's really "number of samples / 4"

It also quickly becomes apparent when you consider "psycho-acoustics" that I don't want 32 bins. As mentioned in the video the upper frequencies are less interesting to use than the lower ones. (I believe this is also taken into account when doing the 32-band pass filter in mp3-encoding?). So aiming for a multiple of "bins per band", such that bins can be grouped (added or averaged) together with more bins assigned to the lower bands than the upper bands. (the video presents a spreadsheet for working this out).

As a suggestion, the wiki page could elaborate a little on the point covering this:

The length defines the number of samples that are used to run the FFT. The length must be a value of the power of 2 (e.g. 1024). This parameter impacts

• the calculation speed and frequency of calculation
• the memory which is needed
  - the frequency resolution of the result

Which is what I was actually after and looked straight past it. Maybe an example?

Length 1024 = 256 bins.
Freq 48kHz = ~ 90Hz bin spacing.

If that is right?

Cheers.

[pschatzmann]

If I remember right, length 1024 gives 512 bins. The size() method will give you the number of bins https://pschatzmann.github.io/arduino-audio-tools/html/classaudio__tools_1_1_audio_f_f_t_base.html

…

On 1 Nov 2022, at 16:36, venquessa ***@***.***> wrote: Hi all! (Hi Phil, we spoke briefly via email, "Paul" here.). I am hoping to write a spectrum analyser for a "toe dipping" project with ESP32 Audio and a WS2812b matrix. I did get the AudioKitFFT project sample up and running, but I was stumped on how to "choose" which "bands" I wanted for the analyser. Ideally, I thought I would want 32 bands for 32 columns of 8 LEDs. Looking first at the code in the example and in the library behind it I thought I needed a "Window" function. That appeared to be a dead end. In my day job a Window function is a completely different thing :) I was almost at the point where I would need to blow dust off my maths skills and go learning how FFTs work. Thankfully I found this video: https://www.youtube.com/watch?v=Mgh2WblO5_c <https://www.youtube.com/watch?v=Mgh2WblO5_c> Straight up described exactly my issue and the gap in communications and understanding around "the bands" or "bins" in the library code. I haven't tried it yet, but the number of frequencies in the results is determined by the length of samples used. total bins = Number of Samples / 2 However, half of the bins are unrelated to amplitude so, it's really "number of samples / 4" It also quickly becomes apparent when you consider "psycho-acoustics" that I don't want 32 bins. As mentioned in the video the upper frequencies are less interesting to use than the lower ones. (I believe this is also taken into account when doing the 32-band pass filter in mp3-encoding?). So aiming for a multiple of "bins per band", such that bins can be grouped (added or averaged) together with more bins assigned to the lower bands than the upper bands. (the video presents a spreadsheet for working this out). As a suggestion, the wiki page could elaborate a little on the point covering this: The length defines the number of samples that are used to run the FFT. The length must be a value of the power of 2 (e.g. 1024). This parameter impacts the calculation speed and frequency of calculation the memory which is needed - the frequency resolution of the result Which is what I was actually after and looked straight past it. Maybe an example? Length 1024 = 256 bins. Freq 48kHz = ~ 90Hz bin spacing. If that is right? Cheers. — Reply to this email directly, view it on GitHub <#435>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAP3RRCW6EOUMHAK3R3CXZTWGE2ILANCNFSM6AAAAAARUFRPOI>. You are receiving this because you are subscribed to this thread.

[venquessa]

Ok. Thanks. Will try that.

I did manage to get a A2DP sink from my phone working. Very low output levels. I have a few more things to try. I shut the output off and put the (silent) FFT in. Seemed to work...

I did however run out of memory... or rather I didn't. I ran into the stack. Constant freertos memory access exceptions in random locations like vListRemote and vListAdd.

Spent a while trying to move as much as I could to psram (64Mbit) turning on sdkconfig flags etc. That just changed the problem to corrupted backtraces and panic handlers. BTC_TASK on CPU0 watchdog timer fired in dwt_wait.

I'm not actually sure it is memory, I noted 130Kb free heap and a few hundred bytes in PSRAM. I think it's the stack is getting corrupted somewhere. Need to look into what it's set to and if I can increase it.

The other thought is that the FFT is blocking the A2DP stream too much, causing it to miss it's watchdog reset.

I suppose I should go back the microphone for the spectrum analyser.

I have bigger plans for this all, but baby steps.

[pschatzmann]

The only way to increase the stack is by creating a separate task.
The different FFT implementations have different memory requirements, but I did not have a detailed look into this.
Maybe if you analyse the stack trace you might be able to figure out what's causing your issue.

I can only recommend to work in small steps and test each small piece separately...

[venquessa]

The current stack dump is this:

`21.53 10489225.00
37.68 19982968.00
37.68 18254740.00
E (419530) task_wdt: Task watchdog got triggered. The following tasks did not reset the watchdog in time:
E (419530) task_wdt: - IDLE (CPU 0)
E (419530) task_wdt: Tasks currently running:
E (419530) task_wdt: CPU 0: BTC_TASK
E (419530) task_wdt: CPU 1: loopTask
E (419530) task_wdt: Aborting.

abort() was called at PC 0x400e0bf5 on core 0

Backtrace:0x400839d9:0x3ffbe7bc |<-CORRUPTED

#0 0x400839d9:0x3ffbe7bc in panic_abort at /Users/ficeto/Desktop/ESP32/ESP32S2/esp-idf-public/components/esp_system/panic.c:402`

Happens about 5-10 FFT results in.

[pschatzmann]

You need to use the EspExceptionDecoder if you work with Arduino.
What exactly are you currently doing with the FFT result ?

[venquessa]

monitor_filters = esp32_exception_decoder

Already applied. I have seen the longer stacks going back to rtos thread.c but ... the reason why I think it's stack related is because the "|<-CORRUPTED" marker in the error trace.

[pschatzmann]

I created a small test sketch, but I can't reproduce your issues: form me it's working like a charm...

[venquessa]

I can get the AD2P stream running the FFT too. The issue is that process is all that core will take. I think I got it to calculate the bands without WDTs. Trying to render the LEDs and it's too much.

My attempts to synchronise them across two cores has run into some issues. Most likely my syncho code has issues, I usually start with the minimum and add until it works.. I did get it working at one point, until I realised I was just ping ponging between the cores. Core 1 waiting on Core 0 to finish and then the inverse. Pointless. I'm currently trying to double buffer the FFT output, but my multi-threading coding in C++ is a little rusty. Spent too long in single core land where volatiles and careful ordering saves most complications.

[venquessa]

A big request, but decoupling it from the BTC Task when calling the user_data_ callback would fix all the problems :)

[pschatzmann]

I agree, the output must be in it's separate thread. To implement this with arduino-freertos-addons should be trivial...

[RWBRWB]

I created a small test sketch, but I can't reproduce your issues: form me it's working like a charm...

I"m using your AD2P libary on the ESP32 to play Bluetooth Audio from devices and I want to use your small text sketch to create a 32 band spectrum analyzer to display music changes on a 400x240 pixel LCD I'm using.

The A2DP library is using output 0:

BluetoothA2DPSink::BluetoothA2DPSink() {
  actualBluetoothA2DPSink = this;
  // default i2s port is 0
  i2s_port = (i2s_port_t) 0;

  // setup default i2s config
  i2s_config = {
      .mode = (i2s_mode_t) (I2S_MODE_MASTER | I2S_MODE_TX),
      .sample_rate = 44100,
      .bits_per_sample = (i2s_bits_per_sample_t)16,
      .channel_format = I2S_CHANNEL_FMT_RIGHT_LEFT,
      .communication_format = (i2s_comm_format_t) (I2S_COMM_FORMAT_I2S | I2S_COMM_FORMAT_I2S_MSB),
      .intr_alloc_flags = 0, // default interrupt priority
      .dma_buf_count = 8,
      .dma_buf_len = 64,
      .use_apll = false
  };

Does this mean that your small sample sketch if added to my code and while streaming bluetooth audio should output the Audio Spectrum without needing to change any of your sample code?

Also does the code require less overhead if only want 16 or 24 bands vs 32 or more?

I have been wanting to add this feature for some time just just haven't figured out how to do it.

I listed to audio using your code here almost daily.

Thanks!

[pschatzmann]

Yes, it should be that simple and the smaller the bands the quicker it runs. When you register the callback, just make sure that you set the option to continue to output the sound via I2S and I would still recommend to put the output logic into the loop and not the fft callback directly.

A comparison between the different FFT implementations you can find in the Wiki of the Audiotools, but I could imagine that if you compare the libraries for smaller and bigger number of bands, the ranging might change...

[RWBRWB]

Hey @pschatzmann

I'm trying to load your basic-a2dp-fft.ino from the Audio Tools Library using Platform IO but I after numerous attempts I have never been able to successful to get the code to compile.

What is the minimum required to get this basic-a2dp-fft.ino sketch to run inside Platform.IO on ESP32?

Really really wanting this spectrum analyzer feature to work so it can display on the nice LCD screen along with the track info.

Any help with this is greatly appreciated.

[pschatzmann]

I don't see any reason why this should not work.
Did you use the recommendations described in the Wiki how to use PlatformIO ?
You also do not give any details what your issue is, so I can't really help...

[RWBRWB]

Basically I get tons of compile errors after I add the libraries that the example code calls for at the top of the sketch.

So I'm just looking for exactly how I should try to use the example basic-a2dp-fft.ino

I started by downloading the Audio Tools Library

Then I opened it up in Platform IO with ESP32 Dev Board as the board.

Then I dragged the basic-a2dp-fft.ino into the SRC folder.

I then had to add the Bluetooth.h and other files so it was not missing those.

Then I get tons of compile errors after that, like 30+ to many to even attempt to fix.

Do you use Platform IO?

I wonder what your file structure is like when you have this setup on your end without any errors.

Basically what is the minimum required to get the basic-a2dp-fft.ino working?

[pschatzmann]

No, read the documentation and learn to use PlatformIO first: The dependencies are specified in the platformio.ini.
By the way this is not the place to get PlatformIO support!

[RWBRWB]

I know how to use Platform IO, I have like 20+ working projects on it.

I have a your Web Radio & A2DP libraries both working just fine.

Just can't seem to get this simple example to compile.

I'll find somebody I can pay to get this working for me.

Thanks for replying.

[pschatzmann]

Just use the platformio.ini described in the Wiki and add all necessary dependencies. In your case this would be at least

lib_deps = https://github.com/pschatzmann/arduino-audio-tools
         https://github.com/pschatzmann/ESP32-A2DP

But I can't guess what exact other libraries you intend to use...
If you still get compile errors, this will guide you what's missing!

If you use AudioEspressifFFT or AudioRealFFT there is nothing to install for FFT. If you use an other FFT library, you will need to add that as well. The details can be found in the documentation!

ps. Only if you share your errors, I will be able to help you!

[pschatzmann]

I was testing with AudioRealFFT and it was compiling w/o any errors...

#include "AudioTools.h"
#include "AudioLibs/AudioRealFFT.h" // or AudioKissFFT
#include "BluetoothA2DPSink.h"

BluetoothA2DPSink a2dp_sink;
AudioRealFFT fft; // or AudioKissFFT
...

And platformio.ini

[platformio]
description = Audio Example
default_envs = esp32dev

[env:esp32dev]
platform = https://github.com/platformio/platform-espressif32.git
board = esp32dev
framework = arduino
lib_deps = https://github.com/pschatzmann/arduino-audio-tools
           https://github.com/pschatzmann/ESP32-A2DP

build_flags = -DCORE_DEBUG_LEVEL=5 -Wno-unused-variable -Wno-unused-but-set-variable -Wno-unused-function -Wno-format-extra-args 
monitor_speed = 115200
monitor_filters = esp32_exception_decoder

ps. I changed the example to use AudioRealFFT so that there is no need for any additional library...