Layer_Normalize or Batch_Normalize for Neural_Receiver ?

[nnk14102002]

I am experimenting with architectural modifications with Neural Receiver. I prepare data for each batch include a range of SNR (-10 -> 5dB). During this process, I found that if I use LayerNorm or None, BCE_Loss will converge vice versa when using BatchNorm. I suspect the problem is due to the data with snr distribution when normalizing (low_SNR => bad data mix with high_SNR => clean data). What do u think about this problem? Thanks!

[SebastianCa]

Hi @nnk14102002,
since your batches contain samples with varying SNR levels, applying normalization across the entire batch seems problematic. I recommend using normalization on a per-sample (slot-wise) basis instead, which is generally more robust in scenarios where SNR varies significantly between samples.