Decoding and WER evaluation

[andrusenkoau]

Hi everyone!
I am trying to decode all librispeech dev and test sets for my Conformer-CTC-BPE model by "eval_beamsearch_ngram.py" script in greedy way. But I am getting WER results that are different from training logs. For example, for the last 100th epoch I have got WER = 10.43% for dev-clean according to "lightning_logs.txt". But after "eval_beamsearch_ngram.py" I have got WER = 12.01%. Maybe i am doing something wrong?

This is my decoding command:

model_dir="nemo_experiments/Conformer-CTC-BPE/2021-12-14_12-47-27"
model_file="${model_dir}/checkpoints/Conformer-CTC-BPE.nemo"

for test_set in dev-clean dev-other test-clean test-other; do
    decode_dir=${model_dir}/decoding_${test_set}
    decode_log=${model_dir}/decoding_${test_set}.log
    decode_manifest=data/manifest_${test_set}.json
    python ${NGRAM_PATH}/eval_beamsearch_ngram.py --nemo_model_file ${model_file} \
                                                 --input_manifest ${decode_manifest} \
                                                 --acoustic_batch_size 10 \
                                                 --preds_output_folder ${decode_dir} \
                                                 --decoding_mode greedy | tee ${decode_log}