Is this distribution normal？

[Checunmily]

New issue checks

• I did not find an existing feature request.

Dorado subcommand

Basecaller

Feature request

Using dorado V5.0.0SUP model doing basecalling we got a quality value distribution like this:

I want to ask that is this distribution normal because we got a mean base qv over 42 and a large number of base qv at 50 which is quite different from the older basecalling models and I think it is too high.

Is there any change in calculating the quality value in the new model? Or just the newest dorado models can reach a performance of Q40/Q35?

I mean no offence but if we can get a higher distribution of basecalling quality, why not remove the limitation of top qv which is 50?

[malton-ont]

Hi @Checunmily,

Is this plot the individual base scores? Yes, that looks pretty typical for a SUP model, but individual base scores are not that informative - if you plot the mean qscore value of the set of reads, you'll find that it probably peaks lower down (likely around 25-26 for the SUP model).

Base values:

Mean read qscores:

This is because the "mean qscore" is not just the mean of the qvalues of each base - it's the qscore derived from the mean error of each base, which is not the same. See the docs here for more information.

[Checunmily]

Hi @Checunmily,

Is this plot the individual base scores? Yes, that looks pretty typical for a SUP model, but individual base scores are not that informative - if you plot the mean qscore value of the set of reads, you'll find that it probably peaks lower down (likely around 25-26 for the SUP model).

Base values:

Mean read qscores:

This is because the "mean qscore" is not just the mean of the qvalues of each base - it's the qscore derived from the mean error of each base, which is not the same. See the docs here for more information.

I see, it's really helpful, thank you!

[Checunmily]

Hello, me again.
Based on the discussion above I did some test. I found the code in (https://github.com/nanoporetech/dorado/tree/release-v1.1/dorado/basecall/decode/beam_search.cpp)

for (size_t i = 0; i < seqLen; ++i) {
      sequence[i] = alphabet[int(sequence[i])];
      baseProbs[i] = 1.0f - (baseProbs[i] / totalProbs[i]);
      baseProbs[i] = -10.0f * std::log10(baseProbs[i]);
      float qscore = baseProbs[i] * scale + shift;
      qscore = std::clamp(qscore, 1.0f, 50.0f);
      qstring[i] = static_cast<char>(33.5f + qscore);
  }

Which shows that Dorado will perform a linear transformation and then clamp the quality value between 1 and 50.

I want to do some calibration and I tried to broaden the limit to 1~90. Actually I've got some problems when compiling dorado, and finally I can only run dorado under cpu mode, then I got a distribution of base qv like this based on several reads:

I think this better matches my expectation, so I reopen this issue again and want to ask several questions:

1. Would you please provide some guidance about how to compile dorado correctly?
2. Why dorado do the linear transformation and why set the upper limit 50?
3. Would it be better to remove or broaden this upper limit in the future? because in the high performance models it seems most bases will have a quality of 50, and this may miss some context information and maybe let something like variant calling tools confused? If you need more evidence, I'd like to help.

[malton-ont]

@Checunmily,

Information on how to compile dorado is provided in this document. Note that it is slightly out of date - you will require CUDA 12.8 and gcc-11 minimum. If you have specific issues, please open a separate ticket and provide the necessary information for us to help.

The linear transformation is a calibration between the model's estimated confidence values and qscores.

As you can see, dorado used to use an upper limit of 90.f (see here) back in v0.2.1. The upper limit is set because per-base qscores are not particularly informative at high qscores - error estimates of 0.99999 vs 0.999999999 - and clamping the value improved robustness when moving to fp16 calculations (see https://github.com/nanoporetech/dorado/blob/v1.1.1/CHANGELOG.md#022-04-apr-2023).