Incorrect logprob sum

[ZeeshanZulfiqarAli]

The logprobs summation seems to incorrectly include the colon(:) token. This can simply be fixed by:

    def _compute_logprob_sum(self, start: int, end: int) -> float:
        token_start = self.token_indices[start+1]
        ...

[ZeeshanZulfiqarAli]

Investigated this a bit more and found that the problem is a tad bit complex: if the value is a string, then their start and end quotes (") are also considered a part of the value and the start and end reflects that. But thats obviously irrelevant if the value is a boolean or a number. So, i ended up with a conditional approach. Here is the JS version:

  _computeLogprobSum(start, end, isString=false) {
    // If string, exclude logprobs of quotes, unless it is empty string
    const increment = isString ? 0 : (end - start > 1 ? 1 : 0);
    const tokenStart = this.tokenIndices[start + increment];
    const tokenEnd = this.tokenIndices[end];
    let total = Math.exp(this.tokens[tokenStart].logprob);

    for (let i = tokenStart + 1; i < tokenEnd; i++) {
      total += Math.exp(this.tokens[i].logprob);
    }

    return total;
  }

  string(meta, children) {
    const logprobSum = this._computeLogprobSum(
      meta.start_pos,
      meta.end_pos,
      true
    );
    return logprobSum;
  }

Side note: I am using a Js port of the lark parser (lark.js)