Suggestion: Handling of null and Boolean Fields

[luizbezerra-pinheiro]

Hello there,

Here goes my two cents...

I have encountered an issue when handling null and boolean fields while using your code. Although I have not conducted an extensive impact analysis, our implementation appears to work correctly with the following modifications.

Proposed Changes (on structured_logprobs/helpers.py)
1. Update the JSON grammar to treat "true", "false", and "null" tokens as valid values. For example:

json_grammar = r"""
    start: value

    ?value: object              # '?' allows the rule to return the value directly without creating an extra parse tree node.
          | array
          | string
          | SIGNED_NUMBER -> number    # '-> number' specifies an alias for the rule.
          | "true" -> true   #  here
          | "false" -> false   #  here
          | "null" -> null   #  here

    array  : "[" [value ("," value)*] "]"
    object : "{" [pair ("," pair)*] "}"
    pair   : key ":" value
    key    : ESCAPED_STRING

    string : ESCAPED_STRING

    %import common.ESCAPED_STRING
    %import common.SIGNED_NUMBER
    %import common.WS
    %ignore WS
"""

2.	Log Probability Computation for null

In the Extractor.null method, compute the logprob_sum similarly to how it is handled for other tokens:

@v_args(meta=True)
class Extractor(Transformer_NonRecursive):
    [...]
    def null(self, meta: Meta, children: list[Token]) -> float:
        logprob_sum = self._compute_logprob_sum(meta.start_pos, meta.end_pos)
        return logprob_sum
    [...]

[ODemidenko]

The changes suggested above helped me resolve issues parsing Optional and bool values.

[amrothemich]

Same!

[eliassoares]

Same here, and the openai version is so old.