Plz help me! Are UTF-16 surrogate pairs correctly handled?

[fcostello85]

I'm using protobuf-json-mapping to parse a JSON string to a Protobuf generated Struct.
I faced an issue parsing a json property with a value like this:

{
  "message": "Hi \ud83d\udc51"
}

In this specific case the combination of the 2 surrogate is an Emoji -> 👑

The error is LexerError:IncorrectUnicodeChar.

It seems that the parser is not combining the surrogate pair in order to form a single Unicode character.

Is there something I'm doing wrong?

Thanks!

[stepancheg]

Quite likely there is a bug in JSON parser.

[fcostello85]

Yes, I found the function that is causing the issue under this file:
rust-protobuf/protobuf-support/src/lexer/lexer_impl.rs

on this function:

pub fn next_json_char_value(&mut self) -> LexerResult<char> {
        match self.next_char()? {
            '\\' => match self.next_char()? {
                '"' => Ok('"'),
                '\'' => Ok('\''),
                '\\' => Ok('\\'),
                '/' => Ok('/'),
                'b' => Ok('\x08'),
                'f' => Ok('\x0c'),
                'n' => Ok('\n'),
                'r' => Ok('\r'),
                't' => Ok('\t'),
                'u' => {
                    let mut v = 0;
                    for _ in 0..4 {
                        let digit = self.next_hex_digit()?;
                        v = v * 16 + digit;
                    }
                    Self::char_try_from(v)
                }
                _ => Err(LexerError::IncorrectJsonEscape),
            },
            c => Ok(c),
        }
    }

it seems that is reading the 4 char after \u in order to check if it is a valid Unicode Char and not taking in consideration that there can be a UTF-16 surrogate pair right after that.

[fcostello85]

Is there a way to prevent this error and make it work?

[fcostello85]

Proposed a fix with this PR #769