Consider adding `sonic.ConfigStd` to the JSON benchmark result

[bimoadityar]

Hi, thanks for the great investigation in jsonbench. I, and I assume some other people, refer to the investigation result to consider which JSON library to use for our usecases.

Now, after some investigation, I discovered that on sonic, there are multiple behavior-changing toggles that may affect the correctness at the cost of performance. Particularly, sonic's ConfigStd seems to aim for more compatibility with encoding/json and better correctness, passing the validateUTF8 test and TestSuite's unescaped control char, newline, and tab testcases.

Now, you might argue that the default config is the one mostly used by the users and better reflects the end-user experience. However, I am thinking that:

1. From the star counts, it seems reasonable that there may be more user of sonic's ConfigStd rather than some libraries like sonnet.
2. If the goal of the document is to give balanced overview to JSON library users on what alternatives to use and their pro cons, then telling the user about the existence of this sonic's ConfigStd, which I think is a really valid alternative to encoding/json, would make a lot of sense, regardless of the previous users counts. I assume that is why you also put JSONv1in2 in the comparison?

In conclusion, I would appreciate it if you would consider adding sonic.ConfigStd to the benchmark result, that may help other future JSON users like me that would want to pick the fastest implementation while also caring about correctness. Thanks for your time!

[dsnet]

Hi, thanks for mentioning ConfigStd. I was not aware of that option when we first created the benchmarks.

Given that:

1. adding another row for sonic.ConfigStd will complicate the explanation of the different labels,
2. there is a relatively little improvement on the JSONTestSuite of 3 cases (while still failing 25 cases), and
3. only around 1% of all sonic usages refer to ConfigStd,

I don't think it's worth it to add another row just for this particular variant of sonic.

There isn't a particular strong rule for what get's added or not, but:

• JSONv1in2 is a distinctly different package, so it's easy to explain what it is. Also, it's what the new implementation of v1 "encoding/json" will become, so it's important that we surface what it's behavior and performance is like.
• While Sonnet has relatively few users, it is inspirational in that it has great performance for an implementation that avoids unsafe. The performance of Sonnet serves as a potential high water mark for what JSONv2 could theoretically achieve in the future by possibly adopting some of their performance ideas.

[bimoadityar]

Hi, thanks for the response.

There is a relatively little improvement on the JSONTestSuite of 3 cases (while still failing 25 cases),

Yes, I see that there are still some failing JSONTestSuite testcases even in ConfigStd. I also think that this is a correctness bug and have tried raising this to the maintainer. However, I feel it is also worth noting that the TestParseSuite in the jsonbench does unmarshal to json.RawMessage which in sonic (and possibly other JSON library) triggers a separate logic to skip or find the end position of a single JSON object. When unmarshaling to interface{}, sonic's ConfigStd actually has the exact behavior as encoding/json.

I don't think it's worth it to add another row just for this particular variant of sonic.

No worries. Thanks for your consideration. I have already stumbled into this sonic's ConfigStd by myself and collect my own test results and benchmarks. I was just raising this issue because I felt it would be helpful for other people approaching the document for the same reason as me. Maybe can also consider adding a small note to tell about these toggles existence in sonic that may trade-off performance with correctness/compatibility.

P.S. Can I ask if you have a complete testing suite to check the compatibility between a JSON library with encoding/json, that is they output the same value/bytes/error after unmarshal or marshal? I see that GoJSON do fuzzing but I am not sure about its completeness, particularly when handling error case.