Use JSONv2 when production ready

[zamicol]

Instead of making various issues for various JSON concerns. I'm going to use this issue to track any concerns with JSON.

Coze needs strictly defined JSON capabilities. Some of these capabilities are not provided by the standard
Go library, or misbehaves under certain circumstances. Currently, there are no known 3rd party libraries that we consider suitable for Coze. Our hope is that we can use JSONv2 when production ready and that will solve any JSON concerns.

A new JSON library should minimally include these characteristics in addition to
the existing behavior of the standard library:

• Duplicate fields should error.
• Preserve order of JSON fields.
• Invalid UTF-8 should error.
• JSON should not be HTML escaped. (For example when containing the characters &, <,>. and while not adding an additional new line as the standard library currently does.)

Three of the larger obstacles:

• encoding/json: add Decoder.DisallowDuplicateFields
• json.Marshal escapes perfectly JSON valid &, <,>
• • encoding/json: no way to preserve the order of map keys

To resolve these obstacles, Coze implemented orderedmap, its own custom JSON unmarshaler, and a "checkDuplicate" helper function.

A search of other Go JSON issues that may be relevant to Coze:
https://github.com/golang/go/issues?q=is%3Aissue+is%3Aopen+encoding%2Fjson+in%3Atitle

JSONv2

See https://pkg.go.dev/github.com/go-json-experiment/json
There are other best practices in JSONv2.

[peterbourgon]

JSON is defined by a spec which doesn't guarantee any of the characteristics you list here. If you want to rely on them you're no longer accepting JSON input, you're accepting something else, based on JSON, but with additional constraints. The JSONv2 library doesn't really solve anything, it encodes stuff in a certain way, but that encoded form isn't guaranteed by JSON and can't be relied upon when decoding input.

[zamicol]

I think JSON spec concerns are addressed on the other issue and that's probably the better place for discussion on Coze's relationship to the JSON spec: #10 (comment)