JsonSchemaExporter.GetJsonSchemaAsNode using TransformSchemaNode example results in duplicate schema information

[scott-dak]

I'm trying to avoid generating schemas via System.Text.Json with duplicate custom attributes when referenced subschemas are created (the attribute is already declared in the referenced schema, but using the Microsoft example on how to add custom attributes, the attribute gets declared a second time within the element using the reference).
 
Following the example provided at
https://learn.microsoft.com/en-us/dotnet/standard/serialization/system-text-json/extract-schema#transform-the-generated-schema
to transform a generated schema to support the System.ComponentModel.DescriptionAttribute causes the "description" attribute to be listed twice when a generated schema results in use of references (via $ref).

In the example below, I start with the exact code from the link provided above, only I change the Person.Name property from a simple string to be an object that contains two child strings (First and Last). Then instead of generating a schema for the Person object, I generate a schema for an object containing two different Person objects so that the generated schema simply makes the second Person reference the first person's schema:

JsonSchemaExporterOptions exporterOptions = new()
{
	TransformSchemaNode = (context, schema) =>
	{
		// Determine if a type or property and extract the relevant attribute provider.
		ICustomAttributeProvider? attributeProvider = context.PropertyInfo is not null
			? context.PropertyInfo.AttributeProvider
			: context.TypeInfo.Type;

		// Look up any description attributes.
		DescriptionAttribute? descriptionAttr = attributeProvider?
			.GetCustomAttributes(inherit: true)
			.Select(attr => attr as DescriptionAttribute)
			.FirstOrDefault(attr => attr is not null);

		// Apply description attribute to the generated schema.
		if (descriptionAttr != null)
		{
			if (schema is not JsonObject jObj)
			{
				// Handle the case where the schema is a Boolean.
				JsonValueKind valueKind = schema.GetValueKind();
				Debug.Assert(valueKind is JsonValueKind.True or JsonValueKind.False);
				schema = jObj = new JsonObject();
				if (valueKind is JsonValueKind.False)
				{
					jObj.Add("not", true);
				}
			}

			jObj.Insert(0, "description", descriptionAttr.Description);
		}

		return schema;
	}
};

JsonSerializerOptions options = JsonSerializerOptions.Default;
JsonNode schema = options.GetJsonSchemaAsNode(typeof(TwoPeople), exporterOptions);
Console.WriteLine(schema.ToString());

[Description("A person")]
public record Person([property: Description("The name of the person")] NameInfo Name);

public record NameInfo(string First, string Last);

public record TwoPeople
{
	public required Person Person1 { get; init; }
	public required Person Person2 { get; init; }
}

The resulting schema now technically lists "description" twice for Person2.Name. Once directly inside Person2.Name's schema, then again indirectly via the referenced schema: "$ref": "#/properties/Person1/properties/Name".

{
  "type": [
    "object",
    "null"
  ],
  "properties": {
    "Person1": {
      "type": "object",
      "properties": {
        "Name": {
          "description": "The name of the person",
          "type": "object",
          "properties": {
            "First": {
              "type": "string"
            },
            "Last": {
              "type": "string"
            }
          },
          "required": [
            "First",
            "Last"
          ]
        }
      },
      "required": [
        "Name"
      ]
    },
    "Person2": {
      "type": "object",
      "properties": {
        "Name": {
          "description": "The name of the person",
          "$ref": "#/properties/Person1/properties/Name"
        }
      },
      "required": [
        "Name"
      ]
    }
  },
  "required": [
    "Person1",
    "Person2"
  ]
}

My current solution is to change the TransformSchemaNode code to check if the current schema contains a child named "$ref" before choosing to insert "description":

if (!jObj.TryGetPropertyValue("$ref", out _))
	jObj.Insert(0, "description", descriptionAttr.Description);

But I'm not sure if this is the correct/best/safest way to accomplish this. For example, this wouldn't work quite right if the current schema generation logic in STJ might point a property that has been attributed with a specific [Description] attribute to reference a different property (that has the same type) but a different [Description] value? In such a case the schema would probably need to be duplicated rather than referenced.

I actually encountered the issue because I am using TransformSchemaNode to add support for adding a "title" to properties using the DisplayNameAttribute. Then other software that happens to be using the Newtonsoft JSON Schema library fail to load any properties that have this extra attribute defined. I suspect the root of the issue is that schemas probably aren't supposed to list any sibling attributes to a "$ref" (all information should be contained within the referenced schema).

Regardless whether my solution is the best way to currently handle this or there is a better solution, I wonder if the example documentation from Microsoft (linked above) should be updated to show how to avoid this situation as it isn't an obvious problem and everything seems fine at first.

[elgonzo]

Just a FYI to clarify a false assumption, my post is however not touching upon your issue.

that schemas probably aren't supposed to list any sibling attributes to a "$ref" (all information should be contained within the referenced schema).

To the contrary; the current JsonSchema specification explicitly calls out that other properties are allowed next to a $ref keyword (https://json-schema.org/draft/2020-12/json-schema-core#section-8.2.3.1):

8.2.3.1. Direct References with "$ref"

The "$ref" keyword is an applicator that is used to reference a statically identified schema. Its results are the results of the referenced schema. Note that this definition of how the results are determined means that other keywords can appear alongside of "$ref" in the same schema object.

The JsonSchema spec also features an example that shows a property next to the $ref keyword (coincidentally also a "description" annotation): https://json-schema.org/draft/2020-12/json-schema-core#section-7.7.1.1

Older JsonSchema drafts/specs stipulated that other properties next to $ref are simply to be ignored. (Draft-05 being the first one with such a definition; curiously Draft-04 didn't define "$ref" whereas Draft-03 had a definition for "$ref" with the soft rule of "it SHOULD replace the current schema with the schema referenced by the value's URI." -- which, except for the "SHOULD", is the same as saying that other properties next to "$ref" are to be ignored.)

In such a case the schema would probably need to be duplicated rather than referenced.

No, because other properties next to a $ref keyword are allowed by the JsonSchema spec, hence why it would not be necessary to duplicate a schema in such a situation (see also the schema example in the specification i already linked to earlier: https://json-schema.org/draft/2020-12/json-schema-core#section-7.7.1.1).

Then other software that happens to be using the Newtonsoft JSON Schema library fail to load any properties that have this extra attribute defined.

Not sure what is going on there on your end, but a quick check using the Newtonsoft JSON Schema online validator shows that Newtonsoft JSON Schema apparently consumes your example schema without any problems:

[scott-dak]

To the contrary; the current JsonSchema specification explicitly calls out that other properties are allowed next to a $ref keyword

Thank you for the response and clarification on the incorrect assumption, as I wasn't quite sure about that. I had read the part about sibling keywords being ignored here: https://json-schema.org/understanding-json-schema/structuring#dollarref but then forgot that it was specific to an older draft.

Knowing this, I don't believe the "solution" I said I was currently using is a safe option anymore. (side note: I agree ContainsKey would have been better. I was previously using the out param from TryGetPropertyValue and simplified things for this code example)

Not sure what is going on there on your end, but a quick check using the Newtonsoft JSON Schema online validator shows that Newtonsoft JSON Schema apparently consumes your example schema without any problems:

The software having an issue is using the Newtonsoft.Json.Schema library to parse the schema then manually traverse it. So perhaps this is more an issue with the Newtonsoft library being unable to handle 'duplicate' entries caused by this scenario (if keywords alongside the $ref are intended to 'override' any matches within the referenced schema)?

To see the actual issue, you can add two lines to the end of my previously posted sample code as such (after pulling in the Newtonsoft.Json.Schema nuget package):

JsonSchemaExporterOptions exporterOptions = new()
{
	TransformSchemaNode = (context, schema) =>
	{
		// Determine if a type or property and extract the relevant attribute provider.
		ICustomAttributeProvider? attributeProvider = context.PropertyInfo is not null
			? context.PropertyInfo.AttributeProvider
			: context.TypeInfo.Type;

		// Look up any description attributes.
		DescriptionAttribute? descriptionAttr = attributeProvider?
			.GetCustomAttributes(inherit: true)
			.Select(attr => attr as DescriptionAttribute)
			.FirstOrDefault(attr => attr is not null);

		// Apply description attribute to the generated schema.
		if (descriptionAttr != null)
		{
			if (schema is not JsonObject jObj)
			{
				// Handle the case where the schema is a Boolean.
				JsonValueKind valueKind = schema.GetValueKind();
				Debug.Assert(valueKind is JsonValueKind.True or JsonValueKind.False);
				schema = jObj = new JsonObject();
				if (valueKind is JsonValueKind.False)
				{
					jObj.Add("not", true);
				}
			}
			jObj.Insert(0, "description", descriptionAttr.Description);
		}

		return schema;
	}
};

JsonSerializerOptions options = JsonSerializerOptions.Default;
JsonNode schema = options.GetJsonSchemaAsNode(typeof(TwoPeople), exporterOptions);
// Load the schema via Newtonsoft and view she schema for Person2.Name property.
var nsSchema = Newtonsoft.Json.Schema.JSchema.Parse(schema.ToString());
Console.WriteLine(nsSchema.Properties["Person2"].Properties["Name"].ToString());

public record Person([property: Description("The name of the person")] NameInfo Name);

public record NameInfo(string First, string Last);

public record TwoPeople
{
	public required Person Person1 { get; init; }
	public required Person Person2 { get; init; }
}

Without any changes to the Microsoft example schema transform documentation, the resulting output is:

{
  "description": "The name of the person",
  "$ref": "#/$ref"
}

This is 'fixed' by my initial post's proposed modification (that I don't think is a good solution anymore):

if (!jObj.ContainsKey("$ref"))
	jObj.Insert(0, "description", descriptionAttr.Description);

as you can see with the corrected output:

{
  "description": "The name of the person",
  "type": "object",
  "properties": {
    "First": {
      "type": "string"
    },
    "Last": {
      "type": "string"
    }
  },
  "required": [
    "First",
    "Last"
  ]
}

Upon further investigation, it looks like root of the issue is that Newtonsoft simply doesn't properly support use of keywords alongside (siblings to) a $ref. The issue occurs even when I manually remove the 'duplicate' description from the referenced schema so that it only appears directly in Person2.

While it still probably isn't ideal to 'duplicate' a keyword even if it would simply 'override' the referenced value to be the same value that it already is, at least the MS documentation isn't providing a sample that can generate schemas that break the rules of the spec. I'll probably reach out to Newtonsoft instead to see if it might update to support siblings to schemas using $ref.

Thank you again for taking the time to respond to my issue and the helpful feedback!

[elgonzo]

The resulting schema now technically lists "description" twice for Person2.Name. Once directly inside Person2.Name's schema, then again indirectly via the referenced schema: "$ref": "#/properties/Person1/properties/Name".

Yes, because the shown code - not STJ - explicitly adds a "description" property to the { "$ref": "..." } json object. Remember, JsonObject just represents a json object. It doesn't have awareness of anything that's JsonSchema/JsonPath/JsonReference-specific. As far as the schema generator is concerned, how could it possibly know that by adding a "description" annotation you actually didn't mean to add a "description" annotation?

My current solution is to change the TransformSchemaNode code to check if the current schema contains a child named "$ref" before choosing to insert "description":

if (!jObj.TryGetPropertyValue("$ref", out _))
jObj.Insert(0, "description", descriptionAttr.Description);
But I'm not sure if this is the correct/best/safest way to accomplish this.

I would use jObj.ContainsKey("$ref") in this instance, but pot-ah-tos pot-ai-tos...

For example, this wouldn't work quite right if the current schema generation logic in STJ might point a property that has been attributed with a specific [Description] attribute to reference a different property (that has the same type)

That's not for a simply illustrative code example in the documentation to deal with. This is getting too complicated for a code example that is meant to illustrate and educate about the TransformSchemaNode delegate, not to provide a production-ready copy&paste solution. That said, i have to admit i have no idea for a better code example for the documentation that is both short, easy to understand and avoids such complications. If anyone has an idea for a better code example, feel free to post it.

Ideally, the STJ schema generator could make many of such complications go away by not embedding referenced schemas as/into other property schemas. Similar to what JsonSchema.Net.Generation does, the STJ schema generator would have to separate schemas generated from a type from the schemas assigned to properties. The "type schemas" would be bundled in $defs with the schemas assigned to properties only referencing those type schemas. This would allow representation of .NET attribute metadata appropriately, respecting whether the attribute metadata is attached to a type or property . (JsonSchema.Net.Generation demo showing this: https://dotnetfiddle.net/JfM7d2 -- uncomment the line with SomeNameStuff to see a slightly more complex result.)