Sketch: Schema validation strictness levels

[tom-sherman]

Background

There is a lot of room for interpretation in the atproto spec around when to apply validation to records and at what level of strictness to apply it at. Lexicons are specified to be extensible via a few ways; open unions and object properties come to mind, but there may be others.

Handling open unions

Let's zoom in on a specific example. Imagine we have a record that has an unknown type inside of an open union, like so:

{
  "$type": "app.bsky.feed.post",
  "text": "Walked into my local pizza place, \"Hello Brian, no medium pizza today?\"\n\nHow did he know my name? I place like 1 order a month max. Legit thought they were running some AI camera stuff but, after calling me Brian 4 more times, it was clear he was trying to learn names. +$4 tip.",
  "embed": {
    "name": "Plaza Pizza",
    "$type": "community.lexicon.embed.geo",
    "latitude": "40.791200096394995",
    "longitude": "-74.05726919273859"
  },
  "createdAt": "2025-03-24T03:03:28.293Z"
}

community.lexicon.embed.geo isn't defined as a type in the embed union on app.bsky.feed.post, which is perfectly valid according to the spec. However there are questions that arise when it comes to resolution and validation of this record:

• Should a validator ever attempt to resolve an unknown type inside an open union within a document?
• If the answer is "yes", what should the behaviour be? There are three cases to consider:
  • community.lexicon.embed.geo resolves to a valid type and the embed value is valid according to that type
  • community.lexicon.embed.geo resolves to a valid type but the embed value is invalid according to that type
  • community.lexicon.embed.geo does not resolve to a valid lexicon type
• If the answer is "sometimes", what are the best practices around this?

I'm using "validator" above to refer to validation libraries, CLIs, services, or dev tools like https://atproto-browser.vercel.app or lpm

I suspect the answer lies somewhere within "it depends" territory, and that there are levels of strictness that you may want to use depending on your use case.

The proposal

Define a model of strictness levels ranging from very lax to very strict. This will allow tooling across implementors and programming languages to agree on terminology and behaviour.

This model can be proved out first into tooling within lexicon.community such as lpm, and then potentially upstreamed into atproto specs or guidance.

Strictness model

There is a matrix of different levels with the dimensions of:

• Extra properties allowed: yes/no
• Open union unknown types: may not resolve, must resolve but may not validate, must resolve and must validate

I'd suggest we bikeshed names for each level so that eg. APIs in libraries can have a "strictness" setting with well-known semantics. There may be more dimensions that I'm not thinking of right now and not every combination may make sense as a level in the model.

[ngerakines]

On the example, I've reached out to Brian M to get them to use one of the existing location types. In this case it should be community.lexicon.location.geo. Doing so would make some of this a non-issue.

[ngerakines]

I think validation modes is a great idea. I propose a few things for consideration:

• The most strict form of validation should enforce both schema and best practices (linting)
• The most strict form of validation should be deep and recursive, validating the entirety of the record.
• Some form of validation may only enforce the root object type
• Some validation may ignore types that it can't resolve

This is a future problem, but we should prepare for the scenario where a lexicon schema definition is private and cannot be resolved. In those cases, the validation logic wouldn't be able to tell if the type doesn't exist or if it just doesn't have access to it.

[bamnet]

It's me, hi, I'm the problem, it's me

I will bikeshed in a slightly different direction than intended - I was surprised that (AFAICT) there aren't client libraries which help use the lexicon correctly when writing code. I initially expected a JS/TS library to help me use these lexicons with some degree of type safety that would have made VSCode / TS highlight my mistake. In the absence of an existing reference implementation to check my usage against, I vibe-coded straight into an invalid type.

(I found AppBskyFeedPost.isRecord and AppBskyFeedPost.validateRecord to be useless in my app and difficult to debug when they error out, especially when I can post successfully)

[aeshna-cyanea]

afaict the python atproto sdk has code generation for lexicon specific namespaces/data types for use in your editor. haven't tested it personally tho

[tom-sherman]

That's a really good point and a use case I hadn't been thinking about before; Application developers that are focusing on extension points in an existing lexicon.

[tom-sherman]

I'm imagining some tooling that allows you to take an existing lexicon and inject an "extra" variant into an open union so that your application can validate it.