Sketch: Lexicon package manager

[tom-sherman]

This is a sketch, it doesn't have many details and is more about sparking discussion.

Now that lexicon authority and resolution is rolling out, it's possible to build a lexicon package manager.

Why?

• For custom-lexicon implementors, it would allow you to fetch lex docs into your repo instead of copying/pasting or using git submodule hacks
• For infra operators, a lex package registry would allow lex resolution as a service
• Ideally such a registry would allow for verification
• I'm sure there will be other tools that do this, but a registry will allow for exploring a lexicon's evolution
• Data around which versions are depended upon and by who is helpful for Lexicon authors

How?

Two main components are needed: a lexicon registry and a package manager CLI

The lexicon registry would store lex docs against their CID by subscribing to the firehose. Ideally it would also store the dependent lexicon CIDs (the refs) as well, for reliable resolution

The CLI can then use this registry to fetch lexicons and store them on disk in a standard sub directory so that they can be used by tooling such as lex-cli or atcute

Open questions

• Do we need to implement versioning? My hunch is that it's not needed, latest or pinned to a specific CID is fine
• This opens up questions about what CIDs the lexicon ref dependencies should be pinned at. Maybe just the CID of the ref lexicon at the time the parent is cached? At least this would get you reproducibility within a single registry
• Bike shed: name? I propose lpm
• Do we need to concretely describe (with a lexicon) how a lex doc is defined? The resolution spec leaves this unspecified right now but there is a schema in use in practice. I think this is required to resolve refs.

[tom-sherman]

How can we make sure lexicon docs are verifiable without the actual record existing on the PDS? This is a requirement for fetching historic lex docs, where a new version has been published to the PDS

[tom-sherman]

Registry interface design

I think the registry appview interface would be pretty simple: given an NSID and an optional CID, return a list of lexicon documents along with URIs and CIDs. When provided a CID the registry should return the full tree of lexicons for that specific CID. When the CID is omitted the registry should return the current tree at that point in time.

{
  "lexicon": 1,
  "id": "community.lexicons.lexicon.resolveLexicon",
  "defs": {
    "main": {
      "type": "query",
      "description": "Resolve a lexicon from an NSID and optional CID to a list of all lexicon docs required to validate the relevant record.",
      "parameters": {
        "type": "params",
        "required": ["nsid"],
        "properties": {
          "nsid": {
            "type": "string",
            "format": "nsid",
            "description": "The NSID of the lexicon."
          },
          "cid": {
            "type": "string",
            "format": "cid",
            "description": "The CID of the version of the lexicon record. If not specified, then return the most recent version."
          }
        }
      },
      "output": {
        "encoding": "application/json",
        "schema": {
          "type": "object",
          "required": ["schemas"],
          "properties": {
            "schemas": {
              "type": "array",
              "items": {
                "type": "object",
                "required": ["record", "ref"],
                "properties": {
                  "record": {
                    "type": "ref",
                    "ref": "com.atproto.lexicon.schema"
                  },
                  "ref": {
                    "type": "ref",
                    "ref": "com.atproto.repo.strongRef"
                  }
                }
              }
            }
          }
        }
      },
      "errors": [{ "name": "LexiconNotFound" }]
    }
  }
}

[tom-sherman]

It's possible for the registry to have a partial tree of lex docs in it's cache, this could happen if a lexicon in the tree hasn't been published yet. In this scenario I'd expect resolveLexicon to return with an error.

[tom-sherman]

How do we account for changing ownership of lexicons? It's possible for an NSID to be published under did:web:foo.com and then be published under did:web:bar.com. Maybe even with the same CID? If we have indexed both, which one should we resolve to?

[verdverm]

The Lexicon Resolution proposal specifies how lexicon definitions are to be resolved. Are you proposing an alternative means?

[tom-sherman]

Ah, good point. On changing of ownership the previous record (in the previous repo) should be invalidated.

[ngerakines]

I think the official and practical answer is the "social consensus" that @bnewbold mentioned in response to bluesky-social/atproto#3074 (comment). If a domain is lost then it's going to be up to clients to hard-code and agree on it.

[tom-sherman]

How do we manage transitive dependencies?

The fetched set of lexicons on disk probably ought to contain a single version per lexicon in the tree so that they can work with existing tooling. But what if two direct dependencies depend on different versions of the same lexicon?

For example, imagine two lexicons: com.foo and com.bar. com.foo depends on com.baz#a and com.bar depends on com.baz#b

Given a lexicon manifest like so:

com.foo#x
com.bar#y

We would end up with a dependency tree like so:

├── com.foo#x
│   └── com.baz#a
└── com.bar#y
    └── com.baz#b

However the resultant file system view probably ought to look like

.
└── com/
    ├── foo.json
    ├── bar.json
    └── baz.json

So, which version of baz do we pick?

[tom-sherman]

One way to resolve this conflict is just to pick one in a deterministic way (eg. sort the CIDs and choose the first one).

[verdverm]

MVS is a great algorithm for deterministic dependency selection (i.e. go mod)

Deterministic selection means we need a way to order / sort different versions of a lexicon

• https://go.dev/ref/mod
• https://research.swtch.com/vgo-mvs (part 4 of a series)

[verdverm]

What need is there for a centralized registry? Is not the Lexicon Resolution proposal and publishing of records the equivalent?

[verdverm]

Do we need to implement versioning? My hunch is that it's not needed, latest or pinned to a specific CID is fine

Once a record is updated, prior CIDs are not recoverable. I'm still of the opinion that we need better versioning in Lexicon than CID or the unused revision int. A number of various schemes for versioning exist in the wild already

[verdverm]

Thought experiment

1. Lexicon foo "v0.1" is published
2. Some third party indexes it
3. Lexicon foo "v0.2" is published

How can I ever verify step 2 was done correctly after step 3? Where does provenance stem from?

[verdverm]

step 0 for me is getting a concept for record history into the spec, so when edits happen, we can access prior revisions

This is important or useful to a number of features users are requesting and would help with the above

Something like this, but properly part of spec and PDS implementation (https://verdverm.com/blog/adding-record-editing-with-history-to-atprotocol)

[ngerakines]

Do we need to implement versioning? My hunch is that it's not needed, latest or pinned to a specific CID is fine

Strong agree here. I think any attempt to wrangle versioning into this is going to create a divergence when versioning is formally introduced down the road. I think it's smart to just treat all lexicons as HEAD versions.

[ngerakines]

Thinking more about this, I think treating everything as a "HEAD" version unless there is a lock file entry for a specific CID is the way to go. That way, if a tool wants to lock against a version (that is cached, big caveat there), they can do so and it won't attempt to fetch newer versions of it.

[tom-sherman]

Agreed. Although it gets slightly weird to think about the dependencies of a lexicon locked to a CID.

Should it resolve the HEAD of those dependencies? The CID of those dependencies that existed at the time the lexicon was crawled? Or something else?

The second option makes sense to me I think, it gives you a "point in time" tree which I think is desirable.

[ngerakines]

This looks great. Having a reference tool to resolve and download lexicon files would be a huge win for people and orgs developing against libraries and applications that need to download and resolve lexicon schema definitions for development, testing, and packaging.

[bnewbold]

I haven't looked through everything here and in #48, but wanted to share my current thinking on all this:

• "Lexicon vendoring" in to software project git repositories as a pattern (described more below)
• we are planning to add lexicons for resolving NSIDs (both resolve-to-DID and resolve-to-schema) to the com.atproto.* namespace, and implement those as part of a "lexicon aggregator" service. https://lexidex.bsky.dev/ is a prototype of such a service, but we plan to have something more polished before too long (eg, probably summer?)
• auth scope are going to use lexicons, and will require low-latency and potentially frequent lexicon resolution at the PDS. we'll probably use something like lexidex to help with caching and resolution, and potentially governance/safety things (like loss-of-domain-registration and other "incidents"). we are hoping to have some kind of draft or memo or at least sketch of this sytem to share at atmosphere conf (aka, in two weeks)
• some ways to version lexicons we are thinking of: CID of the record (works today, strong; not well-ordered); TID of the repo rev when record was updated: compact, time-like, well-ordered, sort of indirect to look up; potentially a semver string in Lexicon schema files (or a TID, or an integer) as a declared version
• lexidex currently tracks a bunch of this history and versioning info about lexicons, but doesn't expose it all via UI or API yet
• if you haven't yet, take a look at the run-time resolution and lexicon validation features in in the indigo:atproto/lexicon implementation: https://pkg.go.dev/github.com/bluesky-social/indigo/atproto/lexicon. has flags at validation time (eg, "leniency", could also impact things like needing to know open union types); handles recursive validation; works with compile-time, from-directory, from-network, and caching scenarios

Lexicon Vendoring Proposal/Concept

This is a minimal workflow i'm thinking of using for Bluesky projects in the next ~6 months or so, as an evolution of the "all Lexicons in the Bluesky atproto repo" situation.

• each software git repo contains directory (nested) with Lexicon schemas as JSON files
• each file is a single NSID, and matches the published record contents. might or might not include $type. would not include the "wrapper" metadata included by getRepo (like the CID, DID, etc)
• files would be in sub-directories separated at least by "registered domain" part of NSID. Could be full nested NSID format (like lexicon/com/atproto/sync/subscribeRepos.json), or just top-level folders like lexicon/com.atproto/com.atproto.sync.subscribeRepos.json. The directory and file naming doesn't really matter, this is just a way to distinguish "locally maintained" lexicons from "externally maintained" lexicons
• all dependent Lexicons would be included, recursively. each software git repo should contain the full "catalog" of Lexicons it might need at compile-time
• code generation and compile-embedding is done via the local copy, not by hitting the network (eg, without remotely resolving NSIDs)
• changes to "locally maintained" lexicons are versioned through git, then a maintainer script helps "publish" them to the network (upload to an atproto repo). publication helper might display changes as a diff, confirm deletions, do syntax and lint checks, etc.
• another helper script would resolve and "pull" updates from the network, recursively
• as an escape-hatch, it isn't hard to manually edit or directly copy files between git repos

This is less ambitious that the dependency tracking stuff you have shared. I think things like the UI for visualizing dependency tree are really great! This is just sharing an idea/approach.

[snarfed]

For libraries, vendoring absolutely makes sense. A number of existing projects do it afaik, including lexrpc: https://github.com/snarfed/lexrpc/tree/main/lexrpc/lexicons

ut of curiosity though, would you still advocate for vendoring for services? Long-lived local copies still definitely make sense for them, but I wonder if they might be better off storing and handling them as data instead of bundled with code. Could resolve new lexicons, store them long-lived, etc.

[tom-sherman]

Lots to digest here 👀

I'm gonna pause working on this to see what shakes out of atmosphere conf then, very exciting!

if you haven't yet, take a look at the run-time resolution and lexicon validation features in in the indigo:atproto/lexicon implementation

StrictRecursiveValidation is interesting, I suppose that's something that the PDS would want to disable on createRecord for example?

[bnewbold]

@snarfed for services, I think it depends! to use an example, I think Ozone would probably want to vendor in and reference local copies. Ozone currently does some com.atproto.* endpoints, and the tools.ozone.* lexicons, then also has specific support for app.bsky.*. I can pull most of these via the @atproto/api package. In the future, it might add support for more apps, eg render whtwnd blog post markdown, instead of just showing raw JSON. For that, it might do codegen locally. Maybe it does that via a library/package? But might vendor in those lexicons, and update them manually.

For the PDS, much more likely to resolve and validate "live", and not have really any Lexicons vendored (other than com.atproto.*).

My guess is that the number of services which do "live" resolution will actually be pretty small: PDS, development and debugging tools. In comparison, there would end up being many, many little appviews, feedgens, bots, and user tools, all of which can use static versions of Lexicons.

@tom-sherman as to what the PDS does by default, good question! There is a fair amount of wiggle room to figure out. It is also possible/allowed for PDS to just not really try to validate at all, by default: the validation flag is three-state, "must validate", "must not validate", "default", and "default" isn't really defined. I think a bare-bones PDS which doesn't do live-network lexicon resolution+validation is probably fine? as long as "most" PDS implementations do support it, and even do it by default.