Skip to content

Shared Tree: Schema FormatV2 and codecs #24812

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 75 commits into from
Jun 17, 2025
Merged
Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
75 commits
Select commit Hold shift + click to select a range
bd8406a
Initial change.
TommyBrosman May 12, 2025
3165fe3
persistedMetadata is now backed by an object instead of a string.
TommyBrosman May 13, 2025
9709ede
Updated codec.spec.ts to cover FormatV2.
TommyBrosman May 13, 2025
ba058e1
Removed extraneous whitespace change.
TommyBrosman May 13, 2025
b819742
- Changed TreeNodeStoredSchema implementations to take the write vers…
TommyBrosman May 13, 2025
0b5c842
- Changed the stored schema implementation to handle multiple schema …
TommyBrosman May 13, 2025
ecdca7a
Merge branch 'main' into metadata-schema-3
TommyBrosman May 13, 2025
1117c2c
Apply suggestions from code review
TommyBrosman May 14, 2025
94a5c39
- Switched to using the SchemaCodecVersion in public APIs. Eventually…
TommyBrosman May 14, 2025
302cb21
Merge branch 'metadata-schema-3' of https://github.com/TommyBrosman/F…
TommyBrosman May 14, 2025
0ba34d1
- Removed the schema version constant and replaced it with SchemaCode…
TommyBrosman May 14, 2025
3275ee0
Minor: reverted changes to FluidClientVersion utils.
TommyBrosman May 15, 2025
7c48b9d
Switched back to using min client version for importCompressed.
TommyBrosman May 15, 2025
a2f4195
Minor: reverted accidental change.
TommyBrosman May 15, 2025
4786dcc
Apply suggestions from code review
TommyBrosman May 15, 2025
6fa8836
Made persisted metadata schema field naming consistent.
TommyBrosman May 15, 2025
e4064c7
- Created a separate storedSchemaDecodeDispatcher for v2 schemas.
TommyBrosman May 16, 2025
ab1b882
- Refactored the node kind dispatch.
TommyBrosman May 19, 2025
1027daa
Updated snapshots.
TommyBrosman May 19, 2025
e527eba
- Removed minimum client version changes. Equivalent changes will be …
TommyBrosman May 19, 2025
3eae1f6
Removed more minimumClientVersion glue.
TommyBrosman May 19, 2025
98e1b76
Updated API files.
TommyBrosman May 19, 2025
ecc77c9
Rename: SchemaCodecVersion -> SchemaVersion.
TommyBrosman May 20, 2025
37e900f
Merge branch 'main' into metadata-schema-3
TommyBrosman May 20, 2025
737633b
Removed top-level persisted metadata.
TommyBrosman May 21, 2025
4bb136c
Fixed comments.
TommyBrosman May 22, 2025
0bcb90f
Exposed persistedMetadata as a JsonCompatibleReadOnlyObject.
TommyBrosman May 22, 2025
a675338
Changeset.
TommyBrosman May 22, 2025
1d1ef5f
Wired up toStoredSchema. Still needs tests.
TommyBrosman May 22, 2025
b9898e8
- Updated an old snapshot that didn't include the metadata field.
TommyBrosman May 22, 2025
6ca189a
Refactor: persistedMetadata -> metadata on persisted types.
TommyBrosman May 27, 2025
85240dc
Apply suggestions from code review
TommyBrosman May 27, 2025
8883d45
- Updated the changeset description
TommyBrosman May 28, 2025
9c2c13f
Reverted unnecessary change.
TommyBrosman May 28, 2025
c62f7ec
Wired up node schema metadata persistence. Currently errors on tests …
TommyBrosman May 29, 2025
2f6b519
Reverted unneeded change.
TommyBrosman May 29, 2025
ae1779b
Removed tests for the alpha API I removed in a previous revision.
TommyBrosman May 29, 2025
7eff8ab
This revision adds simple-tree persistence for node and field schema …
TommyBrosman May 30, 2025
f3f572c
- Fixed missing persistedMetadata on fields. toStoredSchema.ts and sh…
TommyBrosman Jun 2, 2025
faaf2f7
Merge branch 'main' into metadata-schema-3
TommyBrosman Jun 2, 2025
f80d027
Removed changes for persisted format, codecs. A few tests are failing.
TommyBrosman Jun 3, 2025
a7e72f0
Minor cleanup.
TommyBrosman Jun 3, 2025
3dcfb10
Updated changeset description.
TommyBrosman Jun 4, 2025
dcf4e0e
Merge branch 'main' into metadata-in-memory
TommyBrosman Jun 4, 2025
bca4c8c
Regenerated API Extractor output.
TommyBrosman Jun 4, 2025
b66c928
Renamed the in-memory version of the persistedMetadata field to persi…
TommyBrosman Jun 6, 2025
98c853e
Merge branch 'main' into metadata-in-memory
TommyBrosman Jun 6, 2025
155b320
- Added a simple default comparison for persistedMetadata.
TommyBrosman Jun 9, 2025
4ff48d8
Minor: clarified use of persistedMetadata in tests.
TommyBrosman Jun 9, 2025
6400590
- Removed redundant alpha APIs. For now they are exposed only as static
TommyBrosman Jun 10, 2025
b329851
Reverted changes that belong in a different PR.
TommyBrosman Jun 10, 2025
6b425b2
Removed more changes that shouldn't be in this PR.
TommyBrosman Jun 10, 2025
c0eb6dc
Apply suggestions from code review
TommyBrosman Jun 10, 2025
c9a1346
Reverted another unrelated change.
TommyBrosman Jun 10, 2025
783d752
Revert "Removed changes for persisted format, codecs. A few tests are…
TommyBrosman Jun 10, 2025
f1b0c06
Revert "Reverted changes that belong in a different PR."
TommyBrosman Jun 10, 2025
0d43ee7
Revert "Removed more changes that shouldn't be in this PR."
TommyBrosman Jun 10, 2025
1a06f89
Added a changeset with code examples.
TommyBrosman Jun 12, 2025
b788c06
Merge branch 'main' into formatv2
TommyBrosman Jun 13, 2025
ac58d05
Merge branch 'main' into formatv2
TommyBrosman Jun 13, 2025
bfa3c57
- End-to-end test for enabling ST format v5
TommyBrosman Jun 13, 2025
03bce3f
Updated docs. The explanation of how to turn on persisted metadata is…
TommyBrosman Jun 13, 2025
51a6c12
Fixed indentation.
TommyBrosman Jun 13, 2025
9d42ebb
Deleted old changeset.
TommyBrosman Jun 13, 2025
52bbe1a
Merge branch 'main' into formatv2
TommyBrosman Jun 13, 2025
5abf135
Refactored fieldsObject encoding so the code is no longer duplicated.
TommyBrosman Jun 16, 2025
59db6ac
Fixed a test.
TommyBrosman Jun 16, 2025
8454d80
Apply suggestions from code review
TommyBrosman Jun 16, 2025
ef2fb5e
Refactored node schema encoding to reduce duplication.
TommyBrosman Jun 16, 2025
633c551
Merge branch 'main' into formatv2
TommyBrosman Jun 16, 2025
b145898
Small refactor.
TommyBrosman Jun 16, 2025
8a64c0a
Update .changeset/spotty-bottles-rest.md
TommyBrosman Jun 17, 2025
2b2dc5f
Tweaked documentation.
TommyBrosman Jun 17, 2025
05f00bd
Merge branch 'formatv2' of https://github.com/TommyBrosman/FluidFrame…
TommyBrosman Jun 17, 2025
cd49ae1
Merge branch 'main' into formatv2
TommyBrosman Jun 17, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions .changeset/spotty-bottles-rest.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
---
Persisted metadata for Shared Tree schemas

The persisted metadata feature for Shared Tree allows an application author to write metadata along with the schema. This feature is supported for both object node and field schemas.
The persisted metadata feature for Shared Tree allows an application author to write document-persisted metadata along with the schema. This feature is supported for both object node and field schemas.

#### Using the persisted metadata feature

Expand All @@ -32,9 +32,9 @@ const tree = configuredSharedTree({
}).create(runtime);

export const MyContainerSchema = {
initialObjects: {
appData: tree,
},
initialObjects: {
appData: tree,
},
} satisfies ContainerSchema;
```

Expand Down
11 changes: 0 additions & 11 deletions .changeset/whole-days-argue.md

This file was deleted.

49 changes: 22 additions & 27 deletions packages/dds/tree/src/core/schema-stored/schema.ts
Original file line number Diff line number Diff line change
Expand Up @@ -232,49 +232,44 @@ export class ObjectNodeStoredSchema extends TreeNodeStoredSchema {
}

public override encodeV1(): TreeNodeSchemaDataFormatV1 {
const fieldsObject: Record<string, FieldSchemaFormat> = Object.create(null);
// Sort fields to ensure output is identical for for equivalent schema (since field order is not considered significant).
// This makes comparing schema easier, and ensures chunk reuse for schema summaries isn't needlessly broken.
for (const key of [...this.objectNodeFields.keys()].sort()) {
Object.defineProperty(fieldsObject, key, {
enumerable: true,
configurable: true,
writable: true,
value: encodeFieldSchemaV1(
this.objectNodeFields.get(key) ?? fail(0xae7 /* missing field */),
),
});
}
const fieldsObject: Record<string, FieldSchemaFormat> =
this.encodeFieldsObject(encodeFieldSchemaV1);

return {
object: fieldsObject,
};
}

public override encodeV2(): TreeNodeSchemaDataFormatV2 {
const fieldsObject: Record<string, FieldSchemaFormat> =
this.encodeFieldsObject(encodeFieldSchemaV2);
const kind = { object: fieldsObject };

// Omit metadata from the output if it is undefined
return this.metadata !== undefined ? { kind, metadata: this.metadata } : { kind };
}

public override getFieldSchema(field: FieldKey): TreeFieldStoredSchema {
return this.objectNodeFields.get(field) ?? storedEmptyFieldSchema;
}

private encodeFieldsObject(
encodeFieldSchema: (storedFieldSchema: TreeFieldStoredSchema) => FieldSchemaFormat,
): Record<string, FieldSchemaFormat> {
const fieldsObject: Record<string, FieldSchemaFormat> = Object.create(null);
// Sort fields to ensure output is identical for for equivalent schema (since field order is not considered significant).
// This makes comparing schema easier, and ensures chunk reuse for schema summaries isn't needlessly broken.
for (const key of [...this.objectNodeFields.keys()].sort()) {
const value = encodeFieldSchemaV2(
this.objectNodeFields.get(key) ?? fail(0xae7 /* missing field */),
);

Object.defineProperty(fieldsObject, key, {
enumerable: true,
configurable: true,
writable: true,
value,
value: encodeFieldSchema(
this.objectNodeFields.get(key) ?? fail(0xae7 /* missing field */),
),
});
}

const kind = { object: fieldsObject };

// Omit metadata from the output if it is undefined
return this.metadata !== undefined ? { kind, metadata: this.metadata } : { kind };
}

public override getFieldSchema(field: FieldKey): TreeFieldStoredSchema {
return this.objectNodeFields.get(field) ?? storedEmptyFieldSchema;
return fieldsObject;
}
}

Expand Down
44 changes: 23 additions & 21 deletions packages/dds/tree/src/feature-libraries/schema-index/codec.ts
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,6 @@ import {
decodeFieldSchema,
encodeFieldSchemaV1,
encodeFieldSchemaV2,
type schemaFormatV1,
type schemaFormatV2,
storedSchemaDecodeDispatcher,
} from "../../core/index.js";
import { brand, type JsonCompatible } from "../../util/index.js";
Expand Down Expand Up @@ -89,18 +87,8 @@ export function encodeRepo(repo: TreeStoredSchema, version: SchemaVersion): Json
}

function encodeRepoV1(repo: TreeStoredSchema): FormatV1 {
const nodeSchema: Record<string, schemaFormatV1.TreeNodeSchemaDataFormat> =
Object.create(null);
const nodeSchema = encodeNodeSchema(repo, (schema) => schema.encodeV1());
const rootFieldSchema = encodeFieldSchemaV1(repo.rootFieldSchema);
for (const name of [...repo.nodeSchema.keys()].sort()) {
const schema = repo.nodeSchema.get(name) ?? fail(0xb28 /* missing schema */);
Object.defineProperty(nodeSchema, name, {
enumerable: true,
configurable: true,
writable: true,
value: schema.encodeV1(),
});
}
return {
version: SchemaVersion.v1,
nodes: nodeSchema,
Expand All @@ -109,23 +97,37 @@ function encodeRepoV1(repo: TreeStoredSchema): FormatV1 {
}

function encodeRepoV2(repo: TreeStoredSchema): FormatV2 {
const nodeSchema: Record<string, schemaFormatV2.TreeNodeSchemaDataFormat> =
Object.create(null);
const nodeSchema = encodeNodeSchema(repo, (schema) => schema.encodeV2());
const rootFieldSchema = encodeFieldSchemaV2(repo.rootFieldSchema);
return {
version: SchemaVersion.v2,
nodes: nodeSchema,
root: rootFieldSchema,
};
}

/**
* Shared logic for encoding node schemas.
* @param repo - The stored schema to encode.
* @param encodeValue - A function which encodes a single node schema.
* @returns The encoded node schema.
*/
function encodeNodeSchema<TFormat>(
repo: TreeStoredSchema,
encodeValue: (schema: TreeNodeStoredSchema) => TFormat,
): Record<string, TFormat> {
const nodeSchema: Record<string, TFormat> = Object.create(null);
for (const name of [...repo.nodeSchema.keys()].sort()) {
const schema = repo.nodeSchema.get(name) ?? fail(0xb28 /* missing schema */);
Object.defineProperty(nodeSchema, name, {
enumerable: true,
configurable: true,
writable: true,
value: schema.encodeV2(),
value: encodeValue(schema),
});
}
return {
version: SchemaVersion.v2,
nodes: nodeSchema,
root: rootFieldSchema,
};

return nodeSchema;
}

function decodeV1(f: FormatV1): TreeStoredSchema {
Expand Down
5 changes: 1 addition & 4 deletions packages/dds/tree/src/test/shared-tree/sharedTree.spec.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2244,10 +2244,7 @@ describe("SharedTree", () => {
describe("Shared Tree format v5 enablement via `configuredSharedTree`", () => {
it("can create a SharedTree with format v5 enabled", async () => {
// Create and initialize the runtime factory
const runtime = new MockFluidDataStoreRuntime({
idCompressor: createIdCompressor(),
attachState: AttachState.Detached,
});
const runtime = new MockSharedTreeRuntime();

// Enable Shared Tree format v5, which corresponds to schema format v2. Create a Shared Tree instance.
const tree = configuredSharedTree({
Expand Down
Loading