feat: Refine api reference docs generation process #8229

stocaaro · 2025-01-31T21:12:25Z

Description of changes:

This change is process/automation only and contains no docs site changes.

Changes:

Add a command so that a user may call yarn clean-references -- -p amplify-js in the docs repo.
Make process support multiple packages
Move all needed automation into one step/process

This command will:

Look for ../amplify-js/docs/references.json
Transform the references.json file to optimize it for docs use and write the result to ./src/directory/apiReferences/amplify-js.json

This process will allow library package owners to add a workflow that:

Checks out their repo
Checks out the docs repo
Build an updated docs references.json file
Transform it into the docs repo version
Open a PR to update the docs repo
See the workflow process in the draft amplify-js PR: chore: Setup typedoc to build references.json and configure release workflow to update docs site amplify-js#14255

Which product(s) are affected by this PR (if applicable)?

Which platform(s) are affected by this PR (if applicable)?

Please add the product(s)/platform(s) affected to the PR title

Checks

Does this PR conform to the styleguide?
Does this PR include filetypes other than markdown or images? Please add or update unit tests accordingly.
Are any files being deleted with this PR? If so, have the needed redirects been created?
Are all links in MDX files using the MDX link syntax rather than HTML link syntax?

ref: MDX: [link](https://docs.amplify.aws/)
HTML: <a href="https://docs.amplify.aws/">link</a>

When this PR is ready to merge, please check the box below

Ready to merge

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

…e-refactor/main

stocaaro · 2025-03-04T19:47:18Z

src/directory/apiReferences/amplify-js.json

This change moves the config without updating it. We're moving it so that other repo's can be extracted into a reasonable structure.

stocaaro · 2025-03-04T19:48:14Z

src/references/raw-references.json

The raw file is an intermediate transformation artifact that isn't directly used. This change removes it as it serves no purpose.

Thanks for doing that! Imagine the CO2 produced when cloning and copying this file. Should it be added to the gitignore?

I'll add it, however this dir location isn't used in the new process since the file is read from the neighboring package, then the complete product is written directly into the directories, skipping over all the interstitial steps the original process had.

stocaaro · 2025-03-04T19:52:15Z

tasks/clean-references.mjs

@@ -135,13 +155,84 @@ function recursivelyPopulateNodes(node) {
  }
 }

+function processReferences(references, rootPackage) {


This logic is taken directly from the working branch that was used to generate the original references process https://github.com/aws-amplify/amplify-js/blob/feat/inline-api-docs/scripts/process-typedoc.mjs#L31-L108

This change proposes having a single transformation from the generated references.json file to the transformed / optimized apiReferences/*.json file where all of the transform logic is here in this file.

stocaaro · 2025-03-04T19:54:12Z

.github/workflows/update_references.yml

-name: Receive repository dispatch event
-
-on:
-  # Listen to a repository dispatch event by the name of `dispatch-event`
-  repository_dispatch:
-    types: [update-references]
-  workflow_dispatch:
-permissions:
-  contents: write # Used to commit updated references PR
-  pull-requests: write # Used to create updated references PR
-env:
-  GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
-
-jobs:
-  create-pull-request:


I am removing this workflow. Instead of having a trigger-able process to on the docs repo, I propose each repo have a workflow to push a PR against the docs repo as can be seen in the draft js change

AllanZhengYP

The approach makes sense to me. Thank you for doing it!
I think clean-references script needs a lot of work to make the API reference in docs look better. It seems like the node walker logic misses a lot of doc nodes, and make the page misses a lot examples we wrote in source code.

But I think we can make it a follow-up task.

AllanZhengYP · 2025-03-06T01:27:59Z

src/references/raw-references.json

Thanks for doing that! Imagine the CO2 produced when cloning and copying this file. Should it be added to the gitignore?

AllanZhengYP · 2025-03-06T01:28:59Z

src/utils/getApiStaticPath.ts

@@ -1,4 +1,4 @@
-import { API_CATEGORIES, API_SUB_CATEGORIES } from '@/data/api-categories';
+import { API_CATEGORIES, API_SUB_CATEGORIES } from '@/data/api-categories.mjs';


Is this necessary?

Interesting. If I remove the file type, then I get an error

src/utils/getApiStaticPath.ts

There wasn't a *.d.ts file there before so I'm not sure why the prior version worked :-/

stocaaro

I would agree that the clean-references could use a good refactor. Looking at the result, I think a lot of content is being skipped over in the display logic, so there are some quick wins to add comments into the doc without retooling the transformation.

From what I grox of this transformation logic, its there to:

1. Prune a bunch of exports that aren't referenced to make the file smaller.
1. Reindex the references so that access is constant time vs linear when looking up related code/type references

calebpollman · 2025-03-10T21:24:20Z

Love seeing so much red here! With the size of the diff here being so large, how are we validating there is no impact to the existing documentation

tasks/clean-references.mjs

stocaaro · 2025-03-10T22:21:02Z

tasks/clean-references.mjs

@@ -1,8 +1,8 @@
-import references from '../src/references/raw-references.json' assert { type: 'json' };


@calebpollman The original process proposed have the JS repo generate and check in this raw file, which the docs repo would then transform into the apiReferences.json file, which I have moved to scop by package name.

My update process proposes that the JS repo (and other repos) should generate their api docs json files, then use the docs repo transformation logic proposed here to transform the file directly into the finished product instead of having multiple interstitial json files checked in along the way.

I had originally included updating the amplify-js file with this change, but it seems better to do this using the automated process and leave this change a no-op for the docs.

stocaaro added 13 commits January 31, 2025 11:41

fix: Scope amplify-js references build to make room for other packages

1ef0eb2

feat: Refactor api generation

f1e1b48

Wrapping up exploration

6bd1d66

feat: Add script to parse adjacent package typedoc references

19d948f

chore: Cleanup type references with a fresh JS build

01b759e

Merge remote-tracking branch 'origin/main' into stocaaro/api-referenc…

6355e11

…e-refactor/main

Comment update

6688285

Remove amplify-data

5babf98

Remove the old apiReferences.json file

035acc3

Remove test string

ec85c23

Updates from reviewing PR

bd9fc5a

Revert references json to old version

760a6f2

PR cleanup

55315e0

stocaaro commented Mar 4, 2025

View reviewed changes

stocaaro changed the title ~~feat: Refactor api generation~~ feat: Refine api generation process Mar 4, 2025

stocaaro marked this pull request as ready for review March 4, 2025 20:06

stocaaro requested a review from a team as a code owner March 4, 2025 20:06

stocaaro changed the title ~~feat: Refine api generation process~~ feat: Refine api reference docs generation process Mar 4, 2025

AllanZhengYP previously approved these changes Mar 6, 2025

View reviewed changes

Update gitignore

99269a1

stocaaro dismissed AllanZhengYP’s stale review via 99269a1 March 6, 2025 15:56

stocaaro commented Mar 6, 2025

View reviewed changes

AllanZhengYP previously approved these changes Mar 6, 2025

View reviewed changes

cshfang previously approved these changes Mar 10, 2025

View reviewed changes

tasks/clean-references.mjs Outdated Show resolved Hide resolved

tasks/clean-references.mjs Outdated Show resolved Hide resolved

tasks/clean-references.mjs Outdated Show resolved Hide resolved

stocaaro commented Mar 10, 2025

View reviewed changes

PR comment fixes

3be1a7f

stocaaro dismissed stale reviews from cshfang and AllanZhengYP via 3be1a7f March 11, 2025 16:21

cshfang approved these changes Mar 11, 2025

View reviewed changes

AllanZhengYP approved these changes Mar 11, 2025

View reviewed changes

stocaaro merged commit a1d9d81 into main Mar 11, 2025
12 checks passed

stocaaro deleted the stocaaro/api-reference-refactor/main branch March 11, 2025 19:32

josefaidt mentioned this pull request Mar 14, 2025

feat: add field level validation docs #8287

Merged

15 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: Refine api reference docs generation process #8229

feat: Refine api reference docs generation process #8229

Uh oh!

stocaaro commented Jan 31, 2025 •

edited

Loading

Uh oh!

stocaaro Mar 4, 2025

Uh oh!

stocaaro Mar 4, 2025

Uh oh!

AllanZhengYP Mar 6, 2025

Uh oh!

stocaaro Mar 6, 2025

Uh oh!

stocaaro Mar 4, 2025

Uh oh!

stocaaro Mar 4, 2025

Uh oh!

AllanZhengYP left a comment

Uh oh!

AllanZhengYP Mar 6, 2025

Uh oh!

AllanZhengYP Mar 6, 2025

Uh oh!

stocaaro Mar 6, 2025

Uh oh!

stocaaro left a comment

Uh oh!

calebpollman commented Mar 10, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

stocaaro Mar 10, 2025

Uh oh!

Uh oh!

Uh oh!

		@@ -1,4 +1,4 @@
		import { API_CATEGORIES, API_SUB_CATEGORIES } from '@/data/api-categories';
		import { API_CATEGORIES, API_SUB_CATEGORIES } from '@/data/api-categories.mjs';

		@@ -1,8 +1,8 @@
		import references from '../src/references/raw-references.json' assert { type: 'json' };

feat: Refine api reference docs generation process #8229

feat: Refine api reference docs generation process #8229

Uh oh!

Conversation

stocaaro commented Jan 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of changes:

Checks

When this PR is ready to merge, please check the box below

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AllanZhengYP left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

stocaaro left a comment

Choose a reason for hiding this comment

Uh oh!

calebpollman commented Mar 10, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

stocaaro commented Jan 31, 2025 •

edited

Loading