Automatically create JS versions of our TS code in the docs #2638

cprecioso · 2025-04-08T09:32:07Z

Description

Part of #2455

This PR allows to just write TS code directly, and automatically transform it and appear with the JS/TS selector.

It does so by adding two plugins to Docusaurus' MDX handling:

autoJSCode will find any code block with the auto-js meta, transform it with ts-blank-space, and remove the extra blank space with prettier.
Example transformation

Input:
```
 ```ts auto-js title="hello.ts"
 const foo = (n: number) => 8 as string
 ```
```
Equivalent output (actual output is done in AST):
```
 <Tabs groupId="js-ts">
 <TabItem value="js" label="JavaScript">
 
 ```js title="hello.js"
 const foo = (n) => 8
 ```
 
 </TabItem>
 <TabItem value="ts" label="TypeScript">
 
 ```ts title="hello.ts"
 const foo = (n: number) => 8 as string
 ```
 
 </TabItem>
 </Tabs>
```
Why use `ts-blank-space`?

Basically, it's the transformer that does the fewest modifications to the input code, which I think is important to keep the overall visual structure of the code in the examples. It also helps to not introduce TS-only constructs like enums that, when automatically converted, introduce distracting noise.

I also feel validated in its approach since it uses the TypeScript compiler directly to do its job, and not a custom parser that might get outdated. As well, the blank-space-replacing method is used by SWC and Node.js in their TypeScript support.
codeWithHole is made to deal with some of our examples, that just omit some syntactically needed parts of the code with .... In order for the TypeScript transformation to work correctly, this plugin allows us to use an identifier named hole that will be replaced by ..., while still being syntactically correct.
Example transformation

Input:
```
 ```ts with-hole
 const foo = (n: number) => {
 	return {
 		name: "bar",
 		hole
 	}
 }
 ```
```
Equivalent output (actual output is done in AST):
```
 ```ts
 const foo = (n: number) => {
 	return {
 		name: "bar",
 		/* ... */
 	}
 }
 ```
```

With these two plugins, we can transform our examples to be less complex to author while keeping the option to read our docs in JS or TS. If there are any examples that can't be automatically converted, or that shouldn't be converted, we still retain the option to do it manually as we've been doing until now.

I converted two examples so you can see the output, please check them out.

Martinsos

Very cool! Seems like an awesome thing to me, much less duplication! And we can still do it manually if we really need to (we can right? Would we then use those Tabs again?).

I will let others check the details of the PR and approve it, to make sure it works fully with how we do things currently (e.g. if you toggle to JS, does it remmeber the toggle? Mostly checking if we lost anything with not using our Tab system anymore). Aha but I see now that we actually do still use Tabs, they are produced by the plugin -> ok that sounds good then!

One th ing that would certainly be valuable is more documentation on this -> we should mention in docs README.md that we are using this mechanism because it is a bit magic, so having a central place with some discoverability where we describe it would be valuable.

Also, these remark/ files, we should provide some top level header comments in them, let's say one at the top of each file, to explain a bit the motivation and what are they here for, as they are a bit unexpected.
Also I didn't review the code in them, I will let the others do that.

Awesome all together!

web/docs/advanced/apis.md

cprecioso · 2025-04-08T10:28:07Z

we should mention in docs README.md

top level header comments

Yes! I wanted to get this out to get some comments on the approach but I wanted to do this, I'll get on it

Martinsos · 2025-04-08T10:48:51Z

we should mention in docs README.md

top level header comments

Yes! I wanted to get this out to get some comments on the approach but I wanted to do this, I'll get on it

No worries, I assumed so!
Advice: If you want the people to take a look at the PR, but not do a deep review -> make a draft PR. That basically communicates exactly that, that you are looking for comments, but you not done yet. While real PR means it is ready for proper reviewing.

cprecioso · 2025-04-09T13:30:15Z

Can I get a review? 🙏🏼

infomiho

I love this idea. It will make our lives much easier.

Testing

I tried with some more complex examples:

wasp/web/docs/advanced/web-sockets.md

Line 69 in 24920c0

here the tricky part are the TS specific comments (which stay in the JS version), we'd probably need to rewrite those code blocks not to use those comments and move that info somewhere outside of the code block

Another example:

wasp/web/docs/data-model/entities.md

Line 48 in 9977e27

We show Prisma code with the language switch for completeness sake - it toggles between different variants of the text below. So I guess, here we'll just use the tabs directly, you addressed it in the README, that's great 👍

In this example:

wasp/web/docs/data-model/operations/queries.md

Line 106 in 4c2a935

The JS version still has the import statement (without anything imported):

For me, the next step would be - try to convert all the code blocks to see where the current approach fails and try to address those cases. For some of the cases we'll need to modify the code (e.g. to exclude Typescript specific comments), that's okay for me.

web/README.md

web/src/remark/auto-js-code.js

web/src/remark/code-with-hole.js

cprecioso · 2025-04-14T13:06:47Z

@infomiho

try to convert all the code blocks

Are we sure converting 230 occurrences of this pattern is a good use of the team's time? 😅

I was planning on us leaving stuff as it is; writing new stuff with this plugin, and converting on the go whenever we touch a file.

infomiho · 2025-04-14T13:13:01Z

I think it's a valid use of team time since it will speed us the docs authoring process. It took me a couple of minutes per code block to do it, so it might be a few hours max IMHO.

What I looking to achieve: find the the edge cases for which the plugin might not produce 1:1 JS code. While this is still hot, while you have the context, I think it'll be good to figure out and write down the process/limits of the plugin.

Let's do it like this: invest one hour to convert as many examples as possible and try going for as many unique examples as possible 🙂

cprecioso · 2025-04-16T15:01:24Z

Turning to draft until #2658 is merged

infomiho · 2025-05-05T10:11:13Z

I suppose this PR can go forward since #2658 is merged 🎉

cprecioso · 2025-05-07T10:16:05Z

@infomiho ready for review

I tried updating the tutorial and some examples (https://github.com/wasp-lang/wasp/pull/2728/files), and the main drawbacks that I found were

We need to remember to add import type so they don't appear in JS
Edge cases:
- // highlight-next-line comment before a TS-only line will hang around and highlight the wrong line (happened twice)
- We're running prettier so in some cases where we're putting extra spaces around some lines to make them clearer and they get collapsed (only happened once)
- It doesn't replace file names' extensions in clarification comments (I'd say this problem is almost unique to the tutorial)

All in all, those are edge cases that might be out of scope for this, and they're not too impactful. We can always write the tabs manually for whatever edge case arises, so this is a "strictly better" improvement IMO. I'm comfortable merging.

infomiho

Great stuff @cprecioso, I think we are almost there. I'm glad you tested it out on more code blocks to get a better sense of caveats. I think the benefits definitely outweigh the caveats.

I've left some more comments, let me know when I can take another look.

web/src/remark/code-with-hole.ts

web/src/remark/auto-js-code.ts

web/README.md

infomiho · 2025-05-12T11:19:38Z

web/docs/advanced/apis.md

+```tsx title="src/pages/SomePage.tsx" auto-js with-hole
+import React, { useEffect } from "react";
+import { api } from "wasp/client/api";
+
+async function fetchCustomRoute() {
+  const res = await api.get("/foo/bar");
+  console.log(res.data);
+}
+
+export const Foo = () => {
+  useEffect(() => {
+    fetchCustomRoute();
+  }, []);
+
+  return <>{hole}</>;
+};
+```


One thing I noticed that when I try to auto format this code in VS Code, the code block doesn't get formatted and the code in the output is of course formatted.

Is there a way for us to configure Prettier to format the code blocks in the MDX files? I think it makes sense for code blocks in the source code to look as similar as possible to the rendered code blocks for easier searching later on.

If it's not possible to run Prettier on MDX files (this issue suggests so) should we make it a part of the docs authoring process to format the code before including it in the code blocks?

Yep, as you found out, Prettier is stuck in MDXv1 so I'd rather not run it on our MDXv3 files, it will mess up its delicate whitespace situation 😑

should we make it a part of the docs authoring process to format the code before including it in the code blocks?

You mean just adding that to the readme? I can do that.

I'd rather not have to run Prettier at all if we don't have it at the source, but it needs to run on the JS so it removes the blank space (I didn't find another type stripper that doesn't leave the whitespace), and then on the TS for consistency. 😔

Yep, I thought maybe about adding a note in the README to say, hey, format it so the source code and the rendered code look as close as possible for better searchablity.

Too bad about missing MDX 3 support, since Docusaurus moved, you'd expect it to be more demand for it.

Maybe one for the future, npx remark . --ext .mdx --output should probably format MDX 3 if used with the remark-mdx plugin. You probably know more about this than me, just throwing it out there.

I added it here BTW

wasp/web/README.md

Line 112 in beed6b9

- Run `prettier` on the code before pasting it in the document, as `auto-js` will enforce it.

- Use `unist-util-visit`, no need for `unist-util-visit-parents`. - Extract the common functions of the visitor check and the language assert to a shared file - For the async visitor, create a thunk inside of the visitor so we keep all the type narrowing instead of having to do it again

Co-authored-by: Mihovil Ilakovac <mihovil@ilakovac.com>

cprecioso · 2025-05-12T13:12:50Z

@infomiho ready

cprecioso · 2025-05-13T11:34:32Z

@infomiho if everything is ready let's merge ahead of the sprint review? 😁

infomiho

Naming comments, almost there

web/src/remark/util/code-blocks.ts

web/src/remark/auto-js-code.ts

web/README.md

infomiho

Great addition

Martinsos

I am writing remark plugin myself (coderef checker) and therefore got interested in this PR, so I started reviewing this PR while it was still unmerged, but it got merged in the meantime! Which is ok, I joined the party really late.

I however finished the review, so here it is. We can do initial discussion of items here, while you do the fixes in the new PR, and then we switch to that one after the first iteration of changes.

web/README.md

web/src/remark/auto-js-code.ts

web/src/remark/code-with-hole.ts

cprecioso · 2025-05-22T08:38:23Z

@Martinsos's review addressed in #2788

Martinsos · 2025-05-29T12:31:10Z

Ok we took care of everything that was being discussed in this merged PR, the only conversation left is in the new PR (#2788 ), continuing only there.

cprecioso requested review from Martinsos and infomiho April 8, 2025 09:32

cprecioso changed the title ~~Automatically create JS versions of our TS code~~ Automatically create JS versions of our TS code in the docs Apr 8, 2025

cprecioso linked an issue Apr 8, 2025 that may be closed by this pull request

Update docs for userSignupFields to be consistent between JS and TS code examples #2455

Closed

cprecioso removed a link to an issue Apr 8, 2025

Update docs for userSignupFields to be consistent between JS and TS code examples #2455

Closed

Martinsos reviewed Apr 8, 2025

View reviewed changes

web/docs/advanced/apis.md Outdated Show resolved Hide resolved

web/docs/advanced/apis.md Outdated Show resolved Hide resolved

cprecioso added 4 commits April 8, 2025 12:32

Automatically create JS versions of our TS code

3e59636

Apply two examples

e0ee478

Add file comments

793b50f

Refactor some code to make it more understandable

4002350

Add readme docs

3bd0591

cprecioso force-pushed the auto-js-docs branch from 50e0e81 to 3bd0591 Compare April 8, 2025 11:00

cprecioso self-assigned this Apr 8, 2025

infomiho requested changes Apr 11, 2025

View reviewed changes

cprecioso added 7 commits April 14, 2025 14:24

Comments in readme

fd19f6a

Fix admonition

b255f6b

Typo

56112f3

More fixes

b954e89

Simple changes

49d7321

Find the correct prettier confirm

1f68723

Extract functions

f41ef1e

cprecioso marked this pull request as draft April 16, 2025 15:01

cprecioso changed the title ~~Automatically create JS versions of our TS code in the docs~~ WIP: Automatically create JS versions of our TS code in the docs Apr 25, 2025

Correctly parse JSX

c9d54e2

cprecioso marked this pull request as ready for review May 7, 2025 10:41

cprecioso changed the title ~~WIP: Automatically create JS versions of our TS code in the docs~~ Automatically create JS versions of our TS code in the docs May 7, 2025

infomiho reviewed May 12, 2025

View reviewed changes

cprecioso and others added 8 commits May 12, 2025 13:35

Add vfile reporting

2abdc9b

Apply suggestion

d968565

Co-authored-by: Mihovil Ilakovac <mihovil@ilakovac.com>

Apply suggestion

aeec5a8

Apply comment

36eb43a

Co-authored-by: Mihovil Ilakovac <mihovil@ilakovac.com>

Apply suggestion

f7f3d69

Extract prettier config

dbed0a0

Add caveats to readme

beed6b9

Trim final newline

0154adc

cprecioso mentioned this pull request May 13, 2025

Update examples with auto-js #2728

Merged

infomiho reviewed May 13, 2025

View reviewed changes

web/src/remark/util/code-blocks.ts Outdated Show resolved Hide resolved

web/src/remark/auto-js-code.ts Outdated Show resolved Hide resolved

web/README.md Show resolved Hide resolved

cprecioso added 2 commits May 13, 2025 13:58

Rename fn

7d4bd83

Better explain the transforms

3d6253d

infomiho approved these changes May 13, 2025

View reviewed changes

cprecioso merged commit 5420929 into main May 13, 2025

cprecioso deleted the auto-js-docs branch May 13, 2025 12:07

Martinsos reviewed May 14, 2025

View reviewed changes

cprecioso restored the auto-js-docs branch May 15, 2025 13:35

cprecioso deleted the auto-js-docs branch May 15, 2025 13:37

sodic mentioned this pull request May 22, 2025

Address final review of #2638 #2788

Merged

cprecioso added a commit that referenced this pull request Jun 4, 2025

Address final review of #2638 (#2788)

8c49210

Uh oh!

Automatically create JS versions of our TS code in the docs #2638

Automatically create JS versions of our TS code in the docs #2638

Uh oh!

Conversation

cprecioso commented Apr 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Uh oh!

Martinsos left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

cprecioso commented Apr 8, 2025

Uh oh!

Martinsos commented Apr 8, 2025

Uh oh!

cprecioso commented Apr 9, 2025

Uh oh!

infomiho left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Testing

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cprecioso commented Apr 14, 2025

Uh oh!

infomiho commented Apr 14, 2025

Uh oh!

cprecioso commented Apr 16, 2025

Uh oh!

infomiho commented May 5, 2025

Uh oh!

cprecioso commented May 7, 2025

Uh oh!

infomiho left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

infomiho May 12, 2025

Choose a reason for hiding this comment

Uh oh!

cprecioso May 12, 2025

Choose a reason for hiding this comment

Uh oh!

cprecioso May 12, 2025

Choose a reason for hiding this comment

Uh oh!

infomiho May 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

infomiho May 13, 2025

Choose a reason for hiding this comment

Uh oh!

cprecioso May 13, 2025

Choose a reason for hiding this comment

Uh oh!

cprecioso commented May 12, 2025

Uh oh!

cprecioso commented May 13, 2025

Uh oh!

cprecioso commented Apr 8, 2025 •

edited

Loading

infomiho left a comment •

edited

Loading

infomiho May 13, 2025 •

edited

Loading