Automatically create JS versions of our TS code in the docs #2638

cprecioso · 2025-04-08T09:32:07Z

Description

Part of #2455

This PR allows to just write TS code directly, and automatically transform it and appear with the JS/TS selector.

It does so by adding two plugins to Docusaurus' MDX handling:

autoJSCode will find any code block with the auto-js meta, transform it with ts-blank-space, and remove the extra blank space with prettier.
Example transformation

Input:
```
 ```ts auto-js title="hello.ts"
 const foo = (n: number) => 8 as string
 ```
```
Equivalent output (actual output is done in AST):
```
 <Tabs groupId="js-ts">
 <TabItem value="js" label="JavaScript">
 
 ```js title="hello.js"
 const foo = (n) => 8
 ```
 
 </TabItem>
 <TabItem value="ts" label="TypeScript">
 
 ```ts title="hello.ts"
 const foo = (n: number) => 8 as string
 ```
 
 </TabItem>
 </Tabs>
```
Why use `ts-blank-space`?

Basically, it's the transformer that does the fewest modifications to the input code, which I think is important to keep the overall visual structure of the code in the examples. It also helps to not introduce TS-only constructs like enums that, when automatically converted, introduce distracting noise.

I also feel validated in its approach since it uses the TypeScript compiler directly to do its job, and not a custom parser that might get outdated. As well, the blank-space-replacing method is used by SWC and Node.js in their TypeScript support.
codeWithHole is made to deal with some of our examples, that just omit some syntactically needed parts of the code with .... In order for the TypeScript transformation to work correctly, this plugin allows us to use an identifier named hole that will be replaced by ..., while still being syntactically correct.
Example transformation

Input:
```
 ```ts with-hole
 const foo = (n: number) => {
 	return {
 		name: "bar",
 		hole
 	}
 }
 ```
```
Equivalent output (actual output is done in AST):
```
 ```ts
 const foo = (n: number) => {
 	return {
 		name: "bar",
 		/* ... */
 	}
 }
 ```
```

With these two plugins, we can transform our examples to be less complex to author while keeping the option to read our docs in JS or TS. If there are any examples that can't be automatically converted, or that shouldn't be converted, we still retain the option to do it manually as we've been doing until now.

I converted two examples so you can see the output, please check them out.

Martinsos

Very cool! Seems like an awesome thing to me, much less duplication! And we can still do it manually if we really need to (we can right? Would we then use those Tabs again?).

I will let others check the details of the PR and approve it, to make sure it works fully with how we do things currently (e.g. if you toggle to JS, does it remmeber the toggle? Mostly checking if we lost anything with not using our Tab system anymore). Aha but I see now that we actually do still use Tabs, they are produced by the plugin -> ok that sounds good then!

One th ing that would certainly be valuable is more documentation on this -> we should mention in docs README.md that we are using this mechanism because it is a bit magic, so having a central place with some discoverability where we describe it would be valuable.

Also, these remark/ files, we should provide some top level header comments in them, let's say one at the top of each file, to explain a bit the motivation and what are they here for, as they are a bit unexpected.
Also I didn't review the code in them, I will let the others do that.

Awesome all together!

web/docs/advanced/apis.md

cprecioso · 2025-04-08T10:28:07Z

we should mention in docs README.md

top level header comments

Yes! I wanted to get this out to get some comments on the approach but I wanted to do this, I'll get on it

Martinsos · 2025-04-08T10:48:51Z

we should mention in docs README.md

top level header comments

Yes! I wanted to get this out to get some comments on the approach but I wanted to do this, I'll get on it

No worries, I assumed so!
Advice: If you want the people to take a look at the PR, but not do a deep review -> make a draft PR. That basically communicates exactly that, that you are looking for comments, but you not done yet. While real PR means it is ready for proper reviewing.

cprecioso · 2025-04-09T13:30:15Z

Can I get a review? 🙏🏼

infomiho

I love this idea. It will make our lives much easier.

Testing

I tried with some more complex examples:

wasp/web/docs/advanced/web-sockets.md

Line 69 in 24920c0

here the tricky part are the TS specific comments (which stay in the JS version), we'd probably need to rewrite those code blocks not to use those comments and move that info somewhere outside of the code block

Another example:

wasp/web/docs/data-model/entities.md

Line 48 in 9977e27

We show Prisma code with the language switch for completeness sake - it toggles between different variants of the text below. So I guess, here we'll just use the tabs directly, you addressed it in the README, that's great 👍

In this example:

wasp/web/docs/data-model/operations/queries.md

Line 106 in 4c2a935

The JS version still has the import statement (without anything imported):

For me, the next step would be - try to convert all the code blocks to see where the current approach fails and try to address those cases. For some of the cases we'll need to modify the code (e.g. to exclude Typescript specific comments), that's okay for me.

web/README.md

web/src/remark/auto-js-code.js

web/src/remark/code-with-hole.js

cprecioso · 2025-04-14T13:06:47Z

@infomiho

try to convert all the code blocks

Are we sure converting 230 occurrences of this pattern is a good use of the team's time? 😅

I was planning on us leaving stuff as it is; writing new stuff with this plugin, and converting on the go whenever we touch a file.

infomiho · 2025-04-14T13:13:01Z

I think it's a valid use of team time since it will speed us the docs authoring process. It took me a couple of minutes per code block to do it, so it might be a few hours max IMHO.

What I looking to achieve: find the the edge cases for which the plugin might not produce 1:1 JS code. While this is still hot, while you have the context, I think it'll be good to figure out and write down the process/limits of the plugin.

Let's do it like this: invest one hour to convert as many examples as possible and try going for as many unique examples as possible 🙂

cprecioso · 2025-04-16T15:01:24Z

Turning to draft until #2658 is merged

infomiho · 2025-05-05T10:11:13Z

I suppose this PR can go forward since #2658 is merged 🎉

cprecioso · 2025-05-07T10:16:05Z

@infomiho ready for review

I tried updating the tutorial and some examples (https://github.com/wasp-lang/wasp/pull/2728/files), and the main drawbacks that I found were

We need to remember to add import type so they don't appear in JS
Edge cases:
- // highlight-next-line comment before a TS-only line will hang around and highlight the wrong line (happened twice)
- We're running prettier so in some cases where we're putting extra spaces around some lines to make them clearer and they get collapsed (only happened once)
- It doesn't replace file names' extensions in clarification comments (I'd say this problem is almost unique to the tutorial)

All in all, those are edge cases that might be out of scope for this, and they're not too impactful. We can always write the tabs manually for whatever edge case arises, so this is a "strictly better" improvement IMO. I'm comfortable merging.

infomiho

Great stuff @cprecioso, I think we are almost there. I'm glad you tested it out on more code blocks to get a better sense of caveats. I think the benefits definitely outweigh the caveats.

I've left some more comments, let me know when I can take another look.

web/src/remark/code-with-hole.ts

web/src/remark/auto-js-code.ts

web/README.md

infomiho · 2025-05-12T11:19:38Z

web/docs/advanced/apis.md

+```tsx title="src/pages/SomePage.tsx" auto-js with-hole
+import React, { useEffect } from "react";
+import { api } from "wasp/client/api";
+
+async function fetchCustomRoute() {
+  const res = await api.get("/foo/bar");
+  console.log(res.data);
+}
+
+export const Foo = () => {
+  useEffect(() => {
+    fetchCustomRoute();
+  }, []);
+
+  return <>{hole}</>;
+};
+```


One thing I noticed that when I try to auto format this code in VS Code, the code block doesn't get formatted and the code in the output is of course formatted.

Is there a way for us to configure Prettier to format the code blocks in the MDX files? I think it makes sense for code blocks in the source code to look as similar as possible to the rendered code blocks for easier searching later on.

If it's not possible to run Prettier on MDX files (this issue suggests so) should we make it a part of the docs authoring process to format the code before including it in the code blocks?

Yep, as you found out, Prettier is stuck in MDXv1 so I'd rather not run it on our MDXv3 files, it will mess up its delicate whitespace situation 😑

should we make it a part of the docs authoring process to format the code before including it in the code blocks?

You mean just adding that to the readme? I can do that.

I'd rather not have to run Prettier at all if we don't have it at the source, but it needs to run on the JS so it removes the blank space (I didn't find another type stripper that doesn't leave the whitespace), and then on the TS for consistency. 😔

Yep, I thought maybe about adding a note in the README to say, hey, format it so the source code and the rendered code look as close as possible for better searchablity.

Too bad about missing MDX 3 support, since Docusaurus moved, you'd expect it to be more demand for it.

Maybe one for the future, npx remark . --ext .mdx --output should probably format MDX 3 if used with the remark-mdx plugin. You probably know more about this than me, just throwing it out there.

I added it here BTW

wasp/web/README.md

Line 112 in beed6b9

- Run `prettier` on the code before pasting it in the document, as `auto-js` will enforce it.

- Use `unist-util-visit`, no need for `unist-util-visit-parents`. - Extract the common functions of the visitor check and the language assert to a shared file - For the async visitor, create a thunk inside of the visitor so we keep all the type narrowing instead of having to do it again

Co-authored-by: Mihovil Ilakovac <mihovil@ilakovac.com>

cprecioso · 2025-05-12T13:12:50Z

@infomiho ready

cprecioso · 2025-05-13T11:34:32Z

@infomiho if everything is ready let's merge ahead of the sprint review? 😁

infomiho

Naming comments, almost there

web/src/remark/util/code-blocks.ts

web/src/remark/auto-js-code.ts

web/README.md

infomiho

Great addition

Martinsos

I am writing remark plugin myself (coderef checker) and therefore got interested in this PR, so I started reviewing this PR while it was still unmerged, but it got merged in the meantime! Which is ok, I joined the party really late.

I however finished the review, so here it is. We can do initial discussion of items here, while you do the fixes in the new PR, and then we switch to that one after the first iteration of changes.

Martinsos · 2025-05-13T12:06:05Z

web/README.md

+To write docs, you can use Markdown or MDX. The docs are located in the `docs` directory.
+Remember to refer to the [Writing Guide](https://wasp.sh/docs/writingguide) for an explanation
+of how we like to write docs. You can check
+[Docusaurus' documentation](https://docusaurus.io/docs/2.x/markdown-features) to see which special
+Markdown features available (e.g. line highlighting).


You will notice that in the rest of the markdown we (me for sure, I think others also) don't stick to 80/100 lines as the max col, but take as much space as we need. That is because editors do good job of visually wrapping long lines and it makes writing text easier (you don't need to keep formatting it as you edit it).
Instead, what I do is usually start each sentences in a new line. That makes it easy to move from sentence to sentence also. I am not sure we ever communicated this at the team level though, but it might be happening organically, or maybe it is just me and I think everybody else is also doing it hah :D. But I think it has its benefits?

Suggested change

To write docs, you can use Markdown or MDX. The docs are located in the `docs` directory.

Remember to refer to the [Writing Guide](https://wasp.sh/docs/writingguide) for an explanation

of how we like to write docs. You can check

[Docusaurus' documentation](https://docusaurus.io/docs/2.x/markdown-features) to see which special

Markdown features available (e.g. line highlighting).

To write docs, you can use Markdown or MDX. The docs are located in the `docs` directory.

Remember to refer to the [Writing Guide](https://wasp.sh/docs/writingguide) for an explanation

of how we like to write docs.

You can check [Docusaurus' documentation](https://docusaurus.io/docs/2.x/markdown-features) to see which special Markdown features available (e.g. line highlighting).

Martinsos · 2025-05-13T12:07:54Z

web/README.md

@@ -23,6 +23,97 @@ $ npm start
 This command starts a local development server and opens up a browser window.
 Most changes are reflected live without having to restart the server.

+### Writing docs


There are quite a few headers already in this README about dealing with docs. They are also level 3 headers. It somehow feels like this specific header is the entry header for docs though. Would it make more sense to make this one level 2 header, call it ## Docs, and then put this at the start and the rest of the headers under it (which is already happening actually)?

Further question is if some of this content would be better suited in writingguide.md or even docs/README.md (which doesn't exist yet), but we don't have to deal with that now (but maybe in the future we will want to move most of this content into docs/README.md).

We have web/readme.md#writing docs, web/writing-docs.md and web/docs/writingguide.md. It seems like there's an opportunity for centralization here but tbh I don't know where to start from, the end result would be a monster doc

Martinsos · 2025-05-13T12:09:25Z

web/README.md

+#### Polyglot code blocks
+
+For examples that have a JavaScript and TypeScript version, add a `auto-js` meta attribute
+to the code block, like so:


Suggested change

to the code block, like so:

to the TypeScript code block, like so:

Martinsos · 2025-05-13T12:11:20Z

web/README.md

+```
+~~~
+
+And it will automatically generate a JS and TS version with a selector to switch between them:


Suggested change

And it will automatically generate a JS and TS version with a selector to switch between them:

And it will automatically generate a JS version based on the TS one + a selector to switch between them:

otherwise it sound slike it generates both js and ts version, which is not what we wanted to say, right?

Martinsos · 2025-05-13T12:13:15Z

web/README.md

+> You can create a language switcher manually as described in
+> [Docusaurus docs](https://docusaurus.io/docs/2.x/markdown-features/code-blocks#multi-language-support-code-blocks).
+
+If you need to omit some part of the code in a code example, you can use the `with-hole` meta attribute


This is under "polyglot code blocks", but it has nothing to do with polyglot code blocks, right? It would make more sense to put explanation about "hole" under separate header. Or maybe you can rename the header to "code blocks". Even then, each of these parts, auto-js and with-hole, might benefit from their own headers.

Martinsos · 2025-05-14T10:27:49Z

web/src/remark/auto-js-code.ts

+  return formatted.trim() // prettier adds a trailing newline, we remove it
+}
+
+function transformExt(inPath: string, fn: (ext: string) => string) {


Let's define better what this function does.

inPath -> confusing name, in is here because of internal naming (outPath), but me as a caller of this function, I don't know anything about it. I was for real trying to figure out what inPath means, till I looked into the implementation, I thought this is just some part of path, something inside path maybe. We could just name it filepath or even path, it would be clearer.

fn -> ok, name is short but relatively clear from the context what it does. Could be maybe transformFn, but fn also works. One thing I don't know, and it is also not defined, is if fn expects .js or js. I am guessing the second one, but I am not 100% sure, path.extname for example returns .ext, which to me doesn't make sense, but that is how it works. Would be good to have this specified in the comment of the function.

Would we benefit from return type? This is Haskell speaking from me, but I love return types on functions because they communicate clear intention from the developer. I know in TS that is not so clear, it makes sense sometimes to not have a return type, but I don't see anything bad about having one here, I can understand from just reading the function header what it does (does it replace path in place or does it return one? Ok, it returns one, and it is string), without having to read the implementation and guess what will TS infer.

Btw how does this function behave if:

I give it path that is a directory (so has no extension)
1.1. which doesn't end with /.
1.2. which does end with /.

Transformation function doesn't transform anything, ext remains the same -> are we ok with that, if that happens, or do we rather want to fail/throw?

Btw, I am not saying you need to handle all those case with the directory in the code, I wouldn't waste time on that if you don't need it and it is complex, but what you can do is make it clear in the comments what is your expectation from the path -> you expect it to have extension for example, and caller is to ensure that, otherwise it is UB (undefined behaviour). That is enough already, just saying that in the function header comment.

Btw, to explain why I am commenting on this stuff: I understand that this is local utility function, it's not used in any other places, so at the moment it serves its purpose well even with not all this stuff being defined, but:

At some point somebody might want to use it in some other file, maybe another plugin, or even just another place in this same file. Then, they have to guess what was the intended behaviour -> because there certainly was some intended behaviour, that served well the purpose so far. But it is not clear which of that behaviour was intentional, and which was not.

Similar, if somebody needs to modify the existing usage, they will have to guess what were the unwritten assumptions / expectations and think through those, instead of them being clear upfront.

Martinsos · 2025-05-14T10:32:35Z

web/src/remark/auto-js-code.ts

+  const inExt = path.extname(inPath)
+  const outExt = fn(inExt)
+  const outPath = inPath.slice(0, -inExt.length) + outExt
+  return outPath


Nitpick: in and out as nomenclature is a bit unusual. I guess it means in and out of function, but that is not something I usually see used in such way. Something like old and new might be a better fit, or even no prefix (just path) and new (newPath).

Some golfing, I find it more readable (less var names to reason about), but I get it if you don't, you don't have to do it:

Suggested change

const inExt = path.extname(inPath)

const outExt = fn(inExt)

const outPath = inPath.slice(0, -inExt.length) + outExt

return outPath

const ext = path.extname(path)

return path.slice(0, -ext.length) + transformExt(ext)

Martinsos · 2025-05-14T10:45:02Z

web/src/remark/code-with-hole.ts

@@ -0,0 +1,49 @@
+/*
+This file defines a plugin for the unified library that processes code blocks


Also has this interesting start with "plugin for the unified library" hah :D.

Martinsos · 2025-05-14T10:47:31Z

web/src/remark/code-with-hole.ts

+This file defines a plugin for the unified library that processes code blocks
+in Markdown documents. It looks for code blocks with a specific meta flag
+(`with-hole`) and replaces occurrences of the word "hole" in the code with
+a placeholder comment (`...`). This is useful for code examples where


I am curious, so does it replace it with a comment always, or sometimes just raw ...? Because I think I saw somewhere earler that in JS it is just ... while in TS it is /* ... */. Also the example here, in parenthesses, is just ..., so is that a comment or not hm. Ok yeah I see below now in the Example also that it is not in a comment.

Martinsos · 2025-05-14T10:54:48Z

web/src/remark/code-with-hole.ts

+    node.meta = node.meta.replace(META_FLAG_REGEX, '')
+
+    // Replace hole with ellipsis
+    node.value = node.value.replace(HOLE_IDENTIFIER_REGEX, HOLE_REPLACEMENT)


Oh wow this is very crude -> so we just take the whole code from the code block as a string and replace hole with a regex. I was expecting it does something more refined: go through the actual AST of the js/ts, and if it is a certain type of node (identifier, maybe field, ...), and its whole value equals hole, then we replace it.
Because this will trigger on stuff like `"fire in the hole!" (where hole is just a word in a string), or if it is a part of regex, or part of comment even.

But I guess getting that js/ts AST might be quite some work, is that the problem, and also putting it back together? remark doesn't go as far as to already provide the AST for the language in the code block for us, does it? And we think it is not likely that hole will appear in our code blocks in any any string/comment/regex/... ?

If that is all so, I get it, we shouldn't waste time on handling something that is likely never to happen, but should we then at least give it a longer and more specific name, so reduce the change of false positives? Instead of hole, let's go for code-hole or hole-here or $code-hole or $hole-to-be-replaced or something like that. Oh, or CODE_HOLE, that is also nicely visible in the code example then.

cprecioso requested review from Martinsos and infomiho April 8, 2025 09:32

cprecioso changed the title ~~Automatically create JS versions of our TS code~~ Automatically create JS versions of our TS code in the docs Apr 8, 2025

cprecioso linked an issue Apr 8, 2025 that may be closed by this pull request

Update docs for userSignupFields to be consistent between JS and TS code examples #2455

Closed

cprecioso removed a link to an issue Apr 8, 2025

Update docs for userSignupFields to be consistent between JS and TS code examples #2455

Closed

Martinsos reviewed Apr 8, 2025

View reviewed changes

web/docs/advanced/apis.md Outdated Show resolved Hide resolved

web/docs/advanced/apis.md Outdated Show resolved Hide resolved

cprecioso added 4 commits April 8, 2025 12:32

Automatically create JS versions of our TS code

3e59636

Apply two examples

e0ee478

Add file comments

793b50f

Refactor some code to make it more understandable

4002350

Add readme docs

3bd0591

cprecioso force-pushed the auto-js-docs branch from 50e0e81 to 3bd0591 Compare April 8, 2025 11:00

cprecioso self-assigned this Apr 8, 2025

infomiho requested changes Apr 11, 2025

View reviewed changes

cprecioso added 7 commits April 14, 2025 14:24

Comments in readme

fd19f6a

Fix admonition

b255f6b

Typo

56112f3

More fixes

b954e89

Simple changes

49d7321

Find the correct prettier confirm

1f68723

Extract functions

f41ef1e

cprecioso marked this pull request as draft April 16, 2025 15:01

cprecioso changed the title ~~Automatically create JS versions of our TS code in the docs~~ WIP: Automatically create JS versions of our TS code in the docs Apr 25, 2025

cprecioso added 5 commits May 7, 2025 09:56

Forgot hoel

027943d

Fix component

3f3dfe3

Update plugins to ts

0fafea8

Update to MDX v3

2bc974e

Correctly parse JSX

c9d54e2

cprecioso marked this pull request as ready for review May 7, 2025 10:41

cprecioso changed the title ~~WIP: Automatically create JS versions of our TS code in the docs~~ Automatically create JS versions of our TS code in the docs May 7, 2025

infomiho reviewed May 12, 2025

View reviewed changes

cprecioso and others added 8 commits May 12, 2025 13:35

Add vfile reporting

2abdc9b

Apply suggestion

d968565

Co-authored-by: Mihovil Ilakovac <mihovil@ilakovac.com>

Apply suggestion

aeec5a8

Apply comment

36eb43a

Co-authored-by: Mihovil Ilakovac <mihovil@ilakovac.com>

Apply suggestion

f7f3d69

Extract prettier config

dbed0a0

Add caveats to readme

beed6b9

Trim final newline

0154adc

cprecioso mentioned this pull request May 13, 2025

Update examples with auto-js #2728

Merged

infomiho reviewed May 13, 2025

View reviewed changes

web/src/remark/util/code-blocks.ts Outdated Show resolved Hide resolved

web/src/remark/auto-js-code.ts Outdated Show resolved Hide resolved

web/README.md Show resolved Hide resolved

cprecioso added 2 commits May 13, 2025 13:58

Rename fn

7d4bd83

Better explain the transforms

3d6253d

infomiho approved these changes May 13, 2025

View reviewed changes

cprecioso merged commit 5420929 into main May 13, 2025

cprecioso deleted the auto-js-docs branch May 13, 2025 12:07

Martinsos reviewed May 14, 2025

View reviewed changes

cprecioso restored the auto-js-docs branch May 15, 2025 13:35

cprecioso deleted the auto-js-docs branch May 15, 2025 13:37

	to the code block, like so:
	to the TypeScript code block, like so:

	And it will automatically generate a JS and TS version with a selector to switch between them:
	And it will automatically generate a JS version based on the TS one + a selector to switch between them:

		@@ -0,0 +1,49 @@
		/*
		This file defines a plugin for the unified library that processes code blocks

Automatically create JS versions of our TS code in the docs #2638

Automatically create JS versions of our TS code in the docs #2638

Conversation

cprecioso commented Apr 8, 2025 • edited Loading

Description

Martinsos left a comment

Choose a reason for hiding this comment

cprecioso commented Apr 8, 2025

Martinsos commented Apr 8, 2025

cprecioso commented Apr 9, 2025

infomiho left a comment • edited Loading

Choose a reason for hiding this comment

Testing

cprecioso commented Apr 14, 2025

infomiho commented Apr 14, 2025

cprecioso commented Apr 16, 2025

infomiho commented May 5, 2025

cprecioso commented May 7, 2025

infomiho left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

infomiho May 13, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cprecioso commented May 12, 2025

cprecioso commented May 13, 2025

infomiho left a comment

Choose a reason for hiding this comment

infomiho left a comment

Choose a reason for hiding this comment

Martinsos left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cprecioso May 15, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Martinsos May 14, 2025 • edited Loading

Choose a reason for hiding this comment

cprecioso commented Apr 8, 2025 •

edited

Loading

infomiho left a comment •

edited

Loading

infomiho May 13, 2025 •

edited

Loading

cprecioso May 15, 2025 •

edited

Loading

Martinsos May 14, 2025 •

edited

Loading