-
Notifications
You must be signed in to change notification settings - Fork 1.1k
Redocly Migration Scripting and Pre‐Migration Moves
For the migration to the Redocly toolchain, there are several cases where Redocly needs things to be in a different format or syntax than they currently are.
There are three strategies we can use to make these changes, on a case by case basis. In order from most to least preferable, they are:
-
Adapt the current site to a format that's compatible with both tools.
- "Remove" Variant: Remove any Dactyl-specific syntax and switch to something plainer that both can handle.
- "Shim" Variant: Implement a quick Dactyl plugin site so that it's compatible or mostly-compatible.
-
Script a tool that can quickly apply sweeping changes from the Dactyl format to the Redocly format starting from basically any commit. Instead of committing the changes to the Redocly branch and copying work over, we commit the script to the main branch.
- "Extend" Variant: We implement something in Redocly that is a close analogue of what Dactyl supports, and the script does a find-and-replace from old→new syntax.
- Manual changes on a Redocly migration branch, which would have to be kept up-to-date and in-sync over a potentially extended period of time. In many cases we can assume that scripted / shimmed changes above may cover 90% of the work and manual changes can be applied as a last step during a code freeze just before the final migration.
Rather than forking once and having to keep the branches in-sync for a long time, it should be less friction to make a series of incremental changes to make the master branch more "Redocly-ready" while still maintaining compatibility with the current Dactyl site. Some of these changes may involve a large changeset, but by keeping each one constrained to one category of change and having the PR open only a couple days at most should reduce the impact and migration pain of any individual step.
However, we need to have a working branch for Redocly work in the meantime. We can use the redocly-migration branch for this purpose. During development, we can do some widespread conversions, especially by script, and commit them to the redocly-migration branch with a commit message that starts with [DROP]. Then, we can periodically rebase redocly-migration onto the latest master branch, and when we do we can remove and re-create the [DROP] commits by applying the same steps, which should automatically adapt to the latest changes from the master branch. To reduce the risk of merge conflicts, we should avoid doing manual touchups to files extensively modified by [DROP] commits. (This mostly means not editing the content md files by hand until we're ready for a code freeze.)
When it's time for the code freeze, one of the first things we'll do is run the conversion script(s) with a version of the commits that isn't marked [DROP]. We can then add any manual touchups that would otherwise have risked causing merge conflicts.
| Approach: | Adapt |
|---|---|
| Status | ✅ Done (except for snippets) |
| Related PRs/issues | #2206 |
The Dactyl-based site currently uses a convention where translated pages are {file}.ja.md in the same dir as the English versions. Redocly needs these to be under @i18n/ja/ in a parallel folder structure and named {file}.md.
The key is that Dactyl doesn't actually need the md files to follow any specific filename/folder structure. I think we can do one update to put them under an @i18n/ja folder without any breaking changes to how Dactyl handles them; this would only be a small, one-time disturbance to translation efforts and then they could resume and the files would already be in the right place for when we're ready to flip the switch to Redocly.
| Approach: | Adapt & Shim |
|---|---|
| Status | ✅ Scripted (except for snippets & Japanese) |
| Related commits | a5640c6 |
The Dactyl site uses relative links in the form {file}.html in most places. Redocly links can either be a relative path to the source file or an absolute path to the output path. Absolute links are convenient if you know what the final URL for a page will be and they're often shorter and don't need to change when you move the file you're linking from. Relative links work when viewing a page on GitHub, and they sometimes don't need to change, for example if files that link each other are moved as a group such that their relative hierarchy doesn't change.
I already have a Dactyl plugin that finds links in the current Dactyl pages (including ones defined by the includes) and converts them into relative links in the format Redocly expects. It uses a "cheat-sheet" file of reusable links and relies on Dactyl's knowledge of file hierarchy to work, so we need to run it while the Dactyl build is still functional.
For now, the plan is to run the plugin on the Redocly branch, and commit the results with [DROP] in the commit message. As we add and move files on the master branch, the script should automatically handle those correctly if we drop the link-conversion commit and re-run the script to re-create it. However, since this involves massive changes to the .md files, we should avoid manually making changes or moving those files around until the code freeze.
| Approach: | TBD |
|---|---|
| Status | Waiting for more information from Redocly on if/when they can update the way partials work to support this use case. |
The Dactyl site currently defines a bunch of reusable common links and uses {% include '...' %} to pull those definitions into a variety of files.
I've exported the list of reusable links as a yaml file and my link conversion script already uses that to convert links to Redocly-compatible relative links without relying on an include. So as part of the migration of links into Redocly format, we can use this and remove the includes.
It does mean we lose the convenience of being able to write [server_info method][] or [AccountSet transaction][] and have those automatically link to the right place, though.
Redocly has been open to the possibility of supporting a set of "reusable" links defined somehow, like this, but so far we have not found a way to do it. (Using partials, like how we do in Dactyl, is not viable with how the Markdoc engine works.) We are waiting for updates from Redocly on how this might be possible to accomplish in the new portal; the naive solutions (like using partials) don't work. If we don't get anything by the time of the code freeze, we'll proceed with using the link conversion script to change everything to relative links.
| Approach: | Adapt & Script |
|---|---|
| Status | Partially migrated, scripting incomplete |
| Related PRs / Issues | #2215, #2238 |
The Dactyl site has a bunch of auto-generated landing pages that contain just a list of children with blurbs for each; these are defined in the config file but don't generally have any representation in the filesystem. Redocly recently added some tooling so you can do something similar with a plugin, but it still needs the files to have an index.md file (or something like it) in the filesystem where the landing goes.
We've already moved from having index pages that only exist as stanzas in the Dactyl config to having index.md pages with frontmatter. The conversion script to add the necessary syntax to use a Redocly plugin ({% child-pages %} component) is mostly done but needs some adjustment. (Index pages need some extra frontmatter to make the plugin work correctly.)
| Approach: | Manual? |
|---|---|
| Status | Waiting for code freeze unless we can successfully script it sooner |
The Redocly bugs relating to snippets are now fixed.
However, links in snippets are relative to the snippet location, not to where the files are being built in the end. The link conversion script isn't ready to handle this exception, but there aren't many links inside of snippets.
Translated snippets also need to be moved and renamed, and references to them updated.
These are both things that might be possible to script out, but are probably small enough to just do by hand during code freeze.
| Approach: | Script |
|---|---|
| Status | ✅ Scripted |
This is one of a few cases where we use {{ variable }} syntax in the current docs. Redocly has {% $variable %} syntax but which variables it provides in which contexts doesn't quite line up with Dactyl's.
Now that Redocly has added the auto-detected page title to the $frontmatter.seo.title variable and fixed the bug with using variables in a code block, we can do a direct conversion between syntax in many contexts. One major exception is when the variable is used in inline code syntax, like `{{currentpage.name}}`, which we use extensively in reference pages; Redocly's Markdoc parser does not support variables in that context. For these, I've created a custom component, {% code-page-name %/} which produces the desired output, and configured the conversion script to use it in the appropriate places.
We don't have to change the existing pages, although many instances of {{currentpage.name}} in the existing docs are extraneous and we could just remove them. (For example, the header "AccountSet Flags" could just be "Flags" because it's already on the AccountSet page.) This would change/break some links, so this might be best saved for another time.
| Approach: | ✅ Scripted |
|---|---|
| Status | Waiting to test |
| Related PRs/issues | #2242 |
Another case where we use {{ variable }} syntax is the {{ include_svg(...) }} macro, which is helpful for theme-aware diagrams. Redocly has now implemented a component, {% inline-svg ... %} that does something similar, but we encountered some bugs with the first version of it. Supposedly the bugs have been fixed in Redocly 0.65.6.
There are also some things {{include_svg(...)}} does that {% inline-svg ... /%} doesn't:
- link to the original image
- add a caption.
Either we need to do a custom component that is a superset of this, or we
| Approach: | Script & Extend |
|---|---|
| Status | ✅ Scripted |
| Related issues / PRs | #2259 |
Redocly's code-snippet Markdoc component now does what we need (as of Redocly version 0.62.0). The syntax is slightly different, but we have a script that does a mass find-and-replace of the syntax.
| Approach: | Adapt & Remove |
|---|---|
| Status | ✅ Removed |
| Related issues/PRs | #2267 |
There were a few cases where our snippets were supposed to display something slightly different depending on where they're being included, using syntax like the following example:
{% if currentpage.md == "tutorials/manage-the-rippled-server/installation/install-rippled-on-ubuntu.md" or
currentpage.md == "tutorials/manage-the-rippled-server/installation/install-rippled-on-centos-rhel-with-yum" %}
sudo systemctl restart rippled.service
{% elif currentpage.md == "tutorials/manage-the-rippled-server/installation/build-run-rippled-ubuntu.md" or
currentpage.md == "tutorials/manage-the-rippled-server/installation/build-run-rippled-macos.md" %}
* Use Ctrl-C to stop `rippled`, then start it again:
./rippled
{% endif %}In a couple cases these weren't maintained when pages were moved/renamed so they silently stopped working as intended, and frequently weren't that useful anyway, so PR #2259 removed all instances of this syntax from the master branch. These changes should be fixed on the redocly-migration branch next time we rebase it.
Moving forward, if we need snippets to adapt based on the context they're used in, Redocly supports variables passed in as attributes of the {% partial /%} element's syntax.
| Approach: | Adapt & Script |
|---|---|
| Status | ✅ Partially scripted; Waiting for updates from Redocly for advanced features |
| Related issues/PRs | #2259 |
Two variables we use commonly are github_forkurl and github_branch which are set to https://github.com/XRPLF/xrpl-dev-portal and master respectively, but can be overwritten with commandline vars, which lets preview builds refer to in-repo files that come from the fork & branch of the pull request, but automatically switch those links to the master branch when merged.
Example syntax:
For an example of such a file, see the provided [`ledger-file.json`]({{target.github_forkurl}}/blob/{{target.github_branch}}/content/_api-examples/rippled-cli/ledger-file.json).Two problems came up with converting these vars to Redocly syntax:
- Redocly's Markdoc engine doesn't support using variables in the href part of a link; they simply don't get parsed. To work around this, I created a custom component which can be used like this:
{% repo-link path="content/some/file.md" /%}and added a rule to the conversion script to convert instances of the old syntax to the new one. This is more concise than the old way but less flexible (for example, it wouldn't support the "edit" link that the Client Libraries page used to have). - Redocly does not yet support automatically detecting the current GitHub fork & branch from a PR and passing it into the vars, but Roman has suggested that they would be open to adding that feature eventually. For now, the component uses variables from a
.envfile that's committed to the repo.
| Approach: | Script |
|---|---|
| Status | ✅ Scripted |
| Related issues/PRs | #2259 |
These variables are used in a couple places. The idea was any place where we want to tell people what the current reserve values are, we can use these variables and then if/when the reserves later change we can update them all in one place.
The conversion script converts instances of these to Redocly environment vars syntax, using a .env/ file. For example, {% $env.PUBLIC_GITHUB_FORK %}.
| Approach: | Script, Manual touchups |
|---|---|
| Status | In progress (split between Ripple DGE & Ripple Marketing responsibilities) |
| Related issues/PRs | #2265, #2262, #2254, #2243, ... |
Marketing pages and Dev Tools have custom templates in Jinja format. Redocly needs these to be React components instead. Roman has created a script to do the conversions, which is kind of hacky and not thoroughly tested. This can be the starting point for manual edits to get the templates into shape.
| Approach: | Manual |
|---|---|
| Status | To-do; waiting on bigger changes like links |
We have partial proof-of-concept work for interactive tutorials, and Redocly has a new window.onRouteChange(callback) event we can use instead of $(document).ready(callback) for things that need to happen on page load.
Some of this stuff might be scriptable but it's still likely to involve a lot of manual work or at least manual testing.
| Approach: | Adapt, TBD |
|---|---|
| Status | Still need to investigate further |
The current site has two static folders—one for assets referenced by the templates (assets/ from repo top) and one for diagrams referenced by the Markdown contents (img/ from repo top).
Redocly needs these to be located in a folder named static/ from the top of the build folder (content/ under the current preview).
We can do a one-time move and update the Dactyl config and templates to use the path static/ instead of assets/ for the template assets.
TBD if Redocly also needs changes to how diagrams/etc are referenced from within the Markdown, and how this works with translated files.
| Approach: | Manual |
|---|---|
| Status | Done? |
We have some .json files that intentionally contain invalid syntax such as ... (trimmed) ... where long, irrelevant portions have been cut out. Redocly tries to parse these with the YAML parser and reports errors such as the following:
missed comma between flow collection entries (38:9)
35 | }
36 | },
37 | ... (trimmed for length) ...
38 | {
--------------^
39 | "object_id": "F0B9A528CE25FE7 ...
40 | "object": {
YAMLException: missed comma between flow collection entries (38:9)
35 | }
36 | },
37 | ... (trimmed for length) ...
38 | {
--------------^
39 | "object_id": "F0B9A528CE25FE7 ...
40 | "object": {
at generateError (file:///another/devel/xrpl-dev-portal/node_modules/js-yaml/dist/js-yaml.mjs:1273:10)
at throwError (file:///another/devel/xrpl-dev-portal/node_modules/js-yaml/dist/js-yaml.mjs:1277:9)
at readFlowCollection (file:///another/devel/xrpl-dev-portal/node_modules/js-yaml/dist/js-yaml.mjs:1848:7)
at composeNode (file:///another/devel/xrpl-dev-portal/node_modules/js-yaml/dist/js-yaml.mjs:2532:11)
at readFlowCollection (file:///another/devel/xrpl-dev-portal/node_modules/js-yaml/dist/js-yaml.mjs:1881:7)
at composeNode (file:///another/devel/xrpl-dev-portal/node_modules/js-yaml/dist/js-yaml.mjs:2532:11)
at readFlowCollection (file:///another/devel/xrpl-dev-portal/node_modules/js-yaml/dist/js-yaml.mjs:1881:7)
at composeNode (file:///another/devel/xrpl-dev-portal/node_modules/js-yaml/dist/js-yaml.mjs:2532:11)
at readFlowCollection (file:///another/devel/xrpl-dev-portal/node_modules/js-yaml/dist/js-yaml.mjs:1881:7)
at composeNode (file:///another/devel/xrpl-dev-portal/node_modules/js-yaml/dist/js-yaml.mjs:2532:11)
missed comma between flow collection entries (9:19)
6 | "ledger": {
7 | ... (trimmed) ...
8 |
9 | "close_time": 560302643,
------------------------^
10 | "close_time_human": "2017-Oct-02 23:37:23",
11 | "close_time_resolution": 10,
YAMLException: missed comma between flow collection entries (9:19)
6 | "ledger": {
7 | ... (trimmed) ...
8 |
9 | "close_time": 560302643,
------------------------^
10 | "close_time_human": "2017-Oct-02 23:37:23",
11 | "close_time_resolution": 10,
at generateError (file:///another/devel/xrpl-dev-portal/node_modules/js-yaml/dist/js-yaml.mjs:1273:10)
at throwError (file:///another/devel/xrpl-dev-portal/node_modules/js-yaml/dist/js-yaml.mjs:1277:9)
at readFlowCollection (file:///another/devel/xrpl-dev-portal/node_modules/js-yaml/dist/js-yaml.mjs:1848:7)
at composeNode (file:///another/devel/xrpl-dev-portal/node_modules/js-yaml/dist/js-yaml.mjs:2532:11)
at readFlowCollection (file:///another/devel/xrpl-dev-portal/node_modules/js-yaml/dist/js-yaml.mjs:1881:7)
at composeNode (file:///another/devel/xrpl-dev-portal/node_modules/js-yaml/dist/js-yaml.mjs:2532:11)
at readFlowCollection (file:///another/devel/xrpl-dev-portal/node_modules/js-yaml/dist/js-yaml.mjs:1881:7)
at composeNode (file:///another/devel/xrpl-dev-portal/node_modules/js-yaml/dist/js-yaml.mjs:2532:11)
at readBlockMapping (file:///another/devel/xrpl-dev-portal/node_modules/js-yaml/dist/js-yaml.mjs:2194:12)
at composeNode (file:///another/devel/xrpl-dev-portal/node_modules/js-yaml/dist/js-yaml.mjs:2531:12)
Expected "/", "[", "{", boolean, class, end of input, id, identifier, null, number, string, variable, or whitespace but "'" found.
Roman promised a fix "in the next release" on Oct 2. It was not in the next release. Unsure if the fix has been released yet. For now, the redocly-migration branch has a commit that "comments out" the invalid syntax in a couple cases.
| Approach: | TBD |
|---|---|
| Status | To-do |
Markdoc (used by Redocly) disables indented code blocks for some reason, which means that many of our code blocks inside lists end up displaying as plain text instead. It looks like this cannot be easily reconfigured on Redocly's end. However, Python-Markdown (used by Dactyl) does not support indented code fences, so we can't make the conversion in a way that supports both.
Scripting the conversion is likely to be tricky because it involves searching for whitespace, and contextual awareness of how many spaces are needed to make something a code tab. (Inside a list, you need more spaces because indenting a little only makes a paragraph that continues inside the list item.)
| Approach: | Script & Adapt |
|---|---|
| Status | ✅ Scripted (mostly) |
| Related issues / PRs | #2259 |
Dactyl uses a plugin to offer multiple code blocks as tabs. Redocly has a built-in {% tabs %} component that can be used with code blocks or additional content. The conversion script converts from one type to the other (including in occasional cases where the code tabs are indented inside a list).
Not tested/not done: the current website has a script that changes all instances of code tabs on a page when you change one tab, so for example if you tab to see a Python code sample, the later samples on the same page will already be showing Python by the time you scroll to them. I don't think this works with Redocly's tabs out of the box, but it can probably be adapted for similar use.
| Approach: | Script & Adapt |
|---|---|
| Status | ✅ Scripted |
| Related issues/PRs | #2259 |
Redocly's syntax for "admonitions" is a little different than Dactyl's for callouts, so we need to convert it over as part of the migration script.