-
Notifications
You must be signed in to change notification settings - Fork 1.1k
Redocly Migration Scripting and Pre‐Migration Moves
For the migration to the Redocly toolchain, there are several cases where Redocly needs things to be in a different format or syntax than they currently are.
There are three strategies we can use to make these changes, on a case by case basis. In order from most to least preferable, they are:
-
Adapt the current site to a format that's compatible with both tools.
- "Remove" Variant: Remove any Dactyl-specific syntax and switch to something plainer that both can handle.
- "Shim" Variant: Implement a quick Dactyl plugin site so that it's compatible or mostly-compatible.
-
Script a tool that can quickly apply sweeping changes from the Dactyl format to the Redocly format starting from basically any commit. Instead of committing the changes to the Redocly branch and copying work over, we commit the script to the main branch.
- "Extend" Variant: We implement something in Redocly that is a close analogue of what Dactyl supports, and the script does a find-and-replace from old→new syntax.
- Manual changes on a Redocly migration branch, which would have to be kept up-to-date and in-sync over a potentially extended period of time. In many cases we can assume that scripted / shimmed changes above may cover 90% of the work and manual changes can be applied as a last step during a code freeze just before the final migration.
Rather than forking once and having to keep the branches in-sync for a long time, it should be less friction to make a series of incremental changes to make the master branch more "Redocly-ready" while still maintaining compatibility with the current Dactyl site. Some of these changes may involve a large changeset, but by keeping each one constrained to one category of change and having the PR open only a couple days at most should reduce the impact and migration pain of any individual step.
| Approach: | Adapt |
|---|---|
| Status | ✅ Done (except for snippets) |
| Related PRs/issues | #2206 |
The Dactyl-based site currently uses a convention where translated pages are {file}.ja.md in the same dir as the English versions. Redocly needs these to be under @i18n/ja/ in a parallel folder structure and named {file}.md.
The key is that Dactyl doesn't actually need the md files to follow any specific filename/folder structure. I think we can do one update to put them under an @i18n/ja folder without any breaking changes to how Dactyl handles them; this would only be a small, one-time disturbance to translation efforts and then they could resume and the files would already be in the right place for when we're ready to flip the switch to Redocly.
| Approach: | Adapt & Shim |
|---|---|
| Status | Waiting for more info on reusable links (below). Some temporary work in place on redocly-migration branch. |
The Dactyl site uses relative links in the form {file}.html in most places. Redocly links can either be a relative path to the source file or an absolute path to the output path.
I already have a Dactyl plugin that finds links in the current Dactyl pages (including ones defined by the includes) and converts them into relative links in the format Redocly expects; I believe it would be pretty straightforward to reverse the logic, so we could change the links once to a format that's compatible with Redocly, and just have a Dactyl plugin subbing those links back to their existing values (on-build) until we're ready to move. The only exception would be any links from snippets, but that's a much smaller set of things to keep in sync.
Relative links are pretty inconvenient to use in many cases, though, so absolute links would be preferable. Absolute links come with their own challenges; we need to settle on the new / "pretty" URLs; those should, in theory, match the information architecture and navigation.
| Approach: | Adapt & Remove? |
|---|---|
| Status | Waiting for more information from Redocly on if/when they can update the way partials work to support this use case. |
The Dactyl site currently defines a bunch of reusable common links and uses {% include '...' %} to pull those definitions into a variety of files.
I've exported the list of reusable links as a yaml file and my link conversion script already uses that to convert links to the appropriate Redocly-compatible syntax without relying on an include. So as part of the migration of links into Redocly format, we can use this and remove the includes.
It does mean we lose the convenience of being able to write [server_info method][] or [AccountSet transaction][] and have those automatically link to the right place, though.
It would be nice to figure out how to keep some of this functionality if it's at all possible. But so far I haven't found any approaches (using partials for example) that seem to work.
| Approach: | Adapt |
|---|---|
| Related PRs / Issues | #2215, #2238 |
The Dactyl site has a bunch of auto-generated landing pages that contain just a list of children with blurbs for each; these are defined in the config file but don't generally have any representation in the filesystem. Redocly recently added some tooling to do something similar, but it still needs the files to have an index.md file (or something like it) in the filesystem where the landing goes.
We can create (largely empty) index.md pages for the current Dactyl site, and move the properties from the dacyl-config.yml to those pages' frontmatter, which would still generate the exact same site at build time. This move can be done once via a script. Having these index.md pages in place would help with making links that will port cleanly to Redocly, and the final conversion to Redocly would involve a smaller/simpler one-time change to each of these files (which can also be scripted).
Setting up a minimal index.md page using a somewhat generic template doesn't stop us from eventually replacing some of these pages with more customized landing pages.
| Approach: | TBD |
|---|---|
| Status | Waiting for updates from Redocly |
Redocly currently reads snippet files (even if the folder and/or file name starts with _) and throws errors if they contain broken links. Also, any link paths in these snippets are relative to the snippet location, not to where the files are being built in the end.
In our meeting on 2023-10-24, Roman said they had finally finished an improvement to internal handling of Markdoc that was blocking other things, and the ability to define a partials folder is the next thing on their plate (once again "this week"). We should not need to rename all the snippets (although some changes that apply to other pages may also apply to snippets)
| Approach: | Wait & adapt |
|---|---|
| Status | Waiting for updates from Redocly |
This is one of a few cases where we use {{ variable }} syntax in the current docs. Redocly has {% $variable %} syntax but which variables it provides in which contexts doesn't quite line up with Dactyl's.
Many instances of {{currentpage.name}} in the existing docs are extraneous and we can actually just remove them. (For example, the header "AccountSet Flags" could just be "Flags" because it's already on the AccountSet page.) This will change/break some links, but we can add <a id='old-anchor-name'></a> tags to keep the old URLs functional if we need to.
Roman also said that Redocly can add the current page's auto-detected title into the $frontmatter.title variable so we can access it similarly, but that isn't implemented yet (as of 2023-10-24).
| Approach: | TBD. Script & Extend / Adapt & Remove |
|---|---|
| Status | investigating options |
Another case where we use {{ variable }} syntax is the {{ include_svg(...) }} macro. We can probably implement a Component that does something very similar and get the benefits of the current approach (theme-aware diagrams).
| Approach: | Script & Extend |
|---|---|
| Status | Waiting for updates from Redocly |
Redocly has already implemented a Component that does most of this, but currently it only does line numbers, not "starts_with" and "end_before". Roman promised he would add those, so then we could make a simple find-and-replace script to convert from the existing syntax to the Redocly equivalent. As of 2023-10-24, this is still on the backlog.
| Approach: | Adapt & Remove |
|---|---|
| Status | To-do |
There are a few cases where our snippets display something slightly different depending on where they're being included, using syntax like the following example:
{% if currentpage.md == "tutorials/manage-the-rippled-server/installation/install-rippled-on-ubuntu.md" or
currentpage.md == "tutorials/manage-the-rippled-server/installation/install-rippled-on-centos-rhel-with-yum" %}
sudo systemctl restart rippled.service
{% elif currentpage.md == "tutorials/manage-the-rippled-server/installation/build-run-rippled-ubuntu.md" or
currentpage.md == "tutorials/manage-the-rippled-server/installation/build-run-rippled-macos.md" %}
* Use Ctrl-C to stop `rippled`, then start it again:
./rippled
{% endif %}In a couple cases these haven't been maintained as pages have been moved/renamed so they're already not working as intended, and in general we're not getting all that much out of this syntax. We should just switch all the instances to something that will be "close enough" in all cases and remove the special syntax entirely.
Also, in the future, Redocly's partial syntax supports variables passed in as parameters of the {% partial /%} syntax.
| Approach: | TBD |
|---|---|
| Status | Waiting for updates from Redocly |
Two variables we use commonly are github_forkurl and github_branch which are set to https://github.com/XRPLF/xrpl-dev-portal and master respectively, but can be overwritten with commandline vars, which lets preview builds refer to in-repo files that come from the fork & branch of the pull request, but automatically switch those links to the master branch when merged.
Example syntax:
For an example of such a file, see the provided [`ledger-file.json`]({{target.github_forkurl}}/blob/{{target.github_branch}}/content/_api-examples/rippled-cli/ledger-file.json).Redocly does not yet support automatically detecting the current GitHub fork & branch from a PR and passing it into the vars, but Roman said it was possible, and I (@mDuo13) shared info about how we currently get that data.
| Approach: | Manual |
|---|---|
| Status | To-do |
These variables are used in a couple places. The idea was any place where we want to tell people what the current reserve values are, we can use these variables and then if/when the reserves later change we can update them all in one place.
This can be done by adding vars to the content/.env/ file under Redocly and accessing them using Markdoc variable syntax. The vars have to have names starting with PUBLIC_ though.
| Approach: | Script, Manual touchups |
|---|---|
| Status | In progress (split between Ripple DGE & Ripple Marketing responsibilities) |
Marketing pages and Dev Tools have custom templates in Jinja format. Redocly needs these to be React components instead. Roman has created a script to do the conversions, which is kind of hacky and not thoroughly tested. This can be the starting point for manual edits to get the templates into shape.
| Approach: | Manual |
|---|---|
| Status | To-do; waiting on bigger changes like links |
We have partial proof-of-concept work for interactive tutorials, and Redocly has a new window.onRouteChange(callback) event we can use instead of $(document).ready(callback) for things that need to happen on page load.
| Approach: | Adapt, TBD |
|---|---|
| Status | Still need to investigate further |
The current site has two static folders—one for assets referenced by the templates (assets/ from repo top) and one for diagrams referenced by the Markdown contents (img/ from repo top).
Redocly needs these to be located in a folder named static/ from the top of the build folder (content/ under the current preview).
We can do a one-time move and update the Dactyl config and templates to use the path static/ instead of assets/ for the template assets.
TBD if Redocly also needs changes to how diagrams/etc are referenced from within the Markdown.
| Approach: | No action |
|---|---|
| Status | Waiting for fix from Redocly |
We have some .json files that intentionally contain invalid syntax such as ... (trimmed) ... where long, irrelevant portions have been cut out. Redocly tries to parse these with the YAML parser and reports errors such as the following:
missed comma between flow collection entries (38:9)
35 | }
36 | },
37 | ... (trimmed for length) ...
38 | {
--------------^
39 | "object_id": "F0B9A528CE25FE7 ...
40 | "object": {
YAMLException: missed comma between flow collection entries (38:9)
35 | }
36 | },
37 | ... (trimmed for length) ...
38 | {
--------------^
39 | "object_id": "F0B9A528CE25FE7 ...
40 | "object": {
at generateError (file:///another/devel/xrpl-dev-portal/node_modules/js-yaml/dist/js-yaml.mjs:1273:10)
at throwError (file:///another/devel/xrpl-dev-portal/node_modules/js-yaml/dist/js-yaml.mjs:1277:9)
at readFlowCollection (file:///another/devel/xrpl-dev-portal/node_modules/js-yaml/dist/js-yaml.mjs:1848:7)
at composeNode (file:///another/devel/xrpl-dev-portal/node_modules/js-yaml/dist/js-yaml.mjs:2532:11)
at readFlowCollection (file:///another/devel/xrpl-dev-portal/node_modules/js-yaml/dist/js-yaml.mjs:1881:7)
at composeNode (file:///another/devel/xrpl-dev-portal/node_modules/js-yaml/dist/js-yaml.mjs:2532:11)
at readFlowCollection (file:///another/devel/xrpl-dev-portal/node_modules/js-yaml/dist/js-yaml.mjs:1881:7)
at composeNode (file:///another/devel/xrpl-dev-portal/node_modules/js-yaml/dist/js-yaml.mjs:2532:11)
at readFlowCollection (file:///another/devel/xrpl-dev-portal/node_modules/js-yaml/dist/js-yaml.mjs:1881:7)
at composeNode (file:///another/devel/xrpl-dev-portal/node_modules/js-yaml/dist/js-yaml.mjs:2532:11)
missed comma between flow collection entries (9:19)
6 | "ledger": {
7 | ... (trimmed) ...
8 |
9 | "close_time": 560302643,
------------------------^
10 | "close_time_human": "2017-Oct-02 23:37:23",
11 | "close_time_resolution": 10,
YAMLException: missed comma between flow collection entries (9:19)
6 | "ledger": {
7 | ... (trimmed) ...
8 |
9 | "close_time": 560302643,
------------------------^
10 | "close_time_human": "2017-Oct-02 23:37:23",
11 | "close_time_resolution": 10,
at generateError (file:///another/devel/xrpl-dev-portal/node_modules/js-yaml/dist/js-yaml.mjs:1273:10)
at throwError (file:///another/devel/xrpl-dev-portal/node_modules/js-yaml/dist/js-yaml.mjs:1277:9)
at readFlowCollection (file:///another/devel/xrpl-dev-portal/node_modules/js-yaml/dist/js-yaml.mjs:1848:7)
at composeNode (file:///another/devel/xrpl-dev-portal/node_modules/js-yaml/dist/js-yaml.mjs:2532:11)
at readFlowCollection (file:///another/devel/xrpl-dev-portal/node_modules/js-yaml/dist/js-yaml.mjs:1881:7)
at composeNode (file:///another/devel/xrpl-dev-portal/node_modules/js-yaml/dist/js-yaml.mjs:2532:11)
at readFlowCollection (file:///another/devel/xrpl-dev-portal/node_modules/js-yaml/dist/js-yaml.mjs:1881:7)
at composeNode (file:///another/devel/xrpl-dev-portal/node_modules/js-yaml/dist/js-yaml.mjs:2532:11)
at readBlockMapping (file:///another/devel/xrpl-dev-portal/node_modules/js-yaml/dist/js-yaml.mjs:2194:12)
at composeNode (file:///another/devel/xrpl-dev-portal/node_modules/js-yaml/dist/js-yaml.mjs:2531:12)
Expected "/", "[", "{", boolean, class, end of input, id, identifier, null, number, string, variable, or whitespace but "'" found.
Roman promised a fix "in the next release" on Oct 2. It was not in the next release.