Skip to content

Redocly Migration Scripting and Pre‐Migration Moves

Jump to bottom
Rome Reginelli edited this page Oct 24, 2023 · 18 revisions

For the migration to the Redocly toolchain, there are several cases where Redocly needs things to be in a different format or syntax than they currently are.

There are three strategies we can use to make these changes, on a case by case basis. In order from most to least preferable, they are:

  1. Adapt the current site to a format that's compatible with both tools.
    1. "Remove" Variant: Remove any Dactyl-specific syntax and switch to something plainer that both can handle.
    2. "Shim" Variant: Implement a quick Dactyl plugin site so that it's compatible or mostly-compatible.
  2. Script a tool that can quickly apply sweeping changes from the Dactyl format to the Redocly format starting from basically any commit. Instead of committing the changes to the Redocly branch and copying work over, we commit the script to the main branch.
    1. "Extend" Variant: We implement something in Redocly that is a close analogue of what Dactyl supports, and the script does a find-and-replace from old→new syntax.
  3. Manual changes on a Redocly migration branch, which would have to be kept up-to-date and in-sync over a potentially extended period of time. In many cases we can assume that scripted / shimmed changes above may cover 90% of the work and manual changes can be applied as a last step during a code freeze just before the final migration.

Rather than forking once and having to keep the branches in-sync for a long time, it should be less friction to make a series of incremental changes to make the master branch more "Redocly-ready" while still maintaining compatibility with the current Dactyl site. Some of these changes may involve a large changeset, but by keeping each one constrained to one category of change and having the PR open only a couple days at most should reduce the impact and migration pain of any individual step.

Japanese translated pages

Approach: Adapt
Status ✅ Done (except for snippets)
Related PRs/issues #2206

The Dactyl-based site currently uses a convention where translated pages are {file}.ja.md in the same dir as the English versions. Redocly needs these to be under @i18n/ja/ in a parallel folder structure and named {file}.md.

The key is that Dactyl doesn't actually need the md files to follow any specific filename/folder structure. I think we can do one update to put them under an @i18n/ja folder without any breaking changes to how Dactyl handles them; this would only be a small, one-time disturbance to translation efforts and then they could resume and the files would already be in the right place for when we're ready to flip the switch to Redocly.

Links

Approach: Adapt & Shim

The Dactyl site uses relative links in the form {file}.html in most places. Redocly links can either be a relative path to the source file or an absolute path to the output path.

I already have a Dactyl plugin that finds links in the current Dactyl pages (including ones defined by the includes) and converts them into relative links in the format Redocly expects; I believe it would be pretty straightforward to reverse the logic, so we could change the links once to a format that's compatible with Redocly, and just have a Dactyl plugin subbing those links back to their existing values (on-build) until we're ready to move. The only exception would be any links from snippets, but that's a much smaller set of things to keep in sync.

Relative links are pretty inconvenient to use in many cases, though, so absolute links would be preferable. Absolute links come with their own challenges; we need to settle on the new / "pretty" URLs; those should, in theory, match the information architecture and navigation.

Reusable Links

Approach: Adapt & Remove?

The Dactyl site currently defines a bunch of reusable common links and uses {% include '...' %} to pull those definitions into a variety of files.

I've exported the list of reusable links as a yaml file and my link conversion script already uses that to convert links to the appropriate Redocly-compatible syntax without relying on an include. So as part of the migration of links into Redocly format, we can use this and remove the includes.

It does mean we lose the convenience of being able to write [server_info method][] or [AccountSet transaction][] and have those automatically link to the right place, though.

It would be nice to figure out how to keep some of this functionality if it's at all possible.

Index Pages

Approach: Adapt
Related PRs / Issues #2215

The Dactyl site has a bunch of auto-generated landing pages that contain just a list of children with blurbs for each; these are defined in the config file but don't generally have any representation in the filesystem. Redocly recently added some tooling to do something similar, but it still needs the files to have an index.md file (or something like it) in the filesystem where the landing goes.

We can create (largely empty) index.md pages for the current Dactyl site, and move the properties from the dacyl-config.yml to those pages' frontmatter, which would still generate the exact same site at build time. This move can be done once via a script. Having these index.md pages in place would help with making links that will port cleanly to Redocly, and the final conversion to Redocly would involve a smaller/simpler one-time change to each of these files (which can also be scripted).

Setting up a minimal index.md page using a somewhat generic template doesn't stop us from eventually replacing some of these pages with more customized landing pages.

Snippets

Approach: TBD
Status Waiting for updates from Redocly

Redocly currently reads snippet files (even if the folder and/or file name starts with _) and throws errors if they contain broken links. Also, any link paths in these snippets are relative to the snippet location, not to where the files are being built in the end.

In our meeting on 2023-09-28, Roman said they may implement improvements to snippets/partials "this week", so I'd like to wait to see what changes they implement before deciding on migration steps. But if it involves renaming files to have an underscore, we can move and then find-and-replace the files without breaking Dactyl's compatibility.

{{currentpage.name}} instances

Approach: Adapt & Remove
Status Waiting for updates from Redocly

This is one of a few cases where we use {{ variable }} syntax in the current docs. Redocly has {% $variable %} syntax but which variables it provides in which contexts doesn't quite line up with Dactyl's.

Most instances of {{currentpage.name}} in the existing docs are extraneous and we can actually just remove them. (For example, the header "AccountSet Flags" could just be "Flags" because it's already on the AccountSet page.) This will change/break some links, but we can add <a id='old-anchor-name'></a> tags to keep the old URLs functional if we need to.

{{ include_svg(...) }}

Approach: TBD. Script & Extend / Adapt & Remove
Status investigating options

Another case where we use {{ variable }} syntax is the {{ include_svg(...) }} macro. We can probably implement a Component that does something very similar and get the benefits of the current approach (theme-aware diagrams).

{{ include_code(...) }}

Approach: Script & Extend
Status Waiting for updates from Redocly

Redocly has already implemented a Component that does most of this, but currently it only does line numbers, not "starts_with" and "end_before". Roman promised he would add those, so then we could make a simple find-and-replace script to convert from the existing syntax to the Redocly equivalent.

{% if currentpage.md ... %}

Approach: Adapt & Remove
Status To-do

There are a few cases where our snippets display something slightly different depending on where they're being included, using syntax like the following example:

{% if currentpage.md == "tutorials/manage-the-rippled-server/installation/install-rippled-on-ubuntu.md" or
      currentpage.md == "tutorials/manage-the-rippled-server/installation/install-rippled-on-centos-rhel-with-yum" %}
        sudo systemctl restart rippled.service

{% elif currentpage.md == "tutorials/manage-the-rippled-server/installation/build-run-rippled-ubuntu.md" or
        currentpage.md == "tutorials/manage-the-rippled-server/installation/build-run-rippled-macos.md" %}

  * Use Ctrl-C to stop `rippled`, then start it again:

        ./rippled

{% endif %}

In a couple cases these haven't been maintained as pages have been moved/renamed so they're already not working as intended, and in general we're not getting all that much out of this syntax. We should just switch all the instances to something that will be "close enough" in all cases and remove the special syntax entirely.

{{ target.github_* }}

Approach: TBD
Status Investigating options under Redocly

Two variables we use commonly are github_forkurl and github_branch which are set to https://github.com/XRPLF/xrpl-dev-portal and master respectively, but can be overwritten with commandline vars, which lets preview builds refer to in-repo files that come from the fork & branch of the pull request, but automatically switch those links to the master branch when merged.

Example syntax:

For an example of such a file, see the provided [`ledger-file.json`]({{target.github_forkurl}}/blob/{{target.github_branch}}/content/_api-examples/rippled-cli/ledger-file.json).

Probably something similar can be accomplished with Redocly vars, TBD.

{{ target.owner_reserve }}, {{ target.base_reserve }}

Approach: TBD
Status Investigating options under Redocly

These variables are used in a couple places. The idea was any place where we want to tell people what the current reserve values are, we can use these variables and then if/when the reserves later change we can update them all in one place.

Probably something similar can be accomplished with Redocly vars, TBD. If not, it's not a huge loss to remove these instances and use plain text; reserves don't change that often anyway.

Custom Templates

Approach: Script, Manual touchups
Status In progress (split between Ripple DGE & Ripple Marketing responsibilities)

Marketing pages and Dev Tools have custom templates in Jinja format. Redocly needs these to be React components instead. Roman has created a script to do the conversions, which is kind of hacky and not thoroughly tested. This can be the starting point for manual edits to get the templates into shape.

Interactive Tutorials

Approach: TBD
Status Waiting on update from Redocly

Roman has proofed out some Interactive Tutorials stuff but we're still waiting on some kind of Redocly callback for when the page DOM has finished rendering for some of them to work better.

This will also involve changing some includes into Components and other details TBD.

Static files

Approach: Adapt, TBD
Status Still need to investigate further

The current site has two static folders—one for assets referenced by the templates (assets/ from repo top) and one for diagrams referenced by the Markdown contents (img/ from repo top).

Redocly needs these to be located in a folder named static/ from the top of the build folder (content/ under the current preview).

We can do a one-time move and update the Dactyl config and templates to use the path static/ instead of assets/ for the template assets.

TBD if Redocly also needs changes to how diagrams/etc are referenced from within the Markdown.

Clone this wiki locally