Rules for URL transformations?

[jrruethe]

Another feature request (unless this can already be done, I couldn't find it in the docs but maybe the javascript thing can do it).

I'd love to be able to specify a set of regular expression transformation to apply to a URL before it is crawled and scraped. By this, I mean that when a page is crawled and a list of links are produced, before those links are loaded into the crawler queue, there is the option to "transform" that url based on rules I create.

Here are a few examples of how I would use this:

• Converting a url like "https://www.reddit.com/r/LocalLLaMA/comments/1ij5yf2/how_i_built_an_open_source_ai_tool_to_find_my/" to "https://old.reddit.com/r/LocalLLaMA/comments/1ij5yf2/how_i_built_an_open_source_ai_tool_to_find_my/" (Note the change from "normal" reddit to "old" reddit) because I like the look of (and would rather archive) the old interface, plus it is easier to scrape.
• Appending ".json" to reddit posts: "https://old.reddit.com/r/LocalLLaMA/comments/1ij5yf2/how_i_built_an_open_source_ai_tool_to_find_my.json" such that the JSON data is archived, instead of the HTML
• Modifying links like "https://github.com/biolds/sosse" into "https://raw.githubusercontent.com/biolds/sosse/refs/heads/main/README.md" so I can be sure that I am capturing (and archiving) raw markdown for github links

Such a thing could also be used to remove tracking query parameters I suppose.

I imagine a spot to put a list of rules, that looks like:

• Regex to match against URL: "https://www.reddit.com/(.*)"
  URL Rewrite: "https://old.reddit.com/$1"

Bonus points if I can use this to take one URL and create multiple that are queued for crawling:

• Regex to match against URL: "https://www.reddit.com/(.*)"
  URL Rewrite:
  • "https://old.reddit.com/$1"
  • "https://old.reddit.com/$1.json"

^ That would let me archive both the rendered HTML as well as the JSON from a single rule

[biolds]

Yes, you can modify URLs and generate additional links using JavaScript, by modifying the DOM.

I’ve had a similar use case myself, I archive RFC pages, and many sites link to different renderers or variations. Ideally, I'd like a way to apply URL transformations dynamically while browsing rather than during crawling, so new rewrite rules could be added without needing to re-crawl everything.
I'm wondering what would the best approach to implement all or this.