How to emit markdown files for llms.txt?

[jcubic]

I want to create llms.txt file. I've found a plugin, but it just concatenates all the markdown files into a single text file.

llms.txt should include a list of links with more information. I want to create the file by hand and link to markdown files.

What is the best way to do this? Can you use docusaurus config file for this?

[Devon-White]

You may be interested in the plugin i have been working on. It allows you to accomplish the things you are looking for.

1. Granular control of the llms & llms-full file(s). Decide what get puts where, what gets excluded or included. Built in config to mass turn off/on certain kind of pages (docs, pages, blogs, versioned docs, etc...)
2. Ability to generate markdown files for any doc that is processed for the llms.txt file. They will be come available by just appending a .md at the end of your current route.
3. New theme to accompany plugin to add a Copy Page contents button in top right, like you are starting to see with other doc providers like mintlify.

Plugin: https://www.npmjs.com/package/@signalwire/docusaurus-plugin-llms-txt

Theme: https://www.npmjs.com/package/@signalwire/docusaurus-theme-llms-txt

The plugin has a major rework in-progress (pretty much done just waiting for feedback), so be sure to install the latest alpha versions if you would like to test

[jcubic]

This is the code I use to generate Blog thumbnails, I can parse the markdown files. I can emit the markdown files this way. It's actually a workaround, since this is for RSS feed and there is no other way to access the markdown files.

        blog: {
          showReadingTime: true,
          editUrl: 'https://github.com/jcubic/lips/tree/master/docs/',
          feedOptions: {
            type: 'rss',
            limit: 10,
            copyright: `Copyright (C) ${new Date().getFullYear()} Jakub T. Jankiewicz`,
            title: 'LIPS Scheme blog',
            description: 'LIPS Scheme blog RSS Feed',
            createFeedItems: async ({ blogPosts, ...params }) => {
              if (isProd) {
                 await renderSocialCards(blogPosts);
              }
              const feedItems = await params.defaultCreateFeedItems({ blogPosts, ...params });
              feedItems.forEach((feedItem,index) => {
                const blogPost = blogPosts[index]!;
                const permalink = blogPost.metadata.permalink;
                const excerpt = blogPost.content.replace(/<!--\s*truncate\s*-->[\s\S]*$/, '').trim();
                  const html = marked.parse(excerpt, { async: false }) + `<br/><a href="${permalink}">see the rest of the article</a>`
                feedItem.content = html;
              });
              return feedItems;
            }
          }
        },

You can see the whole project here: https://github.com/jcubic/lips/tree/master/docs

In Eleventy it's way easier to emit the markdown files, because Eleventy is more flexible, it don't restrict you like Docusaurus does. I know that Elventy is totally different framework, you can create any website you want.

What I would expect is some kind of hooks where you can access the markdown files. Maybe something where you have Markdown code as input, and should return HTML as output. Similar how you generate the RSS Feed and can hook into its generation.

Generating markdown files from HTML, doesn't make sense if you can just access the markdown files directly. I know that I can write Node.js script that will process the files on my own, but it would be nice if it's part of the project.

[slorber]

I honestly don't understand what you try to do.

You mention blog thumbnails and show related code. How thumbnails are even related to this discussion?

What does "emit the markdown files" even mean? I don't see what you emit exactly in the code above, but renderSocialCards() naming doesn't imply it emits anything related to Markdown 🤷‍♂️

What I would expect is some kind of hooks where you can access the markdown files. Maybe something where you have Markdown code as input, and should return HTML as output. Similar how you generate the RSS Feed and can hook into its generation.

I don't understand what you mean. Show a concrete minimal example. Show me all the inputs, show me your ideal API, and show me the exact static output we should emit.

You mention "accessing markdown files", but what does it even mean?

You mention "returning HTML as output", but to me, that's the role of the React theme. And you have to choose: do you want to output HTML or Markdown, that's really unclear.

Generating markdown files from HTML, doesn't make sense if you can just access the markdown files directly. I know that I can write Node.js script that will process the files on my own, but it would be nice if it's part of the project.

If you can do it in userland, then show us first a concrete minimal example using Node.js code in the config. That would be really helpful to understand what you ask us to add to Docusaurus core. And we might be able to refactor your example and propose a better alternative.

[jcubic]

You mention blog thumbnails and show related code. How thumbnails are even related to this discussion?

This is just my code where I access Markdown content. I'm asking for the same for other markdown files.

You mention "accessing markdown files", but what does it even mean?

It means blogPost.content like in the code above it contain the content of the Markdown files. I can do whatever I want with this. I can write it to a file if I want, but I can also generate thumbnails for all docs pages. This is what I call flexibility. But I need markdown content and the output path.

Show a concrete minimal example.

You have it on above example. I don't want to enforce anything concrete. But I can be something like:

process_file: function({ content, path }) {
    return marked.parse(content);
},

This would be the default code, that create HTML output, I just assume that the framework create HTML files since it work without JavaScript. But having this hook, I can write the markdown file before the HTML file is created.

process_file: async function({ content, path }) {
    const fname = path.replace(/\/$/, '') + '.md';
    const markkdown = content.replace(/---[\s\S]---/, '');
    await fs.writeFile(fname, markdown);
    return marked.parse(content);
},

[slorber]

I understand better now 👍

The problem is that what would you do with MDX content like this:

# Title

import Content from './_content.mdx'

<Content/>

::my-custum-directive

Unlike other frameworks, we use MDX which lets you compose Markdown and JSX in a flexible way.

From the example above, you can't really produce a decent HTML or std Markdown output without running this file within the MDX runtime.

For that reason, what you ask us to build is not really sustainable in general; that's the reason our blog feed plugin parses the output HTML instead of using the MDX source files to generate the feed content. If we ever built a solution to emit Markdown files, this is also the approach we'd take because we absolutely need to support the case above.

---

If you don't really use Docusaurus MDX features (JSX, directives...) and you are aware of the tradeoffs you make, then maybe emitting Markdown files from sources is viable for your specific use case, but not as a general-purpose solution.

To access the source files, I would probably try the following approaches:

Use the parseFrontMatter hook

Although not really designed for that purpose, it gives you a flexible hook that runs every time we try to parse front matter in MD/MDX files.

Tradeoffs:

• doesn't run on MDX partials (usually starting with _, not creating routes)
• runs often, doing something expensive here can slow down your dev experience so maybe add a condition

export default {
  markdown: {
    parseFrontMatter: async (params) => {
      // Reuse the default parser logic
      const result = await params.defaultParseFrontMatter(params);
      // Run your custom code here
      await emitMarkdownFile(params.filePath, params.fileContent)
      return result;
    },
  },
};

Extending the core content plugins

Although we don't really have a first-class API for it, extending existing core plugins remains possible: #4138

I would probably extend the contentLoaded method of the plugin you want to emit markdown files for.

Creating a plugin with allContentLoaded()

Although we haven't publicly announced this API, it exists, is stable and is used internally by our debug plugin.

See #9931

It lets you access in one lifecycle the aggregated content of all the site plugins (including docs, blog, pages data). From there, you can do whatever you want with that data, including copying files.

Tradeoff: also try to keep this method fast because on any MDX file change, it has to re-run in DEV, and can make your DX significantly slower (that's why we keep it secret for now). You can gate your code so that it only emit markdown files in production builds.

Creating a plugin with postBuild using routes metadata

Our core content plugins (docs, blog, pages) now assign metadata to all the routes they create, including which source file triggered the creation of that specific page.

import type {Plugin} from '@docusaurus/types';
import {flattenRoutes} from '@docusaurus/utils';

export default function myPlugin(): Plugin<void> {
  return {
    name: "my-plugin",

    async postBuild({routes}) {
      const allChildRoutes = flattenRoutes(routes);
      
      await Promise.all(allChildRoutes.map(async (route) => {
        if (route.metadata?.sourceFilePath ) {
          await emitMarkdownFile(route.metadata.sourceFilePath)
        }
      }))
    }  
  };
}

Tradeoff: it only works for core plugins that set that metadata, and may not work for routes created by third-party plugins (until they discover this is possible and set it too).

I like this solution the most, and if we ever implement a general-purpose plugin to emit Markdown files for LLMs this is probably the way we'd implement this, because:

• it only runs in prod builds, and doesn't slow down hot reloads
• it is postBuild, so at that time it can read the HTML output and see what has been rendered by MDX (including JSX, partials, directives...), so it would be more reliable that using source files

---

I hope this answers your question

[jcubic]

Thanks, will check it. As for mdx I think that I've seen file extension in some of the llms.txt file, or from Context 7 MCP. But don't remember what it was, and I'm not sure what was their approach. Probably they just show the JSX code directly, since they look like live examples that don't bring any value when you want just text.