This API is being deprecated in favor of our main endpoints. Here is an example repo that generates LLMs.txt files: https://github.com/mendableai/create-llmstxt-py. This API endpoint will still remain active but we will no longer be maintaining it after June 30, 2025.
Generate consolidated text files from websites for LLM training and inference. Powered by @firecrawl_dev for web crawling and GPT-4-mini for text processing.
- Crawls websites and combines content into a single text file
- Generates both standard (
llms.txt) and full (llms-full.txt) versions - Web interface and API access available
- No API key required for basic usage
Visit llmstxt.firecrawl.dev to generate files through the browser.
GET https://llmstxt.firecrawl.dev/[YOUR_URL_HERE]
Note: Processing may take several minutes due to crawling and LLM operations.
Create a .env file with the following variables:
FIRECRAWL_API_KEY=
SUPABASE_URL=
SUPABASE_KEY=
OPENAI_API_KEY=
npm install
npm run dev
