Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 0 additions & 3 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,3 @@ jobs:

- name: Lint
run: npm run lint

- name: Test
run: npm test
76 changes: 0 additions & 76 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -311,8 +311,6 @@ Use this guide to select the right tool for your task:
- **If you want to search the web for info:** use **search**
- **If you want to extract structured data:** use **extract**
- **If you want to analyze a whole site or section:** use **crawl** (with limits!)
- **If you want to do in-depth research:** use **deep_research**
- **If you want to generate LLMs.txt:** use **generate_llmstxt**

### Quick Reference Table

Expand All @@ -324,8 +322,6 @@ Use this guide to select the right tool for your task:
| crawl | Multi-page extraction (with limits) | markdown/html[] |
| search | Web search for info | results[] |
| extract | Structured data from pages | JSON |
| deep_research | In-depth, multi-source research | summary, sources|
| generate_llmstxt | LLMs.txt for a domain | text |

## Available Tools

Expand Down Expand Up @@ -629,78 +625,6 @@ When using a self-hosted instance, the extraction will use your configured LLM.
}
```

### 9. Deep Research Tool (`firecrawl_deep_research`)

Conduct deep web research on a query using intelligent crawling, search, and LLM analysis.

**Best for:**
- Complex research questions requiring multiple sources, in-depth analysis.

**Not recommended for:**
- Simple questions that can be answered with a single search
- When you need very specific information from a known page (use scrape)
- When you need results quickly (deep research can take time)

**Arguments:**
- query (string, required): The research question or topic to explore.
- maxDepth (number, optional): Maximum recursive depth for crawling/search (default: 3).
- timeLimit (number, optional): Time limit in seconds for the research session (default: 120).
- maxUrls (number, optional): Maximum number of URLs to analyze (default: 50).

**Prompt Example:**
> "Research the environmental impact of electric vehicles versus gasoline vehicles."

**Usage Example:**
```json
{
"name": "firecrawl_deep_research",
"arguments": {
"query": "What are the environmental impacts of electric vehicles compared to gasoline vehicles?",
"maxDepth": 3,
"timeLimit": 120,
"maxUrls": 50
}
}
```

**Returns:**
- Final analysis generated by an LLM based on research. (data.finalAnalysis)
- May also include structured activities and sources used in the research process.

### 10. Generate LLMs.txt Tool (`firecrawl_generate_llmstxt`)

Generate a standardized llms.txt (and optionally llms-full.txt) file for a given domain. This file defines how large language models should interact
with the site.

**Best for:**
- Creating machine-readable permission guidelines for AI models.

**Not recommended for:**
- General content extraction or research

**Arguments:**
- url (string, required): The base URL of the website to analyze.
- maxUrls (number, optional): Max number of URLs to include (default: 10).
- showFullText (boolean, optional): Whether to include llms-full.txt contents in the response.

**Prompt Example:**
> "Generate an LLMs.txt file for example.com."

**Usage Example:**
```json
{
"name": "firecrawl_generate_llmstxt",
"arguments": {
"url": "https://example.com",
"maxUrls": 20,
"showFullText": true
}
}
```

**Returns:**
- LLMs.txt file contents (and optionally llms-full.txt)

## Logging System

The server includes comprehensive logging:
Expand Down
75 changes: 45 additions & 30 deletions package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

15 changes: 8 additions & 7 deletions package.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"name": "firecrawl-mcp",
"version": "1.12.0",
"description": "MCP server for Firecrawl web scraping integration. Supports both cloud and self-hosted instances. Features include web scraping, batch processing, structured data extraction, and LLM-powered content analysis.",
"version": "2.0.2",
"description": "MCP server for Firecrawl web scraping integration. Supports both cloud and self-hosted instances. Features include web scraping, search, batch processing, structured data extraction, and LLM-powered content analysis.",
"type": "module",
"bin": {
"firecrawl-mcp": "dist/index.js"
Expand All @@ -20,15 +20,17 @@
"lint:fix": "eslint src/**/*.ts --fix",
"format": "prettier --write .",
"prepare": "npm run build",
"publish": "npm run build && npm publish"
"publish": "npm run build && npm publish",
"publish-beta": "npm run build && npm publish --tag beta"
},
"license": "MIT",
"dependencies": {
"@mendable/firecrawl-js": "^1.19.0",
"@modelcontextprotocol/sdk": "^1.4.1",
"@mendable/firecrawl-js": "^3.0.3",
"@modelcontextprotocol/sdk": "^1.17.3",
"dotenv": "^16.4.7",
"express": "^5.1.0",
"shx": "^0.3.4",
"typescript": "^5.9.2",
"ws": "^8.18.1"
},
"devDependencies": {
Expand All @@ -43,8 +45,7 @@
"jest": "^29.7.0",
"jest-mock-extended": "^4.0.0-beta1",
"prettier": "^3.1.1",
"ts-jest": "^29.1.1",
"typescript": "^5.3.3"
"ts-jest": "^29.1.1"
},
"engines": {
"node": ">=18.0.0"
Expand Down
Loading