Skip to content

Conversation

kanalive
Copy link

MCP Server and Client for CyberScraper

(Sorry for the multiple PRs this weekend! This PR introduces another idea: implementing CyberScraper as an MCP server. This allows it to be integrated with Claude Desktop and enables anyone to write their own MCP client to interact with CyberScraper's capabilities)

This PR adds Model Context Protocol (MCP) support to CyberScraper 2077, allowing the scraper to be used as a service with standardized inputs and outputs.

Changes:

  • Add MCP server implementation (cyberscraper_mcp.py) that wraps the WebExtractor functionality
  • Create Python MCP client (cyberscraper_mcp_client.py) for simple integration
  • Support passing API keys via the protocol rather than environment variables
  • Include proper error handling and task status tracking
  • Add documentation for integration with other systems

Benefits:

  • Enables integration with systems that support MCP (like Claude Desktop)
  • Allows headless/serverless operation of the scraping functionality
  • Makes it easy to build applications on top of CyberScraper's capabilities
  • Preserves all the AI-powered extraction features of the original

All tests passing. This addition doesn't modify any existing functionality - it simply adds new capabilities for services integration.

@itsOwen
Copy link
Owner

itsOwen commented Mar 24, 2025

I will test them out soon, as I don't have much time these days!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants