Crawl management system

[spongebobsmartypants]

We want to develop a robust crawl management system so users have the flexibility to either use our existing crawler or seamlessly plug in their own custom crawler.

Tasks

1. Choose popular crawlers in different languages
   • Python: Scrapy
   • Node.js: Puppeteer
   • Java: Nutch
   • Go: Colly
2. Gather insights on best practices and challenges for each crawler
3. Define a unified framework to more easily integrate each crawler
   • Input: Seed URL and configuration options
   • Output: Standardized structured data formats