📣 Data App - Web Scraper (Apify)

[Luka-Loncar]

🗺 What's left for release

• Implement Apify web scraper integration
• Create content summarization service (not full indexing)
• Integrate OpenAI for summarization
• Design UI for web scraping interface
• Create API endpoints for web scraping

🌍 Features

• [FEATURE] Web Scraper Implementation via Apify (create ticket)
• [BE] Content Summarization Service (create ticket)
• [INTEGRATION] OpenAI API for Summarization (create ticket)
• [FE] Web Scraper UI (create ticket)
• [API] Web Scraping Endpoints (create ticket)

Acceptance Criteria

• Web pages summarized, not fully indexed
• OpenAI integration for intelligent summarization
• Rate limiting and cost controls
• Clear pricing model for users
• Support for various content types (articles, blogs, documentation)

Dependencies

• Decision on OpenAI API key (company vs user-provided)
• Apify account setup and configuration
• Cost model approval from leadership

✅ Release Checklist
Stage 0 - Pre-release

• Finalize OpenAI key decision
• Design summarization strategy
• Set up Apify integration
• Define rate limits and pricing

Stage 1 - Testing

• Test various website types
• Verify summarization quality
• Test rate limiting
• Validate cost calculations

Stage 2 - Production

• Soft launch with selected users
• Monitor costs closely
• Gather feedback on summarization quality
• Optimize based on usage patterns