Content Collector

https://rapidapi.com/vitosgeen/api/web-content-collector

The Content Collector is a GoLang application that allows users to collect and store content from various sources. It provides a simple and efficient way to scrape, parse, and save data from websites, APIs, or any other source.

Features

• Extract web pages: The application can extract web page using http.NewRequest with proxy ip with auth

Features in development

• Extract web pages: The application can extract web page using crome selenium with proxy ip without auth
• Web scraping: The application can extract data from HTML pages using CSS selectors or XPath expressions.
• API integration: It can consume data from RESTful APIs and store it in a structured format.
• Data parsing: The collected data can be parsed and transformed into a desired format.
• Data storage: The application supports storing the collected data in various databases or file formats.
• Customization: Users can define their own scraping rules, data parsing logic, and storage options.

Installation

To install and run the Content Collector, follow these steps:

1. Clone the repository: git clone https://github.com/VitalijKoshin/content_collector.git
2. Navigate to the project directory: cd content_collector
3. Run the application: make run