It allows you to download a website from the Internet to a local directory, building recursively all directories, getting HTML, images, and other files from the server to your computer.
-
Updated
Jun 1, 2023 - Visual Basic .NET
#
website-crawler
Here are 31 public repositories matching this topic...
Python-based web crawling script with randomized intervals, user-agent rotation, and proxy server IP rotation to outsmart website bots and prevent blocking.
crawler scraper user-agent scraping beautiful-soup robots-txt beautifulsoup scrapper website-scraper scrapping-python website-crawler beautifulsoup4 crawling-python iprotation
-
Updated
Jun 10, 2025 - Python
Sneakpeek is a framework that helps to quickly and conviniently develop scrapers. It’s the best choice for scrapers that have some specific complex scraping logic that needs to be run on a constant basis
python crawler scraper vue scraping crawling python3 scrapers scraper-engine crawlers crawling-framework website-crawler scraping-framework crawler-python scraper-api crawling-engine
-
Updated
Aug 19, 2023 - Python
Scraper for https://marvelsnapzone.com to retrieve metadata of Marvel SNAP cards.
game crawler scraper marvel website-scraper website-crawler marvel-characters crawler-python marvel-snap
-
Updated
Jul 1, 2024 - Python
A universal and local phishing toolkit for audit purposes
python web-crawler phishing audit pentesting pentest webscraping pentest-tool website-crawler website-clone phishing-kit phishing-page phishing-script phishing-tool web-clone
-
Updated
Nov 21, 2024 - Python
An almost generic web crawler built using Scrapy and Python 3.7 to recursively crawl entire websites.
python scrapy-spider python3 scrapy scrapy-crawler scrapy-demo website-crawler crawling-sites recursive-crawling
-
Updated
Mar 1, 2022 - Python
A tutorial and code samples of web scraping with PHP
php web-scraping url-scraper screen-scraping website-crawler email-scraper wikipedia-scraper email-scraper-with-proxy
-
Updated
Jun 26, 2025 - PHP
WebKnoGraph is an open research project that uses data processing, vector embeddings, and graph algorithms to optimize internal linking at scale. Built for both academic and industry use, it offers THE FIRST FULLY transparent, AI-driven framework for improving SEO and site navigation through reproducible methods.
marketing-automation world-wide-web network-analysis link-prediction python-development search-engine-optimization website-crawler sentence-embeddings hits-algorithm graph-neural-networks page-rank web-networks internal-linking martech-backend synthethic-data-generation
-
Updated
Jun 28, 2025 - Python
A Simple Script To Scrape DuckDuckGo Search Results Using Python And Selenium WebDriver.
python scraper scraping selenium duckduckgo url-scraper google-dorks dork duckduckgo-search website-crawler bing-search dork-scanner dorking dorkscanner bing-dorking dorking-tool
-
Updated
Nov 1, 2022 - Python
🕷️ | ReconX is a Live-Website Crawler made to gather critical information with an option to take a picture of each site crawled!
python search-engine security website crawler information-retrieval osint hacking pentest information-security opsec information-gathering python-crawler website-scraper security-tools website-crawler livedata website-security osint-tool
-
Updated
Feb 20, 2025 - Python
This a project to demonstrate the use of standard python libraries like os, urllib, HTMLParser to create a minimalist webpage crawler that crawls webpages on a website to gather hyperlinks (URLs)
-
Updated
May 26, 2020 - Python
💫 Crawl urls from a webpage and provide a DomCrawler with Scraper Library
-
Updated
Nov 12, 2024 - PHP
Crawls a website to generate insights
-
Updated
Apr 18, 2019 - Go
Web Link Crawler: A Python script to crawl websites and collect links based on a regex pattern. Efficient and customizable.
python crawler scraper links website-scraper website-crawler clawler link-crawler crawler-python link-scraper-python link-scraper link-crawler-python scraper-python
-
Updated
May 30, 2024 - Python
Crawls website and collect SEO relevant data
-
Updated
Sep 27, 2022 - Go
Recursive website crawler
python sitemap crawler web crawling tor path python3 requests pentesting beautifulsoup pentest python-crawler website-crawler
-
Updated
Mar 23, 2022 - Python
sponge is a website crawler and links downloader command-line tool
kotlin website crawler downloader links sponge command-line wtfpl crawl-pages website-crawler link-downloader crawling-sites file-downloader
-
Updated
Nov 20, 2023 - Kotlin
The most advanced Lightshot (or prnt.sc) scraper ever!
java crawler scraper scraping website-crawler mass-downloader image-collection prntsc lightshot-scraper lightshot-screenshot prntscraper lightshotscraper
-
Updated
Dec 17, 2023 - Java
Java website crawler - library for analyze and testing websites
-
Updated
Dec 30, 2021 - Java
The most advanced Imgur scraper ever!
java crawler scraper imgur scraping image-downloader website-crawler imgur-downloader mass-downloader image-collection imgur-scraper imgurscraper imgurdownloader
-
Updated
Apr 29, 2023 - Java
Improve this page
Add a description, image, and links to the website-crawler topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the website-crawler topic, visit your repo's landing page and select "manage topics."