bs_and_requests.py is a simple single spider, a basic crawler. It just displays the links and does not attempt to queue and crawl the links. Rest of the files are the multi threaded spider crawler. When the main.py file is run it creates and stores links in two text files "queued.txt" and "crawled.txt". There seems to be a problem with the implementation of multi threading, the crawling freezes after some time, should investigate
-
Notifications
You must be signed in to change notification settings - Fork 0
mathav95raj/Link_crawler
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
About
No description, website, or topics provided.
Resources
Stars
Watchers
Forks
Releases
No releases published
Packages 0
No packages published