How to crawl only the suggested site and not external sites #24
Unanswered
benzvi8888
asked this question in
Q&A
Replies: 1 comment
-
Unfortunately, I do not remember designing the code like this. The functionality is in these functions below, but when I designed this, modularity and best practice weren't in mind. Hence, I don't believe there will be an 'easy fix' unless you're willing to rewrite most of the Crawler class. |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
Seems when I checked the database the crawler go into social link and start index their content - it crawl Facebook, Twitter, LinkedIn and so on - it filled my database with it.
is there any way/option/code to limit the crawling to the site itself only?
Beta Was this translation helpful? Give feedback.
All reactions