Look at replacing most of crawler with an external crawling package

[justinccdev]

This is a big one, but it's possible that most of this crawler should be replaced with Apache Nutch or similar. I originally hacked this out as a proof-of-concept but as usual, it grew a bit from there. However, now meeting scalability issues (parallel crawling, possibly on multiple machines, crawling to a large database, etc.) that we need to take a serious use at a well-established alternative like Nutch.

Some questions

• Is Nutch suitable? If so, 1.x or 2.x?

[justinccdev]

If the crawling code is replaced, ideally I would like to keep it around for a while, maybe within a plugin infrastructure. However, if this is too messy (which I suspect is likely), cleaner just to replace and remove all old code.

Also, I expect there will need to be some kind of 'shell' around nutch to present a user-friendly frontend. Python is really still my favourite language for this rather than doing everything in Java, but that might be another decision to make.

[justinccdev]

Also look at existing work by Federico on this at https://github.com/BioSchemas/bioschemas-nutch-indexer

[justinccdev]

Also consider http://stormcrawler.net/

[justinccdev]

From http://digitalpebble.blogspot.co.uk/2017/01/the-battle-of-crawlers-apache-nutch-vs.html, it looks like if we're going to use Nutch it should be 1.x, not 2.x

[justinccdev]

Also https://scrapy.org/, written in Python rather than Java

[justinccdev]

Perhaps also http://www.norconex.com/collectors/collector-http/

[justinccdev]

Rather than keep spamming this page, I've started writing the evaluation at https://github.com/justinccdev/bsbang-crawler/wiki/Transition-to-an-established-crawler-package, but comments can continue here.

[justinccdev]

Having now sampled and read various crawler projects, I think Scrapy/Frontera may be the way to go (see wiki page for more details). Will very soon start a new Github repository to explore re-implementing the crawler in the Scrapy/Frontera infrastructure.

[XiangpengHao]

I think Scrapy a good choice, it has a plugin scrapy-splash which addresses the #7 and far more popular than its competitors :)
I'll investigate more on scarpy these days.