Welcome to ferrum Discussions!

[route]

👋 Welcome!

We’re using Discussions as a place to connect with other members of our community. We hope that you:

• Ask questions you’re wondering about.
• Share ideas.
• Engage with other community members.
• Welcome others and are open-minded. Remember that this is a community we
  build together 💪.

To get started, comment below with an introduction of yourself and tell us about what you do with this community.

[ttilberg]

Howdy! I'm Tim Tilberg and I collect and organize product pricing data at an enterprise level. Most of our scraping is with Scrapy and Puppeteer depending on the nature of the site. I greatly appreciate development in modern browser tools for Ruby, as I think Ruby provides a better experience out of the box for data parsing. I've been watching this project closely since it was first published to GitHub.

[route]

Didn't you move all your projects to ferrum yet? What is the obstacle? :)

[ttilberg]

For 80% of our collections, I don't use a full, legit browser. It would require a lot more overhead. For the 20% that have browser-like requirements, I have used Puppeteer along with Puppeteer Stealth, which has largely gotten me by without having to try. We have code for literally hundreds of sites, and so largely, there's a lot of "if it's not broken, don't fix it". Most of this work is just by myself, and (thankfully) recently one other newer .py dev.

I've used Ferrum a few times for newly-broken things that do require a proper browser, and am using it in one of our collections. However, I've run into a lot of issues related to "that node no longer exists" type stuff during ajax requests. This issue and comment in particular seemed very relevant. I'm glad to see a lot of activity pick up in the last few months, and hoping that as it gets better refined, I'll have better luck.

[route]

If you have reproducible examples you have my email. I think mostly everything should work like in 99% of the cases. I also think I can address your other issues with https://github.com/rubycdp/vessel where I'd like to intro something like mechanize besides ferrum so you have an ability to parse sites without browser like scrapy. There are a lot of things to work on...