[Nodriver] using profiles

[Aperico-com]

I have been away from scraping/reverse engineering for many years and now after returning I have landed in using Nodriver. All other libs I tried failed to meet my requirements. I have to say that I am very impressed by Nodrivers strength in bypassing bot detection and the find(best_match=True) is pure magic!

I would however like to have a few things clarified since I cannot find info on it in docs or github.

1. Using profiles. Do you only have to specify a folder? Can the folder be non existent the first time? Or does it have to be an existing Chrome profile folder?
2. The amount of requests to ad networks and google api is so excessive nowadays and can become expensive if you use a paid proxy service. How do you guys tackle this? Using profiles for each of my "users" should help? Are you using ad-blockers?
3. Simply doing driver = await uc.start() and then tab = await driver.get('https://xxxyyy.com') does not always work for me. Sometimes I only get a blank browser without any address filled in. Are you supposed to do any await driver command or something similar, I can not see anything about that in the docs.

[nathanfallet]

Hi!

1. For profiles, you can use data_dir, which will be automatically created if it does not exist yet. You can have as much directory as you want to have different sessions. Also, if you don't specify one you get a fresh session from a temporary folder each time.
2. For this, you can make a custom chrome extension that you load at startup that blocks requests for a certain domain using rules like this:

[
  {
    "id": 1,
    "priority": 1,
    "action": {
      "type": "block"
    },
    "condition": {
      "urlFilter": "ads",
      "resourceTypes": [
        "image",
        "media",
        "xmlhttprequest",
        "sub_frame",
        "object"
      ]
    }
  }
]

3. Try adding some sleep between the two. It might be some sort of race condition. For this, I'd recommend switching to zendriver which implemented a lot of fixes for this kind of issues.