max_session_rotations not being respected in AdaptivePlaywrightCrawler.with_beautifulsoup_static_parser

[ericvg97]

Hi, its me again, sorry for the amount of issues I am opening:) I think in this case it is a bug. I am using this

concurrency_settings  =  ConcurrencySettings(
    min_concurrency=1,
    max_concurrency=1,
    desired_concurrency=1,
    max_tasks_per_minute=200)
    
crawler = AdaptivePlaywrightCrawler.with_beautifulsoup_static_parser(
    max_requests_per_crawl=10000,
    playwright_crawler_specific_kwargs={
        "browser_type": "chromium",
        "headless": True,
    },
    max_session_rotations=10,
    retry_on_blocked=True,
    concurrency_settings=concurrency_settings,
    use_session_pool=True,
    max_request_retries=5,
    keep_alive=True,
)

and if I crawl this url "http://httpbingo.org/status/429" it keeps rotating the session ("Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...") until infinity

[ericvg97]

This is my full crawler async def crawl(seed_url: str) -> None:

`async def crawl(seed_url: str) -> None:
concurrency_settings = ConcurrencySettings(
min_concurrency=1,
max_concurrency=1,
desired_concurrency=1,
max_tasks_per_minute=200,
)
statistics = Statistics.with_default_state(log_interval=timedelta(minutes=1), statistics_log_format="table")
config = service_locator.get_configuration()
config.purge_on_start = True

engine = create_async_engine("postgresql+psycopg://postgres:password@localhost:5432/lighthouse")
sql_client = SqlStorageClient(engine=engine)

service_locator.set_storage_client(sql_client)

crawler = AdaptivePlaywrightCrawler.with_beautifulsoup_static_parser(
    max_requests_per_crawl=10000,
    playwright_crawler_specific_kwargs={
        "browser_type": "chromium",
        "headless": True,
    },
    max_session_rotations=10,
    retry_on_blocked=True,
    concurrency_settings=concurrency_settings,
    statistics=statistics,
    configure_logging=True,
    use_session_pool=True,
    max_request_retries=5,
    request_handler_timeout=timedelta(seconds=30),
    keep_alive=True,
)

async def default_handler(context: AdaptivePlaywrightCrawlingContext) -> None:
    context.log.info(f"Processing {context.request.url} ...")

    context.request.user_data["something"] = "something"

    await context.enqueue_links(strategy="same-domain")

crawler.router.default_handler(default_handler)

async def pre_navigation_hook(context: BasicCrawlingContext) -> None:
    context.log.info(f"Before processing {context.request.url}")

crawler.pre_navigation_hook(pre_navigation_hook)

async def error_handler(context: BasicCrawlingContext, error: Exception) -> None:
    logger.error(f"Failed request {context.request.url}, after {context.request.retry_count} retries")
    logger.error(f"Exception: {error}")

async def failed_request_handler(context: BasicCrawlingContext, error: Exception) -> None:
    logger.error(
        f"Not retrying any more for request {context.request.url}, after {context.request.retry_count} retries"
    )
    logger.error(f"Exception was: {error}")

    context.request.user_data["error"] = True

crawler.error_handler(error_handler)
crawler.failed_request_handler(failed_request_handler)

logger.info(f"Starting crawling on '{seed_url}'")
await crawler.run([seed_url])

logger.info("Finished crawling")`

[Mantisus]

and if I crawl this url "http://httpbingo.org/status/429" it keeps rotating the session ("Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...") until infinity

The code above performs exactly 10 session rotations. However, note that run will never terminate because you are using keep_alive=True.

[crawlee.crawlers._basic._basic_crawler] WARN  Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
[crawlee.crawlers._basic._basic_crawler] WARN  Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
[crawlee.crawlers._basic._basic_crawler] WARN  Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
[crawlee.crawlers._basic._basic_crawler] WARN  Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
[crawlee.crawlers._basic._basic_crawler] WARN  Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
[crawlee.crawlers._basic._basic_crawler] WARN  Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
[crawlee.crawlers._basic._basic_crawler] WARN  Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
[crawlee.crawlers._basic._basic_crawler] WARN  Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
[crawlee.crawlers._basic._basic_crawler] WARN  Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
[crawlee.crawlers._basic._basic_crawler] ERROR Request to http://httpbingo.org/status/429 failed and reached maximum retries

[ericvg97]

It is not related to keep_alive=True, it happens with keep_alive=False in the same way. These are my logs (I killed it with ctrl+c). I am using Crawlee 1.0.2 in case it helps

[dip.crawler.main] INFO  Starting crawling on 'http://httpbingo.org/status/429'
[crawlee.crawlers._basic._basic_crawler] INFO  Crawled 0/1 pages, 0 failed requests, desired concurrency 1.
[crawlee._autoscaling.autoscaled_pool] INFO  current_concurrency = 0; desired_concurrency = 1; cpu = 0.0; mem = 0.0; event_loop = 0.0; client_info = 0.0
[crawlee.crawlers._basic._basic_crawler] INFO  Before processing http://httpbingo.org/status/429
[dip.crawler.main] ERROR Failed request http://httpbingo.org/status/429, after 0 retries
[dip.crawler.main] ERROR Exception: Assuming the session is blocked based on HTTP status code 429
[crawlee.crawlers._basic._basic_crawler] WARN  Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
[crawlee.crawlers._basic._basic_crawler] INFO  Before processing http://httpbingo.org/status/429
[dip.crawler.main] ERROR Failed request http://httpbingo.org/status/429, after 0 retries
[dip.crawler.main] ERROR Exception: Assuming the session is blocked based on HTTP status code 429
[crawlee.crawlers._basic._basic_crawler] WARN  Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
[crawlee.crawlers._basic._basic_crawler] INFO  Before processing http://httpbingo.org/status/429
[dip.crawler.main] ERROR Failed request http://httpbingo.org/status/429, after 0 retries
[dip.crawler.main] ERROR Exception: Assuming the session is blocked based on HTTP status code 429
[crawlee.crawlers._basic._basic_crawler] WARN  Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
[crawlee.crawlers._basic._basic_crawler] INFO  Before processing http://httpbingo.org/status/429
[dip.crawler.main] ERROR Failed request http://httpbingo.org/status/429, after 0 retries
[dip.crawler.main] ERROR Exception: Assuming the session is blocked based on HTTP status code 429
[crawlee.crawlers._basic._basic_crawler] WARN  Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
[crawlee.crawlers._basic._basic_crawler] INFO  Before processing http://httpbingo.org/status/429
[dip.crawler.main] ERROR Failed request http://httpbingo.org/status/429, after 0 retries
[dip.crawler.main] ERROR Exception: Assuming the session is blocked based on HTTP status code 429
[crawlee.crawlers._basic._basic_crawler] WARN  Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
[crawlee.crawlers._basic._basic_crawler] INFO  Before processing http://httpbingo.org/status/429
[dip.crawler.main] ERROR Failed request http://httpbingo.org/status/429, after 0 retries
[dip.crawler.main] ERROR Exception: Assuming the session is blocked based on HTTP status code 429
[crawlee.crawlers._basic._basic_crawler] WARN  Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
[crawlee.crawlers._basic._basic_crawler] INFO  Before processing http://httpbingo.org/status/429
[dip.crawler.main] ERROR Failed request http://httpbingo.org/status/429, after 0 retries
[dip.crawler.main] ERROR Exception: Assuming the session is blocked based on HTTP status code 429
[crawlee.crawlers._basic._basic_crawler] WARN  Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
[crawlee.crawlers._basic._basic_crawler] INFO  Before processing http://httpbingo.org/status/429
[dip.crawler.main] ERROR Failed request http://httpbingo.org/status/429, after 0 retries
[dip.crawler.main] ERROR Exception: Assuming the session is blocked based on HTTP status code 429
[crawlee.crawlers._basic._basic_crawler] WARN  Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
[crawlee.crawlers._basic._basic_crawler] INFO  Before processing http://httpbingo.org/status/429
[dip.crawler.main] ERROR Failed request http://httpbingo.org/status/429, after 0 retries
[dip.crawler.main] ERROR Exception: Assuming the session is blocked based on HTTP status code 429
[crawlee.crawlers._basic._basic_crawler] WARN  Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
[crawlee.crawlers._basic._basic_crawler] INFO  Before processing http://httpbingo.org/status/429
[dip.crawler.main] ERROR Failed request http://httpbingo.org/status/429, after 0 retries
[dip.crawler.main] ERROR Exception: Assuming the session is blocked based on HTTP status code 429
[crawlee.crawlers._basic._basic_crawler] WARN  Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
[crawlee.crawlers._basic._basic_crawler] INFO  Before processing http://httpbingo.org/status/429
[dip.crawler.main] ERROR Failed request http://httpbingo.org/status/429, after 0 retries
[dip.crawler.main] ERROR Exception: Assuming the session is blocked based on HTTP status code 429
[crawlee.crawlers._basic._basic_crawler] WARN  Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
[crawlee.crawlers._basic._basic_crawler] INFO  Before processing http://httpbingo.org/status/429
[dip.crawler.main] ERROR Failed request http://httpbingo.org/status/429, after 0 retries
[dip.crawler.main] ERROR Exception: Assuming the session is blocked based on HTTP status code 429
[crawlee.crawlers._basic._basic_crawler] WARN  Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
[crawlee.crawlers._basic._basic_crawler] INFO  Before processing http://httpbingo.org/status/429
[dip.crawler.main] ERROR Failed request http://httpbingo.org/status/429, after 0 retries
[dip.crawler.main] ERROR Exception: Assuming the session is blocked based on HTTP status code 429
[crawlee.crawlers._basic._basic_crawler] WARN  Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
[crawlee.crawlers._basic._basic_crawler] INFO  Before processing http://httpbingo.org/status/429
[dip.crawler.main] ERROR Failed request http://httpbingo.org/status/429, after 0 retries
[dip.crawler.main] ERROR Exception: Assuming the session is blocked based on HTTP status code 429
[crawlee.crawlers._basic._basic_crawler] WARN  Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
[crawlee.crawlers._basic._basic_crawler] INFO  Before processing http://httpbingo.org/status/429
[dip.crawler.main] ERROR Failed request http://httpbingo.org/status/429, after 0 retries
[dip.crawler.main] ERROR Exception: Assuming the session is blocked based on HTTP status code 429
[crawlee.crawlers._basic._basic_crawler] WARN  Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
[crawlee.crawlers._basic._basic_crawler] INFO  Before processing http://httpbingo.org/status/429
[dip.crawler.main] ERROR Failed request http://httpbingo.org/status/429, after 0 retries
[dip.crawler.main] ERROR Exception: Assuming the session is blocked based on HTTP status code 429
[crawlee.crawlers._basic._basic_crawler] WARN  Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
[crawlee.crawlers._basic._basic_crawler] INFO  Before processing http://httpbingo.org/status/429
[dip.crawler.main] ERROR Failed request http://httpbingo.org/status/429, after 0 retries
[dip.crawler.main] ERROR Exception: Assuming the session is blocked based on HTTP status code 429
[crawlee.crawlers._basic._basic_crawler] WARN  Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
[crawlee.crawlers._basic._basic_crawler] INFO  Before processing http://httpbingo.org/status/429
[dip.crawler.main] ERROR Failed request http://httpbingo.org/status/429, after 0 retries
[dip.crawler.main] ERROR Exception: Assuming the session is blocked based on HTTP status code 429
[crawlee.crawlers._basic._basic_crawler] WARN  Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
[crawlee.crawlers._basic._basic_crawler] INFO  Before processing http://httpbingo.org/status/429
[dip.crawler.main] ERROR Failed request http://httpbingo.org/status/429, after 0 retries
[dip.crawler.main] ERROR Exception: Assuming the session is blocked based on HTTP status code 429
[crawlee.crawlers._basic._basic_crawler] WARN  Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
[crawlee.crawlers._basic._basic_crawler] INFO  Before processing http://httpbingo.org/status/429
[dip.crawler.main] ERROR Failed request http://httpbingo.org/status/429, after 0 retries
[dip.crawler.main] ERROR Exception: Assuming the session is blocked based on HTTP status code 429
[crawlee.crawlers._basic._basic_crawler] WARN  Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
[crawlee.crawlers._basic._basic_crawler] INFO  Before processing http://httpbingo.org/status/429
[dip.crawler.main] ERROR Failed request http://httpbingo.org/status/429, after 0 retries
[dip.crawler.main] ERROR Exception: Assuming the session is blocked based on HTTP status code 429
[crawlee.crawlers._basic._basic_crawler] WARN  Encountered "crawlee.errors.SessionError: Assuming the session is blocked based on HTTP status code 429", rotating session and retrying...
[crawlee.crawlers._basic._basic_crawler] INFO  Before processing http://httpbingo.org/status/429
^C[crawlee.crawlers._basic._basic_crawler] INFO  Pausing... Press CTRL+C again to force exit.
[crawlee._autoscaling.autoscaled_pool] INFO  Waiting for remaining tasks to finish
??????????????????? 0 5
[crawlee.crawlers._basic._basic_crawler] WARN  Retrying request to http://httpbingo.org/status/429 due to: Page.goto: net::ERR_ABORTED at http://httpbingo.org/status/429
[dip.crawler.main] ERROR Failed request http://httpbingo.org/status/429, after 1 retries
[dip.crawler.main] ERROR Exception: Page.goto: net::ERR_ABORTED at http://httpbingo.org/status/429
Call log:
  - navigating to "http://httpbingo.org/status/429", waiting until "load"

None
[crawlee.crawlers._basic._basic_crawler] INFO  Error analysis: total_errors=1 unique_errors=1
[crawlee.crawlers._basic._basic_crawler] INFO  The crawl was interrupted. To resume, do: CRAWLEE_PURGE_ON_START=0 python scripts/crawling/crawl_url.py
[crawlee.crawlers._basic._basic_crawler] INFO  Final request statistics:
┌───────────────────────────────┬───────┐
│ requests_finished             │ 0     │
│ requests_failed               │ 0     │
│ retry_histogram               │ [0]   │
│ request_avg_failed_duration   │ None  │
│ request_avg_finished_duration │ None  │
│ requests_finished_per_minute  │ 0     │
│ requests_failed_per_minute    │ 0     │
│ request_total_duration        │ 0s    │
│ requests_total                │ 0     │
│ crawler_runtime               │ 7.50s │
└───────────────────────────────┴───────┘
[dip.crawler.main] INFO  Finished crawling
[asyncio] ERROR Future exception was never retrieved
future: <Future finished exception=Exception('Connection closed while reading from the driver')>
      Exception: Connection closed while reading from the driver