Skip to content

[Improve] add adaptive recuring non blocking timeout handling for Scrapper.py get_page method #29

@rajatkb

Description

@rajatkb

Requirement

  • The get_page requirement is currently set to static manually set timeout duration. It would great to have an adaptive method get_page , which will not timeout because of few network stability issues but will also not wait indefinitely.

Where to look

  • Scrapper.py inside commons

Update: An implementation is provided for exponential wait time increase in utility package as AdaptiveRequest class. The issue is not resolved, as in case of fault network the system may end up raising its wait time indefinite, i.e leading back to the old problem of having no wait time for request.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingenhancementNew feature or requestgssoc20GSSOC label for gscco20 taghardGSSOC label for beginner tag

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions