Skip to content

[Feat.] build ChekPoint system for the scrapper. #9

@rajatkb

Description

@rajatkb

In case of a sudden failure and crash, we don't want the process to restart and start scrapping from the beggining.

** REQUIREMENT **
1 . Add method of Database class inside db.py to store check point information
2. Let user implementing plugin decide how to use the checkpointing system.

Note: Database class uses pymongo and contains no data that is modified after creation of object. So it's safe to say it is thread safe.

Metadata

Metadata

Assignees

No one assigned

    Labels

    easyGSSOC label for beginner tagenhancementNew feature or requestgood first issueGood for newcomersgssoc20GSSOC label for gscco20 taghelp wantedExtra attention is needed

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions