Content Manager - Job Scheduler

[AlexRuiz7]

Description

The Content Manager is designed to run on schedule, in order to maintain the content up to date. This is achieved by querying the CTI API regularly.

On this issue, we'll work on including the Job Scheduler to the Content Manager plugin, adapting the code as much as needed to fit the Content Manager's responsibilities into a scheduled based job workflow.

The plugin's workflow for the tier 2 is:

1. On schedule, the plugin:
   1. Requests the latest information of the CVE consumer from the CTI API.
   2. Updates the consumer information in the index (last_offset and last_snapshot_link values as read from the response).
   3. If the offset is 0, the context will be initialized from a snapshot
      1. The Content Manager downloads the snapshot (last_snapshot_link field in the index).
      2. The Content Manager unzips the snapshot.
      3. The Content Manager reads and indexes the content of the snapshot into an index using JSON Streaming.
      4. As new content is available, the Content Manager generates a command for the Command Manager.
   4. If the offset is the same as the offset fetched from the CTI API for that context and consumer. The content is up-to-date and nothing needs to be done.
   5. If the offset is lower than last_offset for that context and consumer, the content needs to be updated.
      1. Subtract the difference in offsets: difference = latest_offsest - offset
      2. While difference > 0
         • Fetch changes in batches of 1000 elements as maximum
         • Apply JSON-patch to the content.
         • If a patch is successful, the offset is updated in the index.
         • If a patch fails, the update is interrupted and offset is set to 0, forcing a re-initialization from a snapshot.
      3. As new content is available, the Content Manager generates a command for the Command Manager.

Functional requirements

• The Content Manager regularly checks for updates.
• The Content Manager automatically updates the content when needed.
• The Content Manager automatically initializes the content when needed (snapshot).
• The Content Manager recovers from failed content updates by reinitializating the content from a snapshot.
• The Content Manager sends a command to the Command Manager when new content is available, either if it was updated or initialized.

Implementation restrictions

• The Job Scheduler plugin must be used.
• Keep any setting related to the Job Scheduler as decoupled as possible, so we can extract it to an external setting later on.
• The Content Manager provided adequate logging.

Plan

• Implement the Job Scheduler interface.
• Adapt the Content Manager code and workflow to fit into a scheduled job workflow.