job/resources expiry delete/cleanup or archive

[foreveremain]

Slivka installations currently have no automated mechanism for removing files/directories associated with jobs.

Ideally, slivka should offer some form of cleaning up old jobs, with configurable behaviour according to the job's status:

• jobs that were cancelled can just be deleted
• jobs that errrored/never completed might be archived for testing/inspection (or simply deleted)
• jobs that have completed
  • who's results have been collected: delete after x days
  • who's results were used in another job: delete after the youngest job is older than x days (?)
  • who's results were never collected: delete after y days

As an alternative to deletion, it would also be great to be able to archive jobs - ie move their data/etc to a read-only store (ie S3 bucket, etc).

[foreveremain]

@stuartmac - can you think of anything else we might need here ?

Suggested approach:

• create a prototype cli tool that queries slivka's database to classify jobs
• look at how to safely mark jobs as gone from the system without affecting usage stats/etc
• create prototype deletion and archiving tools

Once we have that, all that's needed is to have slivka's own scheduler manage their execution :)

[stuartmac]

That's a clever idea to use a cli tool that we can then run via Slivka!

The first goal could be a simple delete after X days without worrying about chained results.

Could we move the job entries for the expired jobs to another table to avoid needing to implement a new job status?

I also like the idea of catching failed jobs.