Refactor otter grade image builds so that assignment data is not included in the grading image

[mbacon-renci]

We're trying to run otter grade inside a running kubernetes pod, and because of security limits those pods can't run Docker (which creates a container-in-container).

I am working on a patch to the otter grader code that would abstract the container creation that happens in otter.grade.containers.grade_submission so that it uses Docker by default but that can be overridden with an environment variable or similar.

One of the issues we're having in trying to use other container runtimes, like possibly kubernetes or OpenShift, is that otter grader wants to build a new image every time otter grade is run on an assignment.

From what I can see, the main changes to the image every time is that the config and assignment files are copied in.

Is there any reason that code couldn't be refactored slightly so that the same image is used for every assignment, and the assignment data copied in along with the submission data in the grade_submission method. This would make it much easier to deploy in more complex environments by making it much easier to have a single grader image published on a repository that could just be updated for each assignment. It looks like this should be possible and not cause any major headaches, but I wanted to check before doing the work and submitting a PR that I'm not missing something.

[chrispyles]

I think this is a really good idea. It should be possible to do a refactor like this; the main thing is that the new logic would need to make sure that it rebuilds the "base" image if the autograder zip has different requirements. To achieve, this it should be relatively easy to create two images in otter grade, one with only the requirements that's tagged with a hash of the environment.yml file from the autograder zip file, and a second built on top of that image with the tests and configurations loaded in. When otter grade runs, it can hash the environment.yml file, check whether an image with that tag exists, and then it won't need to rebuild the image.

If you'd like to contribute this refactor, I'm happy to review any PRs.