Investigate how to migrate off of Windows Server 2016 build agents

[mthalman]

The build infrastructure currently uses Windows Server build agents to build Dockerfiles. Because we're using Docker's default configuration of process isolation mode, we're restricted in which Dockerfiles can be built by a given Windows Server version. For example, with process isolation you can only build a Dockerfile for Windows Server 2016 on a Windows Server 2016 machine (see Windows container version compatibility).

This is problematic, particularly with Windows Server 2016 because of its age. There is a lack of support for build agent pools that target Windows Server 2016 and so we are left to define/manage our own self-hosted pools.

We should investigate ways to migrate off the use of Windows Server 2016 build agents. Since we'd need to continue building WS 2016 container images, this would require using Hyper-V isolation mode. This may have a performance impact. But even more importantly, there's a container size aspect that needs to be dealt with. It is possible to build all of our 2016 Dockerfiles with Hyper-V isolation except for the 3.5 Dockerfiles (or at least runtime, which is the root of everything). For 3.5, we run into issues with the default container size. Attempting to build the 3.5 runtime Dockerfile will produce this error while executing its RUN instruction:

Not enough storage is available to process this command.

I believe the default size limit for Windows containers is 20GB. This can be set in the Docker config file with:

"storage-opts": [ "size=50GB" ]

I had experimented with this in the past and had run into issues with getting it to honor the setting. That was over a year ago so things may have changed by now or I may just have done things incorrectly.

Some resources on this topic:

• An error occurred while attempting to build Docker image docker/for-win#3884
• https://forums.docker.com/t/cannot-change-default-20g-size-limit-on-windows-docker-host/31879/7

[mthalman]

[Triage]
It would be good to have a solution in place that wouldn't require custom build agents with the storage size setting. That would allow us to migrate to a different agent pool that didn't have this setting, if necessary to mitigate any sort of emergency. In that case, the setting would need to be dynamically applied during the pipeline run.