IMPORTANT: Service interruption due to auth.docker.io API change (03/01/22)

[xopham]

tl;dr ⏳

• As of 03/01/22, Connaisseur is not able to validate notaryv1 signatures due to an unexpected api change of auth.docker.io.
• This also affects the ability to upgrade to a newer version.
• Users receive the following error message: aiohttp.client_exceptions.ContentTypeError: 0, message='Attempt to decode JSON with unexpected mimetype: text/plain; charset=utf-8', url=URL('https://auth.docker.io/token?service=notary.docker.io&scope=repository:docker.io/securesystemsengineering/testimage:pull')
• An emergency fix is provided in fix: disable aiohttp content type check #564 .
• It was released on 03/02/22 with version v2.5.0: Release 2.5.0 #538 .
• To fix, it is necessary to helm uninstall the old version and helm install the new version, as upgrade cannot work when the docker API is non-functional.

Details 📃

Docker apparently made an unexpected change to auth.docker.io/token by which the response content-type header was changed from application/json to text/plain; charset=utf-8. Note that the body itself has not changed and remains proper json. The cause for the change is not known.
During validation via notaryv1 validator, Connaisseur uses the auth.docker.io/token endpoint to get an access token via the asyncio aiohttp method to extract the response json. However, while the received payload is still json, the json coroutine also verifies the content-type header which fails with the api change and a ClientResponseError is raised. This causes any notaryv1 validation to fail including upgrading Connaisseur by which Connaisseur's own signature is verified.
#564 resolves this issue by setting content_type=None, thus disabling content-type verification. However, as making a helm upgrade requires verification of Connaisseur's own signature which will fail for any version prior v2.5.0, it is required to helm uninstall and helm install to install the latest version with the fix.

Problems? 🛠️

This is a very unfortunate situation, but we hope the arising problems can be resolved. Feel free to share any feedback or question that may arise in this thread 🙏