This repo is dedicated to mirror images from other organizations into Rancher. There are no packaging changes or changes in the layers of these images.
Important
This repository is currently undergoing an overhaul in order to meet the
changing requirements of the Rancher project. The bespoke scripts that
are used to mirror images are being replaced by
regsync.
Generation of regsync config, among other automations, is being written in Go.
As this migration takes place, no disruption to regular image mirroring
will occur - you may continue to use the old tooling while the new
tooling is built. Documentation for the new setup is under New Documentation.
Documentation for the old setup is under Old Documentation.
regsync is used for mirroring images from one repository to another. regsync
is configured via regsync.yaml. regsync.yaml is generated by running the
generate-regsync subcommand of the Go code in tools/, using config.yaml as
input. You can build the latest version of this code to bin/image-mirror-tools
using scripts/build-tools.sh. Thus, your workflow might look like this:
scripts/build-tools.sh
bin/image-mirror-tools generate-regsync
Once regsync.yaml has been updated, you may run regsync via the command
regsync once --verbosity error --config regsync.yaml --missing
Repositories describes the repositories that image-mirror interfaces with.
This section roughly correlates to the creds section of regsync.yaml.
| Field | Required | Description |
|---|---|---|
BaseUrl |
yes | The base URL for the repository. Appending / plus an image name should be a valid image reference. |
Password |
yes | The password to use when authenticating against the registry. See the regsync documentation for more details. |
Registry |
yes | The registry URL. See the regsync documentation for more details. |
ReqConcurrent |
no | The number of concurrent requests that are made to this registry. See the regsync documentation for more details. |
Target |
no | When true, denotes a target repository. This means that all images will be mirrored to this repository. |
Username |
yes | The username to use when authenticating against the registry. See the regsync documentation for more details. |
Images describes the images that we want to mirror to each target
repository.
| Field | Required | Description |
|---|---|---|
DoNotMirror |
no | Set to true to exclude the entire image from regsync.yaml. Alternatively, set to an array of strings to specify tags to exclude from regsync.yaml. |
SourceImage |
yes | The source image. If there is no host, the image is assumed to be from Docker Hub. |
Tags |
yes | The tags to mirror. |
TargetImageName |
no | By default, the target image name is derived from the source image, and is of the format mirrored-<org>-<name>. For example, banzaicloud/logging-operator becomes mirrored-banzaicloud-logging-operator. However, there are some images that do not follow this convention - this field exists for these cases. New images should not set this field. |
autoupdate.yaml defines configuration for automatically updating image tags
based on various update strategies that monitor sources for new tags. Each
entry specifies a strategy for finding tags of images to potentially add to
config.yaml, which are then submitted as pull requests.
| Field | Required | Description |
|---|---|---|
Name |
yes | A unique identifier for this autoupdate entry. Used for logging and generating branch names for pull requests. |
GithubRelease |
no | See GithubRelease. |
HelmLatest |
no | See HelmLatest. |
Registry |
no | See Registry. |
Reviewers |
no | A list of GitHub users or teams to request review from on created pull requests. Teams should be in the format org/team-slug. |
The GithubRelease strategy fetches all release tags that matches the VersionConstraint from a GitHub
repository and applies it to the specified images.
If LatestOnly is true, it only fetches from the latest release and does not consider the VersionConstraint.
| Field | Required | Description |
|---|---|---|
Owner |
yes | The GitHub repository owner/organization. |
Repository |
yes | The GitHub repository name. |
Images |
yes | See Images. |
LatestOnly |
no | A flag to only fetch the latest github release. |
VersionConstraint |
no | A SemVer constraint used to filter the github releases. |
A list of images to be updated with the latest release tag. Each image will get the same tag as the GitHub release.
| Field | Required | Description |
|---|---|---|
SourceImage |
yes | The GitHub repository name. |
TargetImageName |
no | The TargetImageName of the image in config.yaml that you want to update. |
The HelmLatest strategy templates out the latest version of configured
Helm charts and extracts image references from the rendered manifests. It
recursively searches for fields with an "image" key in the templated YAML
output.
| Field | Required | Description |
|---|---|---|
HelmRepo |
yes | The URL of the Helm chart repository. |
Charts |
yes | A map where keys are the charts to template, and values are another map from environment name to lists of helm values to --set in that environment. helm template is run once for each environment. |
Images |
no | Used to map a given update image to an entry in config.yaml. There may be multiple entries that have the same SourceImage, but different TargetImageNames, so we need to choose which one receives the update image. |
ImageDenylist |
no | A list of images to exclude from the results. |
The Registry strategy fetches all image tags that matches the VersionFilter from a registry defined in the Images provided.
Supported registries are:
- Suse Container Registry (registry.suse.com)
- Docker Hub
- Quay.io
- K8s registry (registry.k8s.io)
- GitHub Container Registry (ghcr.io)
- Google Container Registry (gcr.io)
| Field | Required | Description |
|---|---|---|
Images |
yes | Used to map a given update image to an entry in config.yaml. There may be multiple entries that have the same SourceImage, but different TargetImageNames, so we need to choose which one receives the update image. |
Latest |
no | A flag to only use the latest tag. This only works if all tags are in semver format. |
VersionFilter |
no | A regex to match against the image tags fetched from the registry. |
regsync.yaml is mostly for use by regsync. It is generated from config.yaml,
and is not very easy to read. It should never be modified directly. It can,
however, be useful for checking that config.yaml changes will have the expected
effect on mirroring.
The following documentation describes the old way of mirroring images. It pertains to files like:
images-listimages-list-dailyretrieve-image-tags/Dockerfile.dapperentrypoint.shscripts/
The list is maintained in the images-list file, which is structured with the following format...
<original-image-name> <rancher-image-name> <image-tag>
The basic rancher-image-name structure is mirrored-<org>-<repo> and here is an example...
banzaicloud/logging-operator rancher/mirrored-banzaicloud-logging-operator 3.7.0
Images are mirrored using the scripts/image-mirror.sh script.
When adding new images to the repo, please indicate so in the pull request.
An EIO team member or manager will need to create the repo in DockerHub as well as add the automatedcipublisher as a team member in DockerHub with write access in order for the images to be automatically pushed.
Do not update the tag in the images-list file for an updated image to be pulled/pushed. Add an additional entry with the new tag.
You can use the following commands/scripts to add a tag to an existing image. Make sure the IMAGES environment variable is set to the image(s) you want to add a tag to, and the TAGS environment variable is set to the tags you want to add to the images. The script will check:
- If the image provided is already existing, else it will fail because it only supports adding tags to existing images.
- If there is only one mapping in
images-list, else it will fail because it cannot determine what mapping to use. - If the tag for the image is not already present, else it will fail because it is not new.
- If the tag for the image exists, else it will fail as it cannot be mirrored.
After everything is successfull, it will add the tag to images-list. If all images and tags are added, it will sort images-list.
See an example below:
IMAGES=quay.io/coreos/etcd TAGS=v3.4.20 make add-tag-to-existing-image.sh
There is also a wrapper script to support supplying images with tags. This was added to support the helm-latest version source which extracts images from Helm charts and does not work with the images + tags inputs. The wrapper script for full images can be used as follows:
FULL_IMAGES=quay.io/skopeo/stable:v1.13.3,quay.io/cilium/cilium-envoy:v1.25.9-e198a2824d309024cb91fb6a984445e73033291d make add-full-image-wrapper.sh
The wrapper script will run the add-tag-to-existing-image.sh script for each image, to be aligned with all the checks that are required.
Optionally, you can also check if the newly added image tag exists (this will also be run in GitHub Action):
make check-new-images-exist.sh
You can use the Add tag to existing image workflow to provide a comma separated list of existing images and to be added tags, and it will create a pull request automatically with the changes. See Using scripts what this does in detail.
Example inputs:
Images: quay.io/cilium/cilium,quay.io/cilium/operator-aws,quay.io/cilium/operator-azure,quay.io/cilium/operator-generic
Tags: v1.12.1
or
Full Images: quay.io/skopeo/stable:v1.13.3,quay.io/cilium/cilium-envoy:v1.25.9-e198a2824d309024cb91fb6a984445e73033291d