Why does the zot image use the VOLUME instruction?

[polarathene]

I assume this is the Stacker equivalent of declaring VOLUME? (Dockerfile equivalent for reference):

zot/build/stacker.yaml

Lines 75 to 76 in d465690

	volumes:
	- /var/lib/registry

TL;DR: Rarely makes any sense or does any good. This instruction is typically legacy unless you have a valid use-case to warrant it. Given the PRs that introduced both lines, it seems evident that this is not the case.

---

This creates an implicit anonymous volume when creating the container. If you were to destroy the container after stopping it (docker run --rm), the volume would be deleted as well. In the past before other volume types were supported by Docker leveraging VOLUME had some advantages including with performance, but the default storage driver today is much better in that regard AFAIK.

Thus I usually encounter VOLUME as a decision not well understood by the image author. Or they rely on Docker Compose behaviour which differs. Docker Compose will tie the anonymous volume to the associated service.

• That is you can change the image from zot to harbor, and provided both images declare VOLUME to the same path, that anonymous volume will persist to the container using a different image (but still for the same Docker Compose service).
• Even without VOLUME, you can provide an explicit anonymous volume mount in the compose.yaml and again it'll not be a fresh one, it's keyed by path and service, so it'll represent the same data.
• docker compose down and similar commands that destroy the container won't remove the implicit volume. This can be a bit surprising when the user expects an ephemeral image to be immutable and result in a clean slate when there are no explicit persistence configured. That would be the case with Docker CLI, but as mentioned with Docker Compose this behaviour can be a little unexpected.

For reference docker compose down --volumes (destroy the anonymous volumes with containers) or docker compose up --force-recreate --renew-anon-volumes (keeps the old one on disk, attaches a blank anonymous volume to the newly created container) and similar commands can be used to workaround that. Alternative container engines differ, like Podman which has config support to change VOLUME to ephemeral tmpfs mounts or ignore mounting a volume completely.

Rarely should an image need to declare VOLUME, especially if it can result in 1.7GB of data each time one of these are created implicitly (the main zot image which defaults to extensions.search.cve enabling Trivy for vulnerability scanning). If persistence is desired, an explicit volume should be configured.

The main perk of the Docker Compose behaviour is to have persistence implicitly that is attached to the Compose service, so that the volume is reattached when the image is upgraded to a new release. While it's rare that the image would change to something completely different under the same service name and just happen to share the same VOLUME path, it can happen and that may risk leaking sensitive data unintentionally (and unaware to the typical user). An explicit volume mount avoids all that (or rather more clearly communicates the expectation/behaviour that occurs).

---

For reference:

• Dockerfile originally introduced VOLUME in 2020: Dockerfile for running zot server #64
• stacker.yaml originally introduced volume in 2022: Migrate from docker/build-push-action to stacker-build-push-action #499

Neither PR has any discussion pertaining to the inclusion of VOLUME. Hence I'm inclined it was done so without much thought by the author or concern by the reviewers, rather than with intention to solve any requirements.

[andaaron]

I am not sure of the original reason for using a volume in the dockerfile itself, but in general a volume should be used in order to persist the data.
Maybe the original author wanted to make sure a separate volume is used to safeguard against someone using docker container commit or equivalent to save the data in a new image. Or there were some intricacies of copy-on-write filesystems which they wanted to avoid.

In any case I am 99% sure there is no use case to run the image without specifying a volume when starting a container. I don't know where this message is now, it clear in the readme updated in https://github.com/project-zot/zot/pull/64/files#diff-b335630551682c19a781afebcf4d07bf978fb1f8ac04c6bf87428ed5106870f5R68

It's not just the trivy data which is store there, there's also the images (if local storage is used), image metadata (if boltdb is used as opposed to dynamodb/redis), UI sessions, and possibly other data which is escaping me.

[polarathene]

If there was a good reason for it, the PR should have said such. stacker.yaml was likely updated to better reflect the Dockerfile inclusion of VOLUME.

If you are interested in opposing opinions to remove VOLUME, I have raised this concern with other image authors including official DB images, let me know if you'd like links to those for reference.

• Apart from MariaDB (which lacked sufficient information to reproduce a reported failure), there was nothing I recall that really justified it.
• Some image authors felt removing VOLUME would be a breaking change to anyone supposedly relying on it to persist important data 🤷‍♂️
• Others have expressed it as a way to document/communicate volumes expected for the image and disagree with the concerns I raised as edge cases, rather than pragmatic concerns.

[rchincha]

"/volume" here allows you to store your images on the host filesystem and not lose them on container restarts or upgrades.
Nothing more than that in this case.

[polarathene]

not lose them on container restarts or upgrades

• It's an anonymous volume, it's tied to the container lifetime (differs for compose). You can write to the container layer without such a volume and it will persist across container restarts.
• If you want to persist data across container instances use an explicit volume mount.

Regarding upgrades, this is inaccurate. That logic only applies to Docker Compose.

• The anonymous volume is tied to the Compose service name, not the container itself.
• You can provide an explicit anonymous volume to that service and it will represent the same volume data.
• You can change the image to a completely different one (Redis to Valkey) and when they share the same VOLUME path and Compose service name, the same volume will be used.

Note that last point. Should any other image use /var/lib/registry in the same way, swapping the image will get the same data zot had there implicitly. Sensitive data can be leaked if not careful, this is not an image upgrade scenario but change to a completely different image under a common service name (eg: container-registry, like db for Redis vs Valkey).

---

Nothing more than that in this case.

There is more problems with this behaviour than there is value. Containers should be immutable/ephemeral, no implicit hand-holding for persisting state across image upgrades, any data that is desired to be kept and deemed important should be persisted via an explicit volume mount.

[andaaron]

1. How do we ensure the user uses an explicit volume as opposed to no volume at all? Or do we all agree that is not our problem?
2. How do we know the users aren't using Docker Compose?
3. There's no guarantee this is going to be run by Docker. Do we know there aren't any other runtimes in use with a different copy-on-write fs?

It's not clear to me why specifying the implicit volume is worse than not having it.
A. If the user runs the image with an explicit mount, there's no change between the current Dockerfile and the Dockerfile without the implicit volume.
B. If the user falls back to the implicit volume unintentionally, by not using an explicit volume, that's his mistake, and he can always clean the disk space. This seems to be an acceptable approach when put in balance with potentially breaking someone's currently working setup.

My options in order of preference would be

• find a way to force the user to use an explicit volume, as originally intended, is that possible?
  or
• leave things as they are, at least we don't break anything
  or
• remove the volume directive

If this was a new image nobody used, I would incline to agree with removing the volume from the dockerfile.

[polarathene]

TL;DR: Apologies for the verbosity, I lack the time to write something shorter 😓

I think the below response covers your concerns well and justifies why VOLUME is more of a negative than a positive. However, this is rather niche and your time is better spent elsewhere when I'm the only one here discouraging the use of VOLUME.

Feel free to skip below and only refer to it if more users chime in about the concern.

---

How do we ensure the user uses an explicit volume as opposed to no volume at all?

You document what is relevant to persist in the container with a volume mount, which you already do?

Or do we all agree that is not our problem?

Correct. If a user wants to persist data that is deemed important to not lose, they should be explicit about that with containers.

Containers are meant to be predictable, in that starting a fresh container without any explicit persistence should reflect that immutable image state, not implicitly carry state.

This Docker Compose specific behaviour was something I had to troubleshoot with an image that created a DB at an implicit volume for persisting config when I used it's API. When I had learned about using the software through experimentation, I then wanted to do a clean slate with a fresh container instance and configure it again... but after running docker compose up --force-recreate the container was not behaving as expected, it still had the persisted volume. Ok so docker compose down, that'd destroy the container right? Well yes, but not the implicit volume...

With Docker CLI I would use docker run --rm ... and when the container is destroyed the implicit volume is too. docker compose run --rm exists, but there is no equivalent setting in compose.yaml, nor for docker compose up (there is --renew-anon-volumes, but this replaces with a new one while keeping the old on disk), and docker compose down --volumes to destroy all implicit volumes for all services in that compose.yaml. I had to go look through the docs and discover these, they weren't exactly working until late 2024 either IIRC, and as a maintainer of a dockerized project, these kind of issues with getting a clean slate is something I'm quite familiar with users running into even without VOLUME.

You cannot ensure every container will implicitly persist data on behalf of the user. So to cater to that audience without requiring them to double-check (or add an extra line of config to be explicit), are you part of a problem or solution to such a practice? That user is bound to run a container with the expectation it will persist anything important like other containers they've used, and then one day they lose their important data because that container image did not have a VOLUME. Or the user runs some prune command that wipes the implicit volumes that they never gave much thought to understanding due to the convenience they relied on, now what?

VOLUME is an anti-pattern today for the majority of images I see using it IMO.

---

How do we know the users aren't using Docker Compose?

Why does it matter? You don't want to troubleshoot with a user where one of uses compose and the other doesn't, and behind the scenes the problem turns out to be specific to this "feature" affecting the reproduction. Especially if neither party is even aware of this to consider the possibility.

I use Docker Compose all the time and this is not a feature I'm fond of due to the various problems I've encountered from it.

A better way to phrase the question is why do you want to rely on a feature for persistence that won't be consistent since not all users will use Docker Compose?

There's no guarantee this is going to be run by Docker. Do we know there aren't any other runtimes in use with a different copy-on-write fs?

If you can cite one that doesn't persist the container state across a restart of the same container, let me know. Otherwise your reasoning to justify keeping it due to an unknown seems like a poor way to influence decisions.

The truth is, like with many other image authors, it was added without context and there was no known demand or fix being resolved by it, but to remove it now elicits caution.

The anonymous volume remains and can be "restored" by adding an anonymous volume explicitly in compose.yaml for that service. It just needs the same path. The only reason you wouldn't be able to restore it is if there was some manual or automated pruning that removed it before the user realized the change after upgrading the image (which some have automated instead of checking a changelog).

---

It's not clear to me why specifying the implicit volume is worse than not having it.

Please try run a bunch of containers without volume mounts that create anonymous volumes this way, then identify which one belongs to which container via the docker CLI alone. There's no identifying information last I checked if there is no longer a container running it.

Usually I personally use docker run --rm to avoid this problem (if I didn't, then each run would create a 1.7GB volume on disk with zot when it pulls down trivy), and when I use compose.yaml I add explicit volumes. Sometimes though the volumes declared implicitly aren't documented, or I don't actually want to persist them.

If there weren't implicit volumes, I'd have a predictable container that I could get a clean slate from without having to know the different options per command to opt-out / remove them. Or I could move to Podman which has a feature to use tmpfs or ignore any mount.

If you want to value a user from losing data, consider that the volume accumulation could occur in the background unbeknownst to the user, and if they're low on disk space should this exhaust it before they can act they'll have a much worse situation.

But similar to the leaking of sensitive data from one image to another through the implicit volume behaviour in Compose, these are edge cases. None which exist when you just avoid VOLUME all together.

---

This seems to be an acceptable approach when put in balance with potentially breaking someone's currently working setup.

The only users that you would be breaking are hypothetical. Do you rely on this feature or know of anyone who does? It's often brought up to me as a reason to keep VOLUME, but never with a known audience for it, just an assumed one despite it never having been requested?

You cannot unset VOLUME to my knowledge, but you can extend an image to add VOLUME afterwards. I can cite various issues of users requesting to remove VOLUME, so I know there is demand for that in general. It's the image maintainers that are usually against it, often without many valid reasoning.

I can say as someone who has maintained a Docker oriented project for the past 4-5 years that not having VOLUME in that project image has never been a problem. Users have not requested it, they use explicit volume mounts that we document for persistence. We have users with Podman, Compose, Kubernetes, etc.

As mentioned above, you won't be destroying the anonymous volume by dropping it from the image. So it'll only be breaking if the user does something to remove it on their end afterwards, potentially by accident but honestly if they value persisting data they should be using an explicit mount... and until I'm made aware of any actual user relying on this intentionally, they are hypothetical users only.

My options in order of preference would be
find a way to force the user to use an explicit volume, as originally intended, is that possible?

If the path does not exist in the image when the container starts you can check the path, you can then either communicate via a log message or exit from the lack of a mount to add persistence.

That kind of enforcement would require running from Docker CLI to then have docker run --rm --volume /var/lib/registry ... or docker run --workdir /var/lib/registry.

I don't think that's necessary to enforce.

• A user should not assume persistence by default when using containers, they'd hopefully learn that early on or learn the hard way.
• Documentation should communicate using an explicit volume mount rather clearly, as-such most users would be aware of what to configure should they want to persist.
• There are valid examples (such as reproductions) where you don't want to persist with a volume and explicitly adding one just to make a container run can be a friction point.

leave things as they are, at least we don't break anything

Up to you. This was more of a PSA about the issues VOLUME can pose. I am also curious to insights on why a project chooses to use this instruction since I consider it an anti-pattern.

In the case of zot it seems it slipped in through review without much consideration/discussion, and now a reluctance to remove it because of the unknown potential that you have users relying on it for production deployments (which is why as a fellow maintainer I'm not fond of such a justification for decision making). This is usually how VOLUME gets introduced.

[rchincha]

This sounds like it begs a PR ... pls submit one.

[polarathene]

It's a fairly simple change to make.

I'll submit a PR (EDIT: done), but if you'd rather defer such a change until there's more users expressing a desire for it that's fine too.