systemd files for kube

[Klaas-]

Hi,
I've been wondering if it would be a good idea to use the systemd files that podman generates instead of the file you're currently templating. For example right now it won't notice which processes belong to the service you're running, with generated files systemd can keep track of them.
You've made changes to the single container service in bb16d8e but that does not help with awx and the kube play :)

I've been trying to figure out how to properly get this into a role with idempotency but so far I am still missing a good idea :) I guess I need to ask podman devs to create another output format and create a podman kube module in https://github.com/containers/ansible-podman-collections :)

Greetings
Klaas

[ikke-t]

What piece of idempotent is it missing? If there is something, we need to add notify sections.

TBH, I haven't used the multi container stuff in a while. It's only in AWX that I've used it, and I believe it's good enough for it. It depends a bit if people use it or not. If not, who cares. If someone does, perhaps (s)he raises or fixes the shortcomings of it.

What is the problem at the moment you see with multi container one? Is it systemd not restarting the failed containers or something else?

[Klaas-]

the output of podman generate systemd --name awx will just print the contents of 6 files concatenated. I would have to generate a tmp dir and use --files to output the files properly and then check against contents of /etc/systemd/system, this seems like a lot of work :D if I could output it to json and then read stdout in python it would be easier to put it into a ansible module.

The reason why I like the generated files better: they take care of tracking the processes instead of just starting/stopping and the way kube plays are run atm is creating a new container id each start instead of using start/stop with the same container ids (not actually sure if this makes a big difference)

systemctl status awx-container-pod.service
● awx-container-pod.service - awx Podman Container
   Loaded: loaded (/etc/systemd/system/awx-container-pod.service; enabled; vendor preset: disabled)
   Active: active (exited) since Mon 2020-04-06 15:53:26 CEST; 1 day 3h ago
  Process: 26618 ExecStop=/usr/bin/podman pod rm -f awx (code=exited, status=0/SUCCESS)
  Process: 26913 ExecStart=/usr/bin/podman play kube /etc/containers/pods/awx.yaml (code=exited, status=0/SUCCESS)
  Process: 26876 ExecStartPre=/usr/bin/podman pod rm -f awx (code=exited, status=125)

whereas generated systemd files:

[root@domain systemd]# systemctl status pod-awx container-awx_*
● pod-awx.service - Podman pod-awx.service
   Loaded: loaded (/etc/systemd/system/pod-awx.service; disabled; vendor preset: disabled)
   Active: active (running) since Tue 2020-04-07 19:29:45 CEST; 29s ago
     Docs: man:podman-generate-systemd(1)
  Process: 13828 ExecStart=/usr/bin/podman start bc626f97e7b7-infra (code=exited, status=0/SUCCESS)
 Main PID: 13937 (conmon)
    Tasks: 0
   Memory: 3.3M
   CGroup: /system.slice/pod-awx.service
           ‣ 13937 /usr/bin/conmon --api-version 1 -s -c 34373ed08ee2f8de29d7c3d5deb3a0be43c8f46b291e8b0c6704cb0619526395 -u 34373ed08ee2f8de29d7c3d5deb3a0be43c8f46b291e8b0c6704cb0619526395 -r /usr/bin/runc -b /var/lib/containers/storage/overlay-containers/34373ed08ee2f8de29d7c3d5deb3a0be43c8f46b291e8b0c6704cb0619526395/userdata -p /var/run/containers/storage/overlay-containers/34373ed08ee2f8de29d7c3d5deb3a0be43c8f46b291e8b0c6704cb0619526395/userdata/pidfile -l k8s-file:/var/lib/containers/storage/overlay-containers/34373ed08ee2f8de29d7c3d5deb3a0be43c8f46b291e8b0c6704cb0619526395/userdata/ctr.log --exit-dir /var/run/libpod/exits --socket-dir-path /var/run/libpod/socket --log-level error --runtime-arg --log-format=json --runtime-arg --log --runtime-arg=/var/run/containers/storage/overlay-containers/34373ed08ee2f8de29d7c3d5deb3a0be43c8f46b291e8b0c6704cb0619526395/userdata/oci-log --conmon-pidfile /var/run/containers/storage/overlay-containers/34373ed08ee2f8de29d7c3d5deb3a0be43c8f46b291e8b0c6704cb0619526395/userdata/conmon.pid

Apr 07 19:29:45 domain.tld systemd[1]: Starting Podman pod-awx.service...
Apr 07 19:29:45 domain.tld podman[13828]: 2020-04-07 19:29:45.881962288 +0200 CEST m=+0.411595999 container init 34373ed08ee2f8de29d7c3d5deb3a0be43c8f46b291e8b0c6704cb0619526395 (image=k8s.gcr.io/pause:3.1, name=bc626f97e7b7-infra)
Apr 07 19:29:45 domain.tld podman[13828]: 2020-04-07 19:29:45.892228072 +0200 CEST m=+0.421861783 container start 34373ed08ee2f8de29d7c3d5deb3a0be43c8f46b291e8b0c6704cb0619526395 (image=k8s.gcr.io/pause:3.1, name=bc626f97e7b7-infra)
Apr 07 19:29:45 domain.tld podman[13828]: bc626f97e7b7-infra
Apr 07 19:29:45 domain.tld systemd[1]: Started Podman pod-awx.service.

● container-awx_awxtask.service - Podman container-awx_awxtask.service
   Loaded: loaded (/etc/systemd/system/container-awx_awxtask.service; disabled; vendor preset: disabled)
   Active: active (running) since Tue 2020-04-07 19:29:46 CEST; 29s ago
     Docs: man:podman-generate-systemd(1)
  Process: 13967 ExecStart=/usr/bin/podman start awx_awxtask (code=exited, status=0/SUCCESS)
 Main PID: 14067 (conmon)
    Tasks: 0
   Memory: 1.9M
   CGroup: /system.slice/container-awx_awxtask.service
           ‣ 14067 /usr/bin/conmon --api-version 1 -s -c 582c8aba22d1f504aa48d61142a65bc49cbfe0ae2abe032286c2dce71c4ad2ac -u 582c8aba22d1f504aa48d61142a65bc49cbfe0ae2abe032286c2dce71c4ad2ac -r /usr/bin/runc -b /var/lib/containers/storage/overlay-containers/582c8aba22d1f504aa48d61142a65bc49cbfe0ae2abe032286c2dce71c4ad2ac/userdata -p /var/run/containers/storage/overlay-containers/582c8aba22d1f504aa48d61142a65bc49cbfe0ae2abe032286c2dce71c4ad2ac/userdata/pidfile -l k8s-file:/var/lib/containers/storage/overlay-containers/582c8aba22d1f504aa48d61142a65bc49cbfe0ae2abe032286c2dce71c4ad2ac/userdata/ctr.log --exit-dir /var/run/libpod/exits --socket-dir-path /var/run/libpod/socket --log-level error --runtime-arg --log-format=json --runtime-arg --log --runtime-arg=/var/run/containers/storage/overlay-containers/582c8aba22d1f504aa48d61142a65bc49cbfe0ae2abe032286c2dce71c4ad2ac/userdata/oci-log --conmon-pidfile /var/run/containers/storage/overlay-containers/582c8aba22d1f504aa48d61142a65bc49cbfe0ae2abe032286c2dce71c4ad2ac/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/lib/containers/storage --exit-command-arg --runroot --exit-command-arg /var/run/containers/storage --exit-command-arg --log-level --exit-command-arg error --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /var/run/libpod --exit-command-arg --runtime --exit-command-arg runc --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg container --exit-command-arg cleanup --exit-command-arg 582c8aba22d1f504aa48d61142a65bc49cbfe0ae2abe032286c2dce71c4ad2ac

Apr 07 19:29:45 domain.tld systemd[1]: Starting Podman container-awx_awxtask.service...
Apr 07 19:29:46 domain.tld podman[13967]: 2020-04-07 19:29:46.545255041 +0200 CEST m=+0.615299027 container init 582c8aba22d1f504aa48d61142a65bc49cbfe0ae2abe032286c2dce71c4ad2ac (image=docker.io/ansible/awx_task:10.0.0, name=awx_awxtask)
Apr 07 19:29:46 domain.tld podman[13967]: 2020-04-07 19:29:46.558622544 +0200 CEST m=+0.628666559 container start 582c8aba22d1f504aa48d61142a65bc49cbfe0ae2abe032286c2dce71c4ad2ac (image=docker.io/ansible/awx_task:10.0.0, name=awx_awxtask)
Apr 07 19:29:46 domain.tld podman[13967]: awx_awxtask
Apr 07 19:29:46 domain.tld systemd[1]: Started Podman container-awx_awxtask.service.

● container-awx_awxweb.service - Podman container-awx_awxweb.service
   Loaded: loaded (/etc/systemd/system/container-awx_awxweb.service; disabled; vendor preset: disabled)
   Active: active (running) since Tue 2020-04-07 19:29:46 CEST; 29s ago
     Docs: man:podman-generate-systemd(1)
  Process: 13963 ExecStart=/usr/bin/podman start awx_awxweb (code=exited, status=0/SUCCESS)
 Main PID: 14089 (conmon)
    Tasks: 0
   Memory: 2.1M
   CGroup: /system.slice/container-awx_awxweb.service
           ‣ 14089 /usr/bin/conmon --api-version 1 -s -c 7dbb3c33a6a2761e2d0e178058e7d836d939c9526c0778db8661423aad325769 -u 7dbb3c33a6a2761e2d0e178058e7d836d939c9526c0778db8661423aad325769 -r /usr/bin/runc -b /var/lib/containers/storage/overlay-containers/7dbb3c33a6a2761e2d0e178058e7d836d939c9526c0778db8661423aad325769/userdata -p /var/run/containers/storage/overlay-containers/7dbb3c33a6a2761e2d0e178058e7d836d939c9526c0778db8661423aad325769/userdata/pidfile -l k8s-file:/var/lib/containers/storage/overlay-containers/7dbb3c33a6a2761e2d0e178058e7d836d939c9526c0778db8661423aad325769/userdata/ctr.log --exit-dir /var/run/libpod/exits --socket-dir-path /var/run/libpod/socket --log-level error --runtime-arg --log-format=json --runtime-arg --log --runtime-arg=/var/run/containers/storage/overlay-containers/7dbb3c33a6a2761e2d0e178058e7d836d939c9526c0778db8661423aad325769/userdata/oci-log --conmon-pidfile /var/run/containers/storage/overlay-containers/7dbb3c33a6a2761e2d0e178058e7d836d939c9526c0778db8661423aad325769/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/lib/containers/storage --exit-command-arg --runroot --exit-command-arg /var/run/containers/storage --exit-command-arg --log-level --exit-command-arg error --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /var/run/libpod --exit-command-arg --runtime --exit-command-arg runc --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg container --exit-command-arg cleanup --exit-command-arg 7dbb3c33a6a2761e2d0e178058e7d836d939c9526c0778db8661423aad325769

Apr 07 19:29:45 domain.tld systemd[1]: Starting Podman container-awx_awxweb.service...
Apr 07 19:29:46 domain.tld podman[13963]: 2020-04-07 19:29:46.492425359 +0200 CEST m=+0.566658931 container init 7dbb3c33a6a2761e2d0e178058e7d836d939c9526c0778db8661423aad325769 (image=docker.io/ansible/awx_web:10.0.0, name=awx_awxweb)
Apr 07 19:29:46 domain.tld podman[13963]: 2020-04-07 19:29:46.502676826 +0200 CEST m=+0.576910360 container start 7dbb3c33a6a2761e2d0e178058e7d836d939c9526c0778db8661423aad325769 (image=docker.io/ansible/awx_web:10.0.0, name=awx_awxweb)
Apr 07 19:29:46 domain.tld podman[13963]: awx_awxweb
Apr 07 19:29:46 domain.tld systemd[1]: Started Podman container-awx_awxweb.service.

● container-awx_memcached.service - Podman container-awx_memcached.service
   Loaded: loaded (/etc/systemd/system/container-awx_memcached.service; disabled; vendor preset: disabled)
   Active: active (running) since Tue 2020-04-07 19:29:46 CEST; 29s ago
     Docs: man:podman-generate-systemd(1)
  Process: 13964 ExecStart=/usr/bin/podman start awx_memcached (code=exited, status=0/SUCCESS)
 Main PID: 14041 (conmon)
    Tasks: 0
   Memory: 2.0M
   CGroup: /system.slice/container-awx_memcached.service
           ‣ 14041 /usr/bin/conmon --api-version 1 -s -c a6277326c03c2273d6e3ef87c6f736a27b9447b98fd23450d0db0f1c8e543283 -u a6277326c03c2273d6e3ef87c6f736a27b9447b98fd23450d0db0f1c8e543283 -r /usr/bin/runc -b /var/lib/containers/storage/overlay-containers/a6277326c03c2273d6e3ef87c6f736a27b9447b98fd23450d0db0f1c8e543283/userdata -p /var/run/containers/storage/overlay-containers/a6277326c03c2273d6e3ef87c6f736a27b9447b98fd23450d0db0f1c8e543283/userdata/pidfile -l k8s-file:/var/lib/containers/storage/overlay-containers/a6277326c03c2273d6e3ef87c6f736a27b9447b98fd23450d0db0f1c8e543283/userdata/ctr.log --exit-dir /var/run/libpod/exits --socket-dir-path /var/run/libpod/socket --log-level error --runtime-arg --log-format=json --runtime-arg --log --runtime-arg=/var/run/containers/storage/overlay-containers/a6277326c03c2273d6e3ef87c6f736a27b9447b98fd23450d0db0f1c8e543283/userdata/oci-log --conmon-pidfile /var/run/containers/storage/overlay-containers/a6277326c03c2273d6e3ef87c6f736a27b9447b98fd23450d0db0f1c8e543283/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/lib/containers/storage --exit-command-arg --runroot --exit-command-arg /var/run/containers/storage --exit-command-arg --log-level --exit-command-arg error --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /var/run/libpod --exit-command-arg --runtime --exit-command-arg runc --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg container --exit-command-arg cleanup --exit-command-arg a6277326c03c2273d6e3ef87c6f736a27b9447b98fd23450d0db0f1c8e543283

Apr 07 19:29:45 domain.tld systemd[1]: Starting Podman container-awx_memcached.service...
Apr 07 19:29:46 domain.tld podman[13964]: 2020-04-07 19:29:46.450209648 +0200 CEST m=+0.520525934 container init a6277326c03c2273d6e3ef87c6f736a27b9447b98fd23450d0db0f1c8e543283 (image=docker.io/library/memcached:alpine, name=awx_memcached)
Apr 07 19:29:46 domain.tld podman[13964]: 2020-04-07 19:29:46.465906279 +0200 CEST m=+0.536222577 container start a6277326c03c2273d6e3ef87c6f736a27b9447b98fd23450d0db0f1c8e543283 (image=docker.io/library/memcached:alpine, name=awx_memcached)
Apr 07 19:29:46 domain.tld podman[13964]: awx_memcached
Apr 07 19:29:46 domain.tld systemd[1]: Started Podman container-awx_memcached.service.

● container-awx_postgres.service - Podman container-awx_postgres.service
   Loaded: loaded (/etc/systemd/system/container-awx_postgres.service; disabled; vendor preset: disabled)
   Active: active (running) since Tue 2020-04-07 19:29:46 CEST; 29s ago
     Docs: man:podman-generate-systemd(1)
  Process: 13965 ExecStart=/usr/bin/podman start awx_postgres (code=exited, status=0/SUCCESS)
 Main PID: 14111 (conmon)
    Tasks: 0
   Memory: 2.0M
   CGroup: /system.slice/container-awx_postgres.service
           ‣ 14111 /usr/bin/conmon --api-version 1 -s -c 0bf9dd176c8024e4f8d3aca5cf2fbfeef7abd89129f3e23d505518f0568921f7 -u 0bf9dd176c8024e4f8d3aca5cf2fbfeef7abd89129f3e23d505518f0568921f7 -r /usr/bin/runc -b /var/lib/containers/storage/overlay-containers/0bf9dd176c8024e4f8d3aca5cf2fbfeef7abd89129f3e23d505518f0568921f7/userdata -p /var/run/containers/storage/overlay-containers/0bf9dd176c8024e4f8d3aca5cf2fbfeef7abd89129f3e23d505518f0568921f7/userdata/pidfile -l k8s-file:/var/lib/containers/storage/overlay-containers/0bf9dd176c8024e4f8d3aca5cf2fbfeef7abd89129f3e23d505518f0568921f7/userdata/ctr.log --exit-dir /var/run/libpod/exits --socket-dir-path /var/run/libpod/socket --log-level error --runtime-arg --log-format=json --runtime-arg --log --runtime-arg=/var/run/containers/storage/overlay-containers/0bf9dd176c8024e4f8d3aca5cf2fbfeef7abd89129f3e23d505518f0568921f7/userdata/oci-log --conmon-pidfile /var/run/containers/storage/overlay-containers/0bf9dd176c8024e4f8d3aca5cf2fbfeef7abd89129f3e23d505518f0568921f7/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/lib/containers/storage --exit-command-arg --runroot --exit-command-arg /var/run/containers/storage --exit-command-arg --log-level --exit-command-arg error --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /var/run/libpod --exit-command-arg --runtime --exit-command-arg runc --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg container --exit-command-arg cleanup --exit-command-arg 0bf9dd176c8024e4f8d3aca5cf2fbfeef7abd89129f3e23d505518f0568921f7

Apr 07 19:29:45 domain.tld systemd[1]: Starting Podman container-awx_postgres.service...
Apr 07 19:29:46 domain.tld podman[13965]: 2020-04-07 19:29:46.548383634 +0200 CEST m=+0.619446520 container init 0bf9dd176c8024e4f8d3aca5cf2fbfeef7abd89129f3e23d505518f0568921f7 (image=docker.io/centos/postgresql-10-centos7, name=awx_postgres)
Apr 07 19:29:46 domain.tld podman[13965]: 2020-04-07 19:29:46.609195762 +0200 CEST m=+0.680258606 container start 0bf9dd176c8024e4f8d3aca5cf2fbfeef7abd89129f3e23d505518f0568921f7 (image=docker.io/centos/postgresql-10-centos7, name=awx_postgres)
Apr 07 19:29:46 domain.tld podman[13965]: awx_postgres
Apr 07 19:29:46 domain.tld systemd[1]: Started Podman container-awx_postgres.service.

● container-awx_redis.service - Podman container-awx_redis.service
   Loaded: loaded (/etc/systemd/system/container-awx_redis.service; disabled; vendor preset: disabled)
   Active: active (running) since Tue 2020-04-07 19:29:46 CEST; 29s ago
     Docs: man:podman-generate-systemd(1)
  Process: 13966 ExecStart=/usr/bin/podman start awx_redis (code=exited, status=0/SUCCESS)
 Main PID: 14019 (conmon)
    Tasks: 0
   Memory: 2.0M
   CGroup: /system.slice/container-awx_redis.service
           ‣ 14019 /usr/bin/conmon --api-version 1 -s -c 8e87c55d4e1d4c2b1b94667e66ce10450f2ffd7df689e3262b4de239b9d02718 -u 8e87c55d4e1d4c2b1b94667e66ce10450f2ffd7df689e3262b4de239b9d02718 -r /usr/bin/runc -b /var/lib/containers/storage/overlay-containers/8e87c55d4e1d4c2b1b94667e66ce10450f2ffd7df689e3262b4de239b9d02718/userdata -p /var/run/containers/storage/overlay-containers/8e87c55d4e1d4c2b1b94667e66ce10450f2ffd7df689e3262b4de239b9d02718/userdata/pidfile -l k8s-file:/var/lib/containers/storage/overlay-containers/8e87c55d4e1d4c2b1b94667e66ce10450f2ffd7df689e3262b4de239b9d02718/userdata/ctr.log --exit-dir /var/run/libpod/exits --socket-dir-path /var/run/libpod/socket --log-level error --runtime-arg --log-format=json --runtime-arg --log --runtime-arg=/var/run/containers/storage/overlay-containers/8e87c55d4e1d4c2b1b94667e66ce10450f2ffd7df689e3262b4de239b9d02718/userdata/oci-log --conmon-pidfile /var/run/containers/storage/overlay-containers/8e87c55d4e1d4c2b1b94667e66ce10450f2ffd7df689e3262b4de239b9d02718/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/lib/containers/storage --exit-command-arg --runroot --exit-command-arg /var/run/containers/storage --exit-command-arg --log-level --exit-command-arg error --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /var/run/libpod --exit-command-arg --runtime --exit-command-arg runc --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg container --exit-command-arg cleanup --exit-command-arg 8e87c55d4e1d4c2b1b94667e66ce10450f2ffd7df689e3262b4de239b9d02718

Apr 07 19:29:45 domain.tld systemd[1]: Starting Podman container-awx_redis.service...
Apr 07 19:29:46 domain.tld podman[13966]: 2020-04-07 19:29:46.419956869 +0200 CEST m=+0.494303229 container init 8e87c55d4e1d4c2b1b94667e66ce10450f2ffd7df689e3262b4de239b9d02718 (image=docker.io/library/redis:latest, name=awx_redis)
Apr 07 19:29:46 domain.tld podman[13966]: 2020-04-07 19:29:46.435194756 +0200 CEST m=+0.509541082 container start 8e87c55d4e1d4c2b1b94667e66ce10450f2ffd7df689e3262b4de239b9d02718 (image=docker.io/library/redis:latest, name=awx_redis)
Apr 07 19:29:46 domain.tld podman[13966]: awx_redis
Apr 07 19:29:46 domain.tld systemd[1]: Started Podman container-awx_redis.service.

[ikke-t]

oh, wau, I learned again something new. I didn't know that, but it would be sure good enhancement.

BTW, I never fixed the image update part either. At the time podman didn't return information if images updated or not during the pull, perhaps that has changed. Perhaps you could take a look at that too while digging into this?

[tabic]

kube plays are creating a new container id each start instead of using start/stop with the same container ids
not actually sure if this makes a big difference

In the end there is no difference because containers should not have any state. I think deleting and re-creating a container is better (instead of starting a stopped container) because anything changed manually will be undone. You should not change things inside a container, but some people tend to do it anyways.

I do not like to write kube files for each pod and therefore opted to use systemd files. I wanted to define pods with simple variables. Since this would be a quite huge and breaking change to this role I created my own role: https://github.com/rxcket/ansible-role-podman_systemd#example-pod-variable

@ikke-t if you are interested in merging both roles please let me know. Creating the role was a bit cumbersome especially with keeping duplicated tasks as low as possible.

[ikke-t]

Thanks for info, let's see if I need such. ATM I only run a single container in my pods.