ppc64le machines from fedora infra seem to be unstable

[nikromen]

Everything comes from the same pool - the one set one machines we;ve got from fedora infra after the migration:

backend.log.gz
backend1.log.gz
backend2.log.gz
builder-live.log.gz
builder-live1.log.gz
builder-live2.log.gz

[praiskup]

The first two seem like our hypervisor p09 has some issues.

The third one seems to be #3826 (and should be arch agnostic).

...
File /var/lib/copr-rpmbuild/logger_pid doesn't exist, yet
File /var/lib/copr-rpmbuild/logger_pid doesn't exist, yet
File /var/lib/copr-rpmbuild/logger_pid doesn't exist, yet
File /var/lib/copr-rpmbuild/logger_pid doesn't exist, yet
File /var/lib/copr-rpmbuild/logger_pid doesn't exist, yet

Warning: Permanently added '2620:52:3:1:dead:beef:cafe:c289' (ED25519) to the list of known hosts.
Unexpected exception
Traceback (most recent call last):
  File "/usr/bin/copr-rpmbuild-cancel", line 86, in <module>
    _main()
    ~~~~~^^
  File "/usr/bin/copr-rpmbuild-cancel", line 69, in _main
    with open(pidfile, "r") as pidfd:
         ~~~~^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/copr-rpmbuild/pid'

[praiskup]

Not strictly related to hypervisors:

1. ticket_id=4620801, hostname=2620:52:3:1:dead:beef:cafe:c29f, name=copr_p09_01_prod_06335056_20250820_013413
2. ticket_id=4544126, hostname=140.211.168.240, name=copr_osuosl_p09_prod_06272818_20250816_051026
3. ticket_id=4469727, hostname=2620:52:3:1:dead:beef:cafe:c289, name=copr_p09_01_prod_06233283_20250814_063102

[praiskup]

The first one:

=== /var/log/resallocserver/hooks/6335056_watch ===
...
++ echo '---
name: copr_p09_01_prod_06335056_20250820_013413
host: 2620:52:3:1:dead:beef:cafe:c29f'
++ yq .host
+ IP=2620:52:3:1:dead:beef:cafe:c29f
+ test -z 2620:52:3:1:dead:beef:cafe:c29f
+ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ConnectTimeout=10 root@2620:52:3:1:dead:beef:cafe:c29f true
ssh: connect to host 2620:52:3:1:dead:beef:cafe:c29f port 22: Connection timed out^M

It actually attempted to release.

The second one was being released.

ERROR 2025-08-20 07:17:57,422 PID:903186:140127721920192(Releaser-6335056) Releasing worker failed: pool=copr_p09_01_prod name=copr_p09_01_prod_06335056_20250820_013413 cmd=/var/lib/resallocserver/resalloc_provision/vm-release
 INFO 2025-08-20 07:17:57,431 PID:903186:140127721920192(Releaser-6335056) Releasing copr_p09_01_prod_06335056_20250820_013413 finished
ERROR 2025-08-20 07:17:57,431 PID:903186:140127183558336(Releaser-6335586) Releasing worker failed: pool=copr_p09_01_prod name=copr_p09_01_prod_06335586_20250820_021913 cmd=/var/lib/resallocserver/resalloc_provision/vm-release
 INFO 2025-08-20 07:17:57,440 PID:903186:140127183558336(Releaser-6335586) Releasing copr_p09_01_prod_06335586_20250820_021913 finished
 INFO 2025-08-20 07:17:57,569 PID:903186:140125480658624(Allocator-6337772) Allocating aws_x86_64_spot_prod_06337772_20250820_071634 finished => UP
 INFO 2025-08-20 07:17:57,597 PID:903186:140125480658624(Terminator-6335056) Terminating copr_p09_01_prod_06335056_20250820_013413 started

The third one:

WARNING 2025-08-19 20:42:43,491 PID:903186:140127796283200(Manager) Requesting copr_p09_01_prod_06233283_20250814_063102 removal for continuous failures
 INFO 2025-08-19 20:42:43,528 PID:903186:140127695840960(Terminator-6233283) Terminating copr_p09_01_prod_06233283_20250814_063102 started

Well, I wasn't able to find the real reason why copr-rpmbuild failed starting, but we should get a better idea after the fix for #3826, if this still continues to happen.

[praiskup]

See also: #3693