How to tell if a runner got created?

[ryan-elliott]

I am setting up self-hosted runners following the instruction here, and it seems like the scale-set-controller and listener are running correctly. However, when I push a commit to github, the resulting action just says "working". I watched the listener logs, and noticed that the replica count increased when I pushed a new commit. I was also watching the "arc-runners" namespace for any new pods, but I didn't see any appear (maybe they appeared and shut down too fast for me to see).
Here's a snippet of the logs showing what happened when I triggered 2 new actions on top of the 2 that were already stuck (I believe this is the "JobId 3" and "JobId 4" mentioned in the logs). If it says "currentRunnerCount": 4, I would assume I should see 4 runner pods in the "arc-runners" namespace. Am I missing something?

2025-03-01T07:17:59Z INFO listener-app.listener Getting next message {"lastMessageID": 100000002} 2025-03-01T07:17:59Z INFO listener-app.listener Processing message {"messageId": 100000003, "messageType": "RunnerScaleSetJobMessages"} 2025-03-01T07:17:59Z INFO listener-app.listener New runner scale set statistics. {"statistics": {"totalAvailableJobs":0,"totalAcquiredJobs":0,"totalAssignedJobs":4,"totalRunningJobs":0,"totalRegisteredRunners":0,"totalBusyRunners":0,"totalIdleRunners":0}} 2025-03-01T07:17:59Z INFO listener-app.listener Job assigned message received {"jobId": 4} 2025-03-01T07:17:59Z INFO listener-app.listener Job assigned message received {"jobId": 3} 2025-03-01T07:17:59Z INFO listener-app.listener Deleting last message {"lastMessageID": 100000003} 2025-03-01T07:18:00Z INFO listener-app.worker.kubernetesworker Calculated target runner count {"assigned job": 4, "decision": 4, "min": 0, "max": 2147483647, "currentRunnerCount": 4, "jobsCompleted": 0} 2025-03-01T07:18:00Z INFO listener-app.worker.kubernetesworker Compare {"original": "{\"metadata\":{\"creationTimestamp\":null},\"spec\":{\"replicas\":-1,\"patchID\":-1,\"ephemeralRunnerSpec\":{\"metadata\":{\"creationTimestamp\":null},\"spec\":{\"containers\":null}}},\"status\":{\"currentReplicas\":0,\"pendingEphemeralRunners\":0,\"runningEphemeralRunners\":0,\"failedEphemeralRunners\":0}}", "patch": "{\"metadata\":{\"creationTimestamp\":null},\"spec\":{\"replicas\":4,\"patchID\":401,\"ephemeralRunnerSpec\":{\"metadata\":{\"creationTimestamp\":null},\"spec\":{\"containers\":null}}},\"status\":{\"currentReplicas\":0,\"pendingEphemeralRunners\":0,\"runningEphemeralRunners\":0,\"failedEphemeralRunners\":0}}"} 2025-03-01T07:18:00Z INFO listener-app.worker.kubernetesworker Preparing EphemeralRunnerSet update {"json": "{\"spec\":{\"patchID\":401,\"replicas\":4}}"} 2025-03-01T07:18:00Z INFO listener-app.worker.kubernetesworker Ephemeral runner set scaled. {"namespace": "arc-runners", "name": "arc-runner-set-cpcmj", "replicas": 4}

[ryan-elliott]

After restarting two of the three physical nodes in this Kubernetes cluster (the two nodes had crashed), the Listener was able to launch Runner pods for the actions and ran them all to completion. So this seems to have been a problem with insufficient resources.