Agent gets http code 500 after perfectly 10 minutes of running, and (presumably) shutsdown later without logging.

[Chri6s]

Hello! I have a droplet running cloudron and installed the wp-server from their app store. Setting up the agent was a painful road, mainly because of the ports and configs, but after all this hard work, it finally starts. Only problem now, starting the agent manually, running a pipeline(doesnt think it affects this problem but maybe) and waiting for 10 minutes after starting, it gives me this beautiful cryptic error:

From the Agent's logs:

{"level":"error","error":"rpc error: code = Unknown desc = unexpected HTTP status code received from server: 500 (Internal Server Error); transport: received unexpected content-type \"text/html\"","time":"2024-12-29T21:43:41Z","caller":"/src/cmd/agent/core/agent.go:278","message":"runner done with error"}

From the server's logs:

[error] 15#15: *16930 upstream timed out (110: Unknown error) while reading response header from upstream, client: 172.19.0.3, server: , request: "POST /proto.Woodpecker/Next HTTP/2.0", upstream: "grpc://127.0.0.1:9000", host: "<woodpecker-server-host>:9090"
[error] 15#15: *16930 could not find named location "@grpc_unavailable" while sending to client, client: 172.19.0.3, server: , request: "POST /proto.Woodpecker/Next HTTP/2.0", upstream: "grpc://127.0.0.1:9000", host: "<woodpecker-server-host>:9090"

After all of this, woodpecker-agent still seems to work, can run pipelines etc. But checking back on it after a couple of hours, It... dies, without error message. The container will say its, healthy, its running, but no response, no Healthcheck grpc requests... nothing.
The logs doesnt say anything before its "ghost-shutdown" this above is that always shows up when this happens.
If anything got any solution or explaination for this, please share it with me! Thank you!

[qwerty287]

Looks like a duplicate of #4503
But a solution isn't there as well...

[Chri6s]

Bummer... I suspect its something with the WOODPECKER_GRPC_SECURE env. Just as in #4503 I have that flag set to true. I guess I'll try some workarounds.

[justjulian]

@Chri6s I'm having the exact same issue. Did you find a suitable workaround?

[Chri6s]

@justjulian I did actually. Its just a cronjob restarting the agent's container every 10 minutes, because from my experiences it's suspiciously after 10 minutes of the agent running. You could maybe try that. Only problem with this is that when you have a currently running task for the agent, and the cronjob's restart hits while its running, it will error out in the dashboard, but it has a small chance of happening. If you need like a very sustainable, very 24/7 always up workaround this sadly cant do that job... But I can assure you, this helps out the more "casual" users of woodpecker.

[justjulian]

@Chri6s Thanks for the ideas! Restarting every 10 minutes will not work so super well for me because I have some pipelines that need a bit more time than that.
Though, I experimented with the cron feature and have now a cron job that just echos some message every 7 minutes-ish. That seems to do the trick, the agent continues to respond after 10 minutes and the http 500 code doesn't show.
More complete example:

1. In /repos//settings/crons add a job with schedule */7 * * * *
2. For that repo add a woodpecker ci file with something like:

steps:
  - name: keepalive
    image: debian
    commands:
      - echo "Keep connection alive"
    when:
      event: cron

I used a dedicated repo for that to not clutter other projects with pipelines every few minutes. Seems to work well so far. Still, would be better if the agent could do that by itself.

Also, I assume this has to be configured so that every agent runs this cron job, in case one has multiple agents.

[Chri6s]

@justjulian Cool idea! Also, just out of curiosity, what kind of pipelines are you running? Usually and I think by standard, woodpecker has a strict rule for pipeline runtime, if i remember correctly 3 minutes. I'm asking cause I had some pipelines that I wanted to run like this and was over the 3 minutes of max runtime and had to hack it together in another way.

[justjulian]

Let me rephrase that, I will have pipelines that run longer than 10 minutes. So far, I don't ;)
Though, I thought it was possible to configure the timeout: https://woodpecker-ci.org/docs/usage/project-settings#timeout

[Chri6s]

I tried changing that timeout yeah, but it still timed out after time. Probably because of this very issue I first encountered with the agent or some other anomaly.