Sudden error: "No staging files found

[jannikbend]

Describe the bug
After running rudderstack for a while a “no staging files found” message gets logged with error log level pretty frequently. See below:

“{"level":"WARN","ts":"2024-08-07T05:26:49.148Z","logger":"runner.warehouse.router.CLICKHOUSE","caller":"router/upload.go:600","msg":"upload error","uploadJobID":30514,"namespace":"<…>","sourceID":"<…>","sourceType":"Node","destinationID":"CLICKHOUSE_DESTINATION","destinationType":"CLICKHOUSE","workspaceID":"<…>","useRudderStorage":false,"uploadStatus":"internal_processing_failed","error":"no staging files found","priority":100,"retried":false,"attempt":0,"loadFileType":"csv","errorMapping":"uncategorised","destinationCredsValid":null}”
{"level":"ERROR","ts":"2024-08-07T06:32:13.889Z","logger":"runner.warehouse.router.CLICKHOUSE","caller":"router/router.go:274","msg":"[WH] Failed in handle Upload jobs for worker: %!w(*errors.errorString=&{s:no staging files found})"}

The error is often combined with the following:

“{"level":"WARN","ts":"2024-08-07T05:26:49.148Z","logger":"runner.warehouse.router.CLICKHOUSE","caller":"router/upload.go:505","msg":"error setting upload status","uploadJobID":30514,"namespace”:”<…>”,”sourceID":"<…>","sourceType":"Node","destinationID":"CLICKHOUSE_DESTINATION","destinationType":"CLICKHOUSE","workspaceID":"<…>","useRudderStorage":false,"error":"upload not found"}”,

however that message is logged as warning and not as an error.

The Job IDs mentioned in the log message (30514) does indeed not exist in the Postgres database, but we do also observe no missing or failed events in any case.

Steps to reproduce the bug
Unclear: On a freshly installed rudderstack instance this does not occur.
The error suddenly occurs after running low impact load tests for a few hours.

Expected behavior
No errors should occur

Screenshots
None

Any additional context
Rudder-Server: 1.29.0
Transformer: 1.49.0
Related Slack Post: https://rudderstack.slack.com/archives/C01E4PLB135/p1723477378948789

[contributor-support]

Thanks for opening this issue! We'll get back to you shortly. If it is a bug, please make sure to add steps to reproduce the issue.

[gitcommitshow]

@jannikbend Ideally it should not happen if it is a valid upload job. Although there is one scenario where we merge old waiting job with a new one and delete the old upload job. If for some reason because of race condition the old upload job got picked up to process it will not see any staging files because all of them will be marked for the new one.

[jannikbend]

Thanks for the explanation. We will ignore this error then for now.

[gitcommitshow]

@jannikbend v1.39 release fixes the race condition issue. So you should not encounter this message again if upgraded to v1.39. It will be helpful if you can confirm the same when you upgrade.

Note: While upgrading rudder-server, make sure you upgrade rudder-transformer to compatible version

[jannikbend]

Hi, I just tested it and it seems the error still persist, but is much less frequent. I used rudderstack v1.39.0 and transformer version v1.50.0.
The error message seems a little different was well:
v1.29.0: {"level":"ERROR","ts":"2025-01-16T10:28:03.525Z","logger":"runner.warehouse.router.CLICKHOUSE","caller":"router/router.go:274","msg":"[WH] Failed in handle Upload jobs for worker: %!w(*errors.errorString=&{s:no staging files found})"}

v1.39.0: {"level":"ERROR","ts":"2025-01-16T07:30:42.119Z","logger":"runner.warehouse.router.CLICKHOUSE","caller":"router/router.go:273","msg":"[WH] Failed in handle Upload jobs for worker: no staging files found"}

[gitcommitshow]

@jannikbend thanks for the update. Can you please share the complete server logs?

[jannikbend]

The error occurred at 2025-01-17T14:13:20.760Z in the attached log file. We are running multiple instances of rudderstack but the logs are from the instance where the error happened only.

Rudderstack-Logs.txt