Airbyte could not start the process within time limit. The workload was claimed but never started.

[estebanlgs]

Hello,

I've been trying to synchronize tables from a MySQL v3.11.0 schema using a self-hosted instance within an EC2 VM (4GB, 3xCPU). As both the source and destination (which is also a MySQL v-3.11.0 instance) pass the connection tests, whenever I'm trying to set up a connector (standard replication mode), the process will get stuck up to 5 hours, and the next error will pop up within the notifications tab:

message='Airbyte could not start the process within time limit. The workload was claimed but never started.', type='io.airbyte.workers.exception.WorkloadMonitorException', nonRetryable=false

Here are the last logs from the failed sync:

2025-02-13 21:36:08 platform INFO Starting async replication
2025-02-13 21:36:08 platform INFO Creating workload 513f5755-478f-4065-8c68-d09bf402d575_7_4_sync
2025-02-13 21:36:09 platform INFO APPLY Stage: CLAIM — (workloadId = 513f5755-478f-4065-8c68-d09bf402d575_7_4_sync) — (dataplaneId = local)
2025-02-13 21:36:09 platform INFO Claimed: true for 513f5755-478f-4065-8c68-d09bf402d575_7_4_sync via API for local
2025-02-13 21:36:09 platform INFO APPLY Stage: CHECK_STATUS — (workloadId = 513f5755-478f-4065-8c68-d09bf402d575_7_4_sync) — (dataplaneId = local)
2025-02-13 21:36:09 platform INFO No pod found running for workload 513f5755-478f-4065-8c68-d09bf402d575_7_4_sync
2025-02-13 21:36:09 platform INFO APPLY Stage: BUILD — (workloadId = 513f5755-478f-4065-8c68-d09bf402d575_7_4_sync) — (dataplaneId = local)
2025-02-13 21:36:09 platform INFO APPLY Stage: MUTEX — (workloadId = 513f5755-478f-4065-8c68-d09bf402d575_7_4_sync) — (dataplaneId = local)
2025-02-13 21:36:09 platform INFO Mutex key: 513f5755-478f-4065-8c68-d09bf402d575 specified for workload: 513f5755-478f-4065-8c68-d09bf402d575_7_4_sync. Attempting to delete existing pods...
2025-02-13 21:36:09 platform INFO Existing pods for mutex key: 513f5755-478f-4065-8c68-d09bf402d575 deleted.
2025-02-13 21:36:09 platform INFO APPLY Stage: LAUNCH — (workloadId = 513f5755-478f-4065-8c68-d09bf402d575_7_4_sync) — (dataplaneId = local)
2025-02-13 21:36:09 platform INFO [initContainer] image: airbyte/workload-init-container:1.4.0 resources: ResourceRequirements(claims=[], limits={memory=4Gi, cpu=3}, requests={memory=2Gi, cpu=2}, additionalProperties={})
2025-02-13 21:36:09 platform INFO Launching replication pod: replication-job-7-attempt-4 with containers:
2025-02-13 21:36:09 platform INFO [source] image: airbyte/source-mysql:3.11.0 resources: ResourceRequirements(claims=[], limits={memory=4Gi, cpu=3}, requests={memory=1Gi, cpu=1}, additionalProperties={})
2025-02-13 21:36:09 platform INFO [destination] image: airbyte/destination-mysql:1.0.3 resources: ResourceRequirements(claims=[], limits={memory=4Gi, cpu=3}, requests={memory=1Gi, cpu=1}, additionalProperties={})
2025-02-13 21:36:09 platform INFO [orchestrator] image: airbyte/container-orchestrator:1.4.0 resources: ResourceRequirements(claims=[], limits={memory=4Gi, cpu=3}, requests={memory=2Gi, cpu=2}, additionalProperties={})
2025-02-13 22:36:58 platform INFO Retry State: RetryManager(completeFailureBackoffPolicy=BackoffPolicy(minInterval=PT10S, maxInterval=PT30M, base=3), partialFailureBackoffPolicy=null, successiveCompleteFailureLimit=5, successivePartialFailureLimit=1000, totalCompleteFailureLimit=10, totalPartialFailureLimit=20, successiveCompleteFailures=5, successivePartialFailures=0, totalCompleteFailures=5, totalPartialFailures=0)
 Backoff before next attempt: 13 minutes 30 seconds
2025-02-13 22:36:58 platform ERROR Failing job: 7, reason: Job failed after too many retries for connection 513f5755-478f-4065-8c68-d09bf402d575
2025-02-13 23:06:19 platform ERROR Pipeline Error
2025-02-13 23:06:19 platform INFO Attempting to update workload: 513f5755-478f-4065-8c68-d09bf402d575_7_4_sync to FAILED.
2025-02-13 23:06:19 platform INFO Pipeline aborted after error for workload: 513f5755-478f-4065-8c68-d09bf402d575_7_4_sync.

I appreciate any suggestions.

[scastria]

I have exactly the same problem with exactly the same setup. I see lots of similar errors in Google searches and think it is a bug. I am trying to install kubernetes utilities so that I can look at the pod logs to better understand what the real error is.

[scastria]

If I install kubectl and look at the logs of the workload-launcher pod, I see errors like:

Error
io.airbyte.workload.launcher.pipeline.stages.model.StageError: io.airbyte.workers.exception.ResourceConstraintException: Unable to start the REPLICATION pod. This may be due to insufficient system resources. Please check available resources and try again.

[scastria]

I am just a newbie with Kubernetes. It seems I need to lower the requirements for jobs according to the airbyte slack channel. However, it never says HOW to edit the config values. I used abctl so all of the help stuff happened automatically so I have no idea where to to make any changes to my pods.

[ofranz]

did you fix it eventually?

[UmerImranUI]

did you fix it eventually?

HI any updates, I'm facing the same problem?