pulp replication task get stuck in the running state and never completes

[mchithu]

Version
pulpcore == 3.74.1
pulp-rpm == 3.29.0

Describe the bug
Sometimes the pulp-worker tasks are getting stuck in the running state.
For sample, the following task started on 1-Oct-2025, it's not been completed yet. Other containers pulp-content, pulp-api, pulp-web are active. Only this worker task is stuck. The containers are running a podman quadlet in a VSI. Not able view the container logs of the podman but able to run commands or take control of container via shell.

{
    "pulp_href": "/pulp/api/v3/tasks/01999f2c-cbac-7f9c-bc71-5da4b0fc2f07/",
    "prn": "prn:core.task:01999f2c-cbac-7f9c-bc71-5da4b0fc2f07",
    "pulp_created": "2025-10-01T09:48:57.132567Z",
    "pulp_last_updated": "2025-10-01T09:48:57.132581Z",
    "state": "running",
    "name": "pulp_rpm.app.tasks.synchronizing.synchronize",
    "logging_cid": "a2b687bf417640d186cea6f019b3ddcb",
    "created_by": "/pulp/api/v3/users/2/",
    "unblocked_at": "2025-10-01T09:48:57.150378Z",
    "started_at": "2025-10-01T09:48:57.209869Z",
    "finished_at": null,
    "error": null,
    "worker": "/pulp/api/v3/workers/019641ea-2684-7712-826a-ed3e3661956a/",
    "parent_task": "/pulp/api/v3/tasks/01999f2c-43fb-7918-9b01-fef178642804/",
    "child_tasks": [],
    "task_group": "/pulp/api/v3/task-groups/01999f2c-43f1-7627-8869-6d6d439f84eb/",
    "progress_reports": [
        {
            "message": "Skipping Packages",
            "code": "sync.skipped.packages",
            "state": "completed",
            "total": 0,
            "done": 0,
            "suffix": null
        },
        {
            "message": "Parsed Packages",
            "code": "sync.parsing.packages",
            "state": "completed",
            "total": 30572,
            "done": 30572,
            "suffix": null
        },
        {
            "message": "Downloading Metadata Files",
            "code": "sync.downloading.metadata",
            "state": "completed",
            "total": null,
            "done": 6,
            "suffix": null
        },
        {
            "message": "Downloading Artifacts",
            "code": "sync.downloading.artifacts",
            "state": "running",
            "total": null,
            "done": 878,
            "suffix": null
        },
        {
            "message": "Parsed Comps",
            "code": "sync.parsing.comps",
            "state": "completed",
            "total": 25,
            "done": 25,
            "suffix": null
        },
        {
            "message": "Parsed Advisories",
            "code": "sync.parsing.advisories",
            "state": "completed",
            "total": 4775,
            "done": 4775,
            "suffix": null
        },
        {
            "message": "Associating Content",
            "code": "associating.content",
            "state": "running",
            "total": null,
            "done": 897,
            "suffix": null
        }
    ],
    "created_resources": [
        "<unavailable>"
    ],
    "reserved_resources_record": [
        "prn:rpm.rpmrepository:0196ced8-f207-7c7e-866e-14795a82267b",
        "shared:prn:rpm.rpmremote:0196ced8-f1fa-702e-9381-a7212ab26988",
        "shared:prn:core.upstreampulp:0196463e-55e1-72bb-a2ed-738161ade2cd",
        "shared:prn:core.domain:0195d0e8-a1b4-7f67-ae32-6daf27d7464e"
    ]
}

To Reproduce

Expected behavior
What are the possible reasons a task in worker is stuck? Anything to check further because the container is in active state.

Additional context

[pedro-psb]

I've seen this happen before on a very similar context (an rpm sync task on the downloading artifacts stage). It would happen ocasionally under certain types of connection failures.

In this case, the problem was a bug with the Remote Downloader timeout configurations, and it was fixed and backported to 3.74.2. I see you are one z version behind, can you upgrade and report back?