[vpj] Push Job Timeout

[KaiSernLim]

Venice Push Job Diagram as a refresher:

Problem Statement

There is no timeout on the Compute Engine DataWriterComputeJob. This leads to some push jobs taking days and even sometimes weeks to complete, which significantly delays the debugging process. There is an existing timeout (bootstrapToOnlineTimeoutInHours) on the job status polling, but that takes place after the data writer job portion (which is where we've been witnessing significant delays). With the increased frequency of KIF repushes, this issue will become even more pronounced.

Solution

There should be a timeout on the entire push job as a whole. If exceeded, the push job should be cancelled. The existing configuration bootstrapToOnlineTimeoutInHours can be repurposed for this, and this should've be the original purpose of the configuration.

Concurrency-Specific Checks

Both reviewer and PR author to verify

• Code has no race conditions or thread safety issues.
• No blocking calls inside critical sections that could lead to deadlocks or performance degradation.
• Verified thread-safe collections are used (e.g., ConcurrentHashMap, CopyOnWriteArrayList).
• Validated proper exception handling in multi-threaded code to avoid silent thread termination.

How was this PR tested?

• New unit tests added.

• Added testPushJobTimeout() which tests the poll job status gets cancelled.
• Added testDataWriterComputeJobTimeout() which tests the data writer job gets killed.
• Deprecated testPushJobPollStatus() and testPushJobUnknownPollStatusDoesWaiting(), which pertained to the old usage of the bootstrapToOnlineTimeoutInHours configuration.

Does this PR introduce any user-facing or breaking changes?

• No. You can skip the rest of this section.

[nisargthakkar]

Some comments in tests. But main code change LGTM

[nisargthakkar]

LGTM. Thanks @KaiSernLim!

[nisargthakkar]

LGTM. Thanks for the dilligence @KaiSernLim!