perf: Implement retry logic to improve resiliency

[Shreyas-Microsoft]

Purpose

This pull request introduces optimizations and refactors to the SQL agent backend, focusing on improved performance, parallel processing, and enhanced retry logic. It removes outdated rate-limiting workarounds and streamlines agent communication strategies. The most significant changes include the overhaul of the CommsManager class, the introduction of batch processing, and the removal of unnecessary asyncio imports.

Performance and Parallel Processing Improvements:

• src/backend/sql_agents/helpers/comms_manager.py: Refactored the CommsManager class to include parallel processing capabilities, batch processing, optimized retry logic, and faster termination and selection strategies. Added a thread pool for parallel execution and reduced retry delays and truncation thresholds for improved efficiency.

Removal of Deprecated Rate-Limiting Workarounds:

• src/backend/sql_agents/convert_script.py: Commented out the temporary await asyncio.sleep(5) workaround in the convert_script function.
• src/backend/sql_agents/process_batch.py: Commented out the temporary await asyncio.sleep(5) workaround in the process_batch_async function.

Code Cleanup:

• src/backend/sql_agents/convert_script.py: Removed an unused asyncio import to clean up the code.
• src/backend/sql_agents/process_batch.py: Removed an unused asyncio import to clean up the code.

Does this introduce a breaking change?

• Yes
• No

Golden Path Validation

• I have tested the primary workflows (the "golden path") to ensure they function correctly without errors.

Deployment Validation

• I have validated the deployment process successfully and all services are running as expected with this change.

What to Check

Verify that the following are valid

• ...

Other Information