Parallel Execution: Update Functions

[gvatsal60]

Challenges:

• Parallel Execution: Background processes (&) allow tasks to run concurrently, but maintaining ordered logs can be tricky.
• Log Synchronization: Standard output can get mixed up when multiple commands write at the same time.
• Avoiding File Logging: We need an alternative way to structure logs without redirecting them to files.
• Error Handling: Ensuring failures in individual updates don't break the entire process.

Discussion Points:

• Should we prioritize strict execution order, or is some level of disorder acceptable?
• How should we handle failures in one update process while others are still running?
• Would using named pipes (mkfifo) help with structured logging?
• Is there a lightweight way to enforce sequential log prints without slowing down execution?

What are your thoughts? Let's brainstorm solutions together. 🚀

[andmpel]

Discussion Points:

• Should we prioritize strict execution order, or is some level of disorder acceptable?

I'm personally think any order of execution is fine as long as it all outputs neatly.

• How should we handle failures in one update process while others are still running?

Continue running other update modules if one fails. Display failure at the end?

• Would using named pipes (mkfifo) help with structured logging?

I'm not sure what this means.

• Is there a lightweight way to enforce sequential log prints without slowing down execution?

Not sure.

[gvatsal60]

I still haven't found the right solution yet. I’m specifically looking for an approach that avoids writing errors and output to a file, so that adds an extra challenge. If anyone has ideas for handling failures and structured logging without relying on file storage, I’d love to hear them.