enhance: workforce quality check (#3254)

Co-authored-by: Wendong-Fan <133094783+Wendong-Fan@users.noreply.github.com>
Co-authored-by: Wendong-Fan <w3ndong.fan@gmail.com>

Author: nitpicker55555

camel/societies/workforce/prompts.py

@@ -293,58 +293,139 @@
 - Written without any relative references (e.g., "the previous task").
 """
 
-FAILURE_ANALYSIS_PROMPT = TextPrompt(
-    """You need to analyze a task failure and decide on the best recovery strategy.
-
-**TASK FAILURE DETAILS:**
-Task ID: {task_id}
-Task Content: {task_content}
-Failure Count: {failure_count}/3
-Error Message: {error_message}
-Worker ID: {worker_id}
-Task Depth: {task_depth}
-Additional Info: {additional_info}
-
-**AVAILABLE RECOVERY STRATEGIES:**
-
-1. **RETRY**: Attempt the same task again without changes
-   - Use for: Network errors, temporary API issues, random failures
-   - Avoid for: Fundamental task misunderstanding, capability gaps
-
-2. **REPLAN**: Modify the task content to address the underlying issue
-   - Use for: Unclear requirements, insufficient context, correctable errors
-   - Provide: Modified task content that addresses the failure cause
-   - **CRITICAL**: The replanned task MUST be a clear, actionable
-     instruction for an AI agent, not a question or request for a human.
-
-3. **DECOMPOSE**: Break the task into smaller, more manageable subtasks
-   - Use for: Complex tasks, capability mismatches, persistent failures
-   - Consider: Whether the task is too complex for a single worker
-
-4. **CREATE_WORKER**: Create a new worker node to handle the task
-   - Use for: Fundamental task misunderstanding, capability gaps
-
-**ANALYSIS GUIDELINES:**
-
-- **Connection/Network Errors**: Almost always choose RETRY
-- **Model Processing Errors**: Consider REPLAN if the task can be clarified, otherwise DECOMPOSE
-- **Capability Gaps**: Choose DECOMPOSE to break into simpler parts. If a
-  replan can work, ensure the new task is a command for an agent, not a
-  request to a user.
-- **Ambiguous Requirements**: Choose REPLAN with clearer instructions
-- **High Failure Count**: Lean towards DECOMPOSE rather than repeated retries
-- **Deep Tasks (depth > 2)**: Prefer RETRY or REPLAN over further
-  decomposition
+TASK_ANALYSIS_PROMPT = TextPrompt(
+    """You are analyzing a task to evaluate its quality and determine recovery actions if needed.
+
+**TASK INFORMATION:**
+- Task ID: {task_id}
+- Task Content: {task_content}
+- Task Result: {task_result}
+- Failure Count: {failure_count}
+- Task Depth: {task_depth}
+- Assigned Worker: {assigned_worker}
+
+**ISSUE TYPE: {issue_type}**
+
+{issue_specific_analysis}
+
+**STEP 1: EVALUATE TASK QUALITY**
+
+First, assess whether the task was completed successfully and meets quality standards:
+
+**For Task Failures (with error messages):**
+- The task did not complete successfully
+- An error occurred during execution
+- Quality is automatically insufficient
+- Focus on analyzing the error cause
+
+**For Quality Issues (task completed but needs evaluation):**
+Evaluate the task result based on these criteria:
+1. **Completeness**: Does the result fully address all task requirements?
+2. **Accuracy**: Is the result correct and well-structured?
+3. **Missing Elements**: Are there any missing components or quality issues?
+
+Provide:
+- Quality score (0-100): Objective assessment of result quality
+- Specific issues list: Any problems found in the result
+- Quality sufficient: Boolean indicating if quality meets standards
+
+**STEP 2: DETERMINE RECOVERY STRATEGY (if quality insufficient)**
+
+If the task quality is insufficient, select the best recovery strategy:
+
+**Available Strategies:**
+
+1. **retry** - Retry with the same worker and task content
+   - **Best for**:
+     * Network errors, connection timeouts, temporary API issues
+     * Random failures that are likely temporary
+     * Minor quality issues that may resolve on retry
+   - **Not suitable for**:
+     * Fundamental task misunderstandings
+     * Worker capability gaps
+     * Persistent quality problems
+
+2. **reassign** - Assign to a different worker
+   - **Best for**:
+     * Current worker lacks required skills/expertise
+     * Worker-specific quality issues
+     * Task requires different specialization
+   - **Not suitable for**:
+     * Task description is unclear (use replan instead)
+     * Task is too complex (use decompose instead)
+   - **Note**: Only available for quality issues, not failures
+
+3. **replan** - Modify task content with clearer instructions
+   - **Best for**:
+     * Unclear or ambiguous requirements
+     * Missing context or information
+     * Task description needs improvement
+   - **Requirements**:
+     * Provide modified_task_content with enhanced, clear instructions
+     * Modified task must be actionable for an AI agent
+     * Address the root cause identified in issues
+
+4. **decompose** - Break into smaller, manageable subtasks
+   - **Best for**:
+     * Task is too complex for a single worker
+     * Multiple distinct sub-problems exist
+     * Persistent failures despite retries
+     * Capability mismatches that need specialization
+   - **Consider**:
+     * Task depth (avoid if depth > 2)
+     * Whether subtasks can run in parallel
+
+5. **create_worker** - Create new specialized worker
+   - **Best for**:
+     * No existing worker has required capabilities
+     * Need specialized skills not currently available
+   - **Consider**:
+     * Whether decomposition could work instead
+     * Cost of creating new worker vs alternatives
+   - **Note**: Only available for task failures, not quality issues
+
+**DECISION GUIDELINES:**
+
+**Priority Rules:**
+1. Connection/Network Errors → **retry** (almost always)
+2. Deep Tasks (depth > 2) → Avoid decompose, prefer **retry** or **replan**
+3. Worker Skill Mismatch → **reassign** (quality) or **decompose** (failure)
+4. Unclear Requirements → **replan** with specifics
+5. Task Too Complex → **decompose** into subtasks
 
 **RESPONSE FORMAT:**
-You must return a valid JSON object with these fields:
-- "strategy": one of "retry", "replan", or "decompose" 
-- "reasoning": explanation for your choice (1-2 sentences)
-- "modified_task_content": new task content if strategy is "replan", null otherwise
+{response_format}
 
-**Example Response:**
-{{"strategy": "retry", "reasoning": "The connection error appears to be temporary and network-related, a simple retry should resolve this.", "modified_task_content": null}}
-
-**CRITICAL**: Return ONLY the JSON object. No explanations or text outside the JSON structure.
+**CRITICAL**:
+- Return ONLY a valid JSON object
+- No explanations or text outside the JSON structure
+- Ensure all required fields are included
+- Use null for optional fields when not applicable
 """
 )
+
+FAILURE_ANALYSIS_RESPONSE_FORMAT = """JSON format:
+{
+  "reasoning": "explanation (1-2 sentences)",
+  "recovery_strategy": "retry|replan|decompose|create_worker",
+  "modified_task_content": "new content if replan, else null",
+  "issues": ["error1", "error2"]
+}"""
+
+QUALITY_EVALUATION_RESPONSE_FORMAT = """JSON format:
+{
+  "quality_score": 0-100,
+  "reasoning": "explanation (1-2 sentences)", 
+  "issues": ["issue1", "issue2"],
+  "recovery_strategy": "retry|reassign|replan|decompose or null",
+  "modified_task_content": "new content if replan, else null"
+}"""
+
+TASK_AGENT_SYSTEM_MESSAGE = """You are an intelligent task management assistant responsible for planning, analyzing, and quality control.
+
+Your responsibilities include:
+1. **Task Decomposition**: Breaking down complex tasks into manageable subtasks that can be executed efficiently and in parallel when possible.
+2. **Failure Analysis**: Analyzing task failures to determine the root cause and recommend appropriate recovery strategies (retry, replan, decompose, or create new worker).
+3. **Quality Evaluation**: Assessing completed task results to ensure they meet quality standards and recommending recovery strategies if quality is insufficient (retry, reassign, replan, or decompose).
+
+You must provide structured, actionable analysis based on the task context, failure history, worker capabilities, and quality criteria. Your decisions directly impact the efficiency and success of the workforce system."""

camel/societies/workforce/structured_output_handler.py

@@ -19,8 +19,8 @@
 
 from camel.logger import get_logger
 from camel.societies.workforce.utils import (
-    RecoveryDecision,
     RecoveryStrategy,
+    TaskAnalysisResult,
     TaskAssignResult,
     WorkerConf,
 )
@@ -65,9 +65,9 @@ class StructuredOutputHandler:
                 r'description.*?:\s*"([^"]+)"'
             ),
         ],
-        'RecoveryDecision': [
-            r'"strategy"\s*:\s*"([^"]+)".*?"reasoning"\s*:\s*"([^"]+)"',
-            r'strategy.*?:\s*"([^"]+)".*?reasoning.*?:\s*"([^"]+)"',
+        'TaskAnalysisResult': [
+            r'"recovery_strategy"\s*:\s*"([^"]+)".*?"reasoning"\s*:\s*"([^"]+)"',
+            r'recovery_strategy.*?:\s*"([^"]+)".*?reasoning.*?:\s*"([^"]+)"',
         ],
     }
 
@@ -239,27 +239,38 @@ def _extract_with_schema_patterns(
                     except (IndexError, AttributeError):
                         continue
 
-        elif schema_name == 'RecoveryDecision':
+        elif schema_name == 'TaskAnalysisResult':
             for pattern in patterns:
                 match = re.search(pattern, text, re.DOTALL | re.IGNORECASE)
                 if match:
                     try:
-                        strategy = match.group(1)
+                        recovery_strategy = match.group(1)
                         reasoning = match.group(2)
                         # Look for modified_task_content
                         content_match = re.search(
                             r'"modified_task_content"\s*:\s*"([^"]*)"',
                             text,
                             re.IGNORECASE,
                         )
+                        # Look for quality_score (for quality evaluation)
+                        score_match = re.search(
+                            r'"quality_score"\s*:\s*(\d+)',
+                            text,
+                            re.IGNORECASE,
+                        )
                         return {
-                            'strategy': strategy,
+                            'recovery_strategy': recovery_strategy,
                             'reasoning': reasoning,
                             'modified_task_content': (
                                 content_match.group(1)
                                 if content_match
                                 else None
                             ),
+                            'quality_score': (
+                                int(score_match.group(1))
+                                if score_match
+                                else None
+                            ),
                         }
                     except (IndexError, AttributeError):
                         continue
@@ -370,21 +381,22 @@ def _fix_common_issues(
                         else:
                             assignment['dependencies'] = []
 
-        elif schema_name == 'RecoveryDecision':
-            # Ensure strategy is valid
-            if 'strategy' in fixed_data:
-                strategy = fixed_data['strategy'].lower()
+        elif schema_name == 'TaskAnalysisResult':
+            # Ensure recovery_strategy is valid
+            if 'recovery_strategy' in fixed_data:
+                strategy = fixed_data['recovery_strategy'].lower()
                 valid_strategies = [
                     'retry',
                     'replan',
                     'decompose',
                     'create_worker',
+                    'reassign',
                 ]
                 if strategy not in valid_strategies:
                     # Try to match partial
                     for valid in valid_strategies:
                         if valid.startswith(strategy) or strategy in valid:
-                            fixed_data['strategy'] = valid
+                            fixed_data['recovery_strategy'] = valid
                             break
 
         return fixed_data
@@ -410,10 +422,10 @@ def _create_default_instance(schema: Type[BaseModel]) -> BaseModel:
                 sys_msg="You are a helpful assistant.",
                 description="A general-purpose worker",
             )
-        elif schema_name == 'RecoveryDecision':
-            return RecoveryDecision(
-                strategy=RecoveryStrategy.RETRY,
+        elif schema_name == 'TaskAnalysisResult':
+            return TaskAnalysisResult(
                 reasoning="Unable to parse response, defaulting to retry",
+                recovery_strategy=RecoveryStrategy.RETRY,
                 modified_task_content=None,
             )
         else:
@@ -482,11 +494,11 @@ def create_fallback_response(
                 description=f"Fallback worker for task: {task_content}...",
             )
 
-        elif schema_name == 'RecoveryDecision':
+        elif schema_name == 'TaskAnalysisResult':
             # Default to retry strategy
-            return RecoveryDecision(
-                strategy=RecoveryStrategy.RETRY,
+            return TaskAnalysisResult(
                 reasoning=f"Fallback decision due to: {error_message}",
+                recovery_strategy=RecoveryStrategy.RETRY,
                 modified_task_content=None,
             )