feat: enhance MCP tool response handling to support image responses

[shivangag]

Related GitHub Issue

Fixes: #5233

Description

This PR enhances MCP (Model Context Protocol) tool response handling to support images alongside text content. The implementation allows MCP servers to return images in their tool responses, which are then properly displayed in the Roo Code UI.

Key implementation details:

• Modified processToolContent() to extract and return both text and images from MCP tool results
• Updated executeToolAndProcessResult() to handle image data and pass it through the response pipeline
• Enhanced combineCommandSequences() to preserve images from MCP server responses during command sequence processing
• Updated UI components (McpExecution.tsx and ChatRow.tsx) to render images alongside text responses

Design choices:

• Images are handled as part of the tool result structure, maintaining backward compatibility with text-only responses
• The implementation supports both text-only, image-only, and combined text+image responses from MCP tools

Test Procedure

Unit Tests Added:

• Added comprehensive tests in useMcpToolTool.spec.ts to verify:
  • Correct extraction of both text and images from tool results
  • Proper handling of image-only responses
  • Backward compatibility with text-only responses
• Updated tests in combineCommandSequences.spec.ts to ensure images are preserved during command sequence processing

Manual Testing:

1. Connect to an MCP server that supports image responses (e.g., browser MCP server with screenshot capability)
2. Execute a tool that returns images (e.g., take a screenshot)
3. Verify that both text and images are displayed correctly in the chat interface
4. Test with various MCP tools to ensure backward compatibility with text-only responses

Reviewers can test by:

1. Running the test suite: npm test -- useMcpToolTool.spec.ts
2. Setting up an MCP server with image capabilities and testing the integration manually

Pre-Submission Checklist

• Issue Linked: This PR is linked to an approved GitHub Issue (see "Related GitHub Issue" above).
• Scope: My changes are focused on the linked issue (one major feature/fix per PR).
• Self-Review: I have performed a thorough self-review of my code.
• Testing: New and/or updated tests have been added to cover my changes (if applicable).
• Documentation Impact: I have considered if my changes require documentation updates (see "Documentation Updates" section below).
• Contribution Guidelines: I have read and agree to the Contributor Guidelines.

Screenshots / Videos

Before: MCP tool responses only displayed text content, images were not supported.

After: MCP tool responses now display both text and images seamlessly in the chat interface.
codicon-file-media indicates if images are present in mcp response and the number of images.

Images are presented using the Thumbnail component inside mcp-server-response block.

Documentation Updates

• No documentation updates are required.
• Yes, documentation updates are required. The MCP integration documentation should be updated to mention image support capabilities for tool responses.

Additional Notes

• This enhancement maintains full backward compatibility with existing MCP servers that only return text
• The implementation follows the existing patterns for handling MCP tool responses
• Images are processed and displayed using the same infrastructure as other image content in Roo Code

Improvements

Old chat loaded from memory show tool call argument inside the response.

---

Important

Enhances MCP tool response handling to support image responses, updating processing logic, UI components, and adding new settings for image handling.

• Behavior:
  • processToolContent() in useMcpToolTool.ts now extracts and returns both text and images from MCP tool results.
  • executeToolAndProcessResult() in useMcpToolTool.ts handles image data and passes it through the response pipeline.
  • combineCommandSequences() in combineCommandSequences.ts preserves images from MCP server responses.
  • UI components McpExecution.tsx and ChatRow.tsx updated to render images alongside text responses.
• Settings:
  • Adds mcpMaxImagesPerResponse and mcpMaxImageSizeMB to global-settings.ts for image handling configuration.
• Tests:
  • Adds tests in useMcpToolTool.spec.ts for image extraction and handling.
  • Updates combineCommandSequences.spec.ts to test image preservation in command sequences.
• Misc:
  • Updates webviewMessageHandler.ts to handle new image settings.
  • Adds image handling logic to Thumbnails.tsx for UI display.

^{This description was created by for a416090. You can customize this summary. It will automatically update as commits are pushed.}

[daniel-lxs]

Hey @shivangag,

I noticed this PR doesn't have a linked GitHub issue. As per the contributing guidelines, all PRs need to be linked to an approved issue. Could you create an issue describing this enhancement and link it to this PR? This helps us keep things traceable and ensures features are properly discussed before implementation.

I'll mark this PR as "In progress" for now until the dev team has a chance to review the issue.

[shivangag]

Thank you @daniel-lxs for the thorough review and excellent suggestions! I've implemented all the recommended enhancements to make MCP image handling more robust and secure.

Security & Performance Controls

• Image size limits: Added configurable mcpMaxImageSizeMB setting with 10MB default
• Count limits: Added configurable mcpMaxImagesPerResponse setting with 20 image default
• User controls: Both settings are configurable through the MCP settings panel

Robustness Improvements

• Error handling: Wrapped image processing in try-catch blocks to prevent crashes from corrupted data
• Validation: Added base64 regex validation and type checking for data integrity
• Graceful degradation: Invalid images are skipped with detailed warning logs instead of failing

Code Quality Enhancements

• Maintainability: Extracted SUPPORTED_IMAGE_TYPES as a module-level constant
• Test coverage: Added comprehensive test suite covering all suggested edge cases including corrupted data, large images, unsupported MIME types, and malformed content

The implementation now safely handles potentially malicious or buggy MCP servers while maintaining full backward compatibility with existing functionality. I believe this addresses all the security, performance, and reliability concerns raised in the review.

Also created GitHub issue #5233 to properly document this feature enhancement.

[daniel-lxs]

Hey @shivangag, Thank you for taking a look at my previous suggestions. I took another look and left some new suggestions from my review. Let me know if you have any questions!

[daniel-lxs]

Hey @shivangag, sorry for the delay, the implementation looks solid, I left a couple of suggestions to handle potential issues.

Let me know what you think!

[daniel-lxs]

The image size calculation happens after the entire base64 string is already in memory. For very large images, this could cause memory spikes before the size check rejects them.

Consider adding an early size check based on string length:

// Quick size check before full validation
const approximateSizeMB = (item.data.length * 0.75) / (1024 * 1024);
if (approximateSizeMB > maxImageSizeMB * 1.5) { // Allow some margin for encoding overhead
    console.warn(`MCP image likely exceeds size limit based on string length`);
    return;
}

This would prevent processing extremely large strings that are obviously over the limit.

[daniel-lxs]

This hardcoded English string needs to be translated. Please add a translation key for "Failed to load image".

Suggested fix:

title={t("common:thumbnails.failedToLoad")}

Don't forget to add the translation to all locale files.

[daniel-lxs]

The alt text contains a hardcoded English string with string interpolation. This needs to be translated.

Suggested fix:

alt={t("common:thumbnails.altText", { index: index + 1 })}

The translation key should support interpolation, e.g., "Thumbnail {{index}}"

[daniel-lxs]

The current implementation uses the settings values without validating whether they're within reasonable bounds. For example, if someone manually edits their settings file and sets a negative number or an extremely large value, it could lead to unexpected behavior.

It might be worth adding validation like this:

const maxImagesPerResponse = Math.max(1, Math.min(100, state?.mcpMaxImagesPerResponse ?? 20));
const maxImageSizeMB = Math.max(0.1, Math.min(50, state?.mcpMaxImageSizeMB ?? 10));

This helps keep the values within safe operational limits, even in cases where the settings file is corrupted or manually modified.

[daniel-lxs]

The image count indicator is shown even when the response is collapsed. Is this intentional? It might be confusing to show a count for content that isn't visible.

Maybe you could:

1. Only showing the count when expanded
2. Adding a tooltip to explain what the count represents
3. Using different styling to indicate collapsed content with images

[daniel-lxs]

Images are validated sequentially in the forEach loop. For responses with many images, this could be slow.

Consider batching validation or using Promise.all for parallel processing:

const validatedImages = await Promise.all(
  toolResult.content
    .filter((item: any) => item.type === "image")
    .map(async (item: any) => validateAndProcessImage(item))
);
images.push(...validatedImages.filter(Boolean));

This would improve performance for MCP tools that return multiple images.

[daniel-lxs]

The base64 validation regex could be extracted as a constant for better maintainability and potential reuse:

const BASE64_REGEX = /^[A-Za-z0-9+/]*={0,2}$/;

This would make it easier to maintain and test the validation pattern.