anthropic pdf #1074

narengogi · 2025-04-30T14:27:24Z

Author Description

so couple of things in this PR,
I've added two new parameters in the chat completions request body file content part

// file content part
{
  type: 'file';
  file?: {
    file_data?: string;
    file_id?: string;
    file_name?: string;
    file_url?: string; // new field
    mime_type?: string; // new field
  };
}

I've tested with anthropic, vertex anthropic and bedrock anthropic
note: vertex anthropic does not support plain text and pdf urls, bedrock does not support plain text, and only supports s3urls

here are request bodies to test
pdf url

{
    "model": "claude-3-5-sonnet-20240620",
    "max_tokens": 200,
    "stream": false,
    "messages": 
        }
    ]
}

pdf base64

{
    "model": "claude-3-5-sonnet-20240620",
    "max_tokens": 200,
    "stream": false,
    "messages": 
        }
    ]
}

plain text document

{
    "model": "claude-3-5-sonnet-20240620",
    "max_tokens": 200,
    "stream": false,
    "messages": 
        }
    ]
}

Summary By MatterAI

🔄 What Changed

This PR adds support for PDF files in Anthropic and Bedrock providers by introducing two new parameters in the chat completions request body file content part: file_url and mime_type. The implementation supports both PDF URLs and base64-encoded PDFs, as well as plain text documents.

🔍 Impact of the Change

This enhancement allows users to send PDF documents to Anthropic and Bedrock models for analysis. Different providers have varying support: Vertex Anthropic doesn't support plain text and PDF URLs, while Bedrock only supports S3 URLs for PDFs.

📁 Total Files Changed

3 files changed with 120 additions and 3 deletions:

src/providers/anthropic/chatComplete.ts: Added PDF handling for Anthropic
src/providers/bedrock/chatComplete.ts: Added PDF handling for Bedrock
src/types/requestBody.ts: Updated content type interfaces

🧪 Test Added

Manual testing was performed with Anthropic, Vertex Anthropic, and Bedrock Anthropic providers using various request bodies (PDF URL, PDF base64, plain text).

🔒 Security Vulnerabilities

N/A

Type of Change

New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
Documentation update
Refactoring (no functional changes)

How Has This Been Tested?

Unit Tests
Integration Tests
Manual Testing

Screenshots (if applicable)

N/A

Checklist

My code follows the style guidelines of this project
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
My changes generate no new warnings
I have added tests that prove my fix is effective or that my feature works
New and existing unit tests pass locally with my changes

Related Issues

N/A

Quality Recommendations

Add error handling for unsupported file types or formats
Add validation for file_url and file_data to ensure they are properly formatted
Consider adding unit tests to verify the PDF handling functionality
Add documentation comments for the new interfaces and functions

Sequence Diagram

sequenceDiagram
    participant Client
    participant Gateway
    participant AnthropicProvider
    participant BedrockProvider
    
    Client->>Gateway: POST /chat/completions
    Note over Client,Gateway: Request with file_url or file_data
    
    alt Anthropic Provider
        Gateway->>AnthropicProvider: transformAndAppendFileContentItem()
        Note over AnthropicProvider: Process file content based on type
        
        alt PDF URL
            AnthropicProvider-->>AnthropicProvider: Create AnthropicUrlPdfContentItem
            Note over AnthropicProvider: { type: 'document', source: { type: 'url', url: file_url } }
        else PDF Base64
            AnthropicProvider-->>AnthropicProvider: Create AnthropicBase64PdfContentItem
            Note over AnthropicProvider: { type: 'document', source: { type: 'base64', data: file_data, media_type: mime_type } }
        else Plain Text
            AnthropicProvider-->>AnthropicProvider: Create AnthropicPlainTextContentItem
            Note over AnthropicProvider: { type: 'document', source: { type: 'text', data: file_data, media_type: mime_type } }
        end
    else Bedrock Provider
        Gateway->>BedrockProvider: getMessageContent()
        Note over BedrockProvider: Process file content based on type
        
        alt File URL (S3)
            BedrockProvider-->>BedrockProvider: Create document with s3Location
            Note over BedrockProvider: { document: { format: fileFormat, name: UUID, source: { s3Location: { uri: file_url } } } }
        else File Data
            BedrockProvider-->>BedrockProvider: Create document with bytes
            Note over BedrockProvider: { document: { format: fileFormat, name: UUID, source: { bytes: file_data } } }
        end
    end
    
    Gateway-->>Client: Return completion response

#1075

VisargD · 2025-05-05T05:02:54Z

@narengogi - Can you please fix the examples added in the description. Not sure if they were updated by Matter bot or were present in the original description.

narengogi · 2025-05-05T11:55:25Z

here are request bodies to test
pdf url

{
    "model": "claude-3-5-sonnet-20240620",
    "max_tokens": 200,
    "stream": false,
    "messages": [
        {
            "role": "system",
            "content": "Say Hi"
        },
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "what is in this pdf"
                },
                {
                    "type": "file",
                    "file": {
                        "file_url": "https://pdfobject.com/pdf/sample.pdf",
                        "mime_type": "application/pdf"
                    }
                }
            ]
        }
    ]
}

pdf base64

{
    "model": "claude-3-5-sonnet-20240620",
    "max_tokens": 200,
    "stream": false,
    "messages": [
        {
            "role": "system",
            "content": "Say Hi"
        },
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "what is in this pdf"
                },
                {
                    "type": "file",
                    "file": {
                        "mime_type": "application/pdf",
                        "file_data": "BASE_64_PDF"
                    }
                }
            ]
        }
    ]
}

plain text document

{
    "model": "claude-3-5-sonnet-20240620",
    "max_tokens": 200,
    "stream": false,
    "messages": [
        {
            "role": "system",
            "content": "Say Hi"
        },
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "what is in this pdf"
                },
                {
                    "type": "file",
                    "file": {
                        "mime_type": "text/plain",
                        "file_data": "hello how are you sir"
                    }
                }
            ]
        }
    ]
}

matter-code-review · 2025-05-06T14:36:11Z

Summary By MatterAI

🔄 What Changed

Added support for PDF and plain text document handling in Anthropic and Bedrock providers. The PR introduces new interfaces for PDF content items (URL and Base64) and plain text content items, along with helper functions to transform and append file content items to messages.

🔍 Impact of the Change

Enables the system to process and send PDF documents and plain text files to AI models through both direct URLs and base64-encoded data. Different providers have varying levels of support: Vertex Anthropic doesn't support plain text and PDF URLs, while Bedrock only supports S3 URLs and doesn't support plain text.

📁 Total Files Changed

3 files modified with 109 additions and 3 deletions:

src/providers/anthropic/chatComplete.ts: Added PDF and text document handling
src/providers/bedrock/chatComplete.ts: Added document handling for Bedrock
src/types/requestBody.ts: Extended ContentType interface with file properties

🧪 Test Added

N/A - No tests were added in this PR, though sample request bodies were provided for manual testing.

🔒Security Vulnerabilities

N/A - No security vulnerabilities identified.

Type of Change

New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
Documentation update
Refactoring (no functional changes)

How Has This Been Tested?

Manual Testing
Unit Tests
Integration Tests

Checklist

My code follows the style guidelines of this project
I have performed a self-review of my own code
I have added tests that prove my fix is effective or that my feature works
I have made corresponding changes to the documentation

Sequence Diagram

sequenceDiagram
    participant Client
    participant API
    participant AnthropicProvider
    participant BedrockProvider
    
    Client->>API: POST /chat/completions with file content
    Note over Client,API: Content can include PDF or text files
    
    API->>AnthropicProvider: Process request with file content
    Note over AnthropicProvider: transformAndAppendFileContentItem()
    
    alt PDF URL
        AnthropicProvider-->>AnthropicProvider: Create document with URL source
        Note over AnthropicProvider: {type: 'document', source: {type: 'url', url: item.file.file_url}}
    else PDF Base64
        AnthropicProvider-->>AnthropicProvider: Create document with base64 source
        Note over AnthropicProvider: {type: 'document', source: {type: 'base64', data: item.file.file_data}}
    else Plain Text
        AnthropicProvider-->>AnthropicProvider: Create document with text source
        Note over AnthropicProvider: {type: 'document', source: {type: 'text', data: item.file.file_data}}
    end
    
    API->>BedrockProvider: Process request with file content
    Note over BedrockProvider: getMessageContent()
    
    alt PDF URL (S3)
        BedrockProvider-->>BedrockProvider: Create document with s3Location
        Note over BedrockProvider: {document: {format: fileFormat, source: {s3Location: {uri: item.file.file_url}}}}
    else PDF Base64
        BedrockProvider-->>BedrockProvider: Create document with bytes
        Note over BedrockProvider: {document: {format: fileFormat, source: {bytes: item.file.file_data}}}
    end
    
    AnthropicProvider-->>API: Return transformed request for Anthropic API
    BedrockProvider-->>API: Return transformed request for Bedrock API
    API-->>Client: Return model response

matter-code-review · 2025-05-07T10:21:16Z

Important

PR Review Skipped

PR review skipped as per the configuration setting. Run a manually review by commenting /matter review

💡Tips to use Matter AI

Command List

/matter summary: Generate AI Summary for the PR
/matter review: Generate AI Reviews for the PR
/matter release-notes: Generate AI release-notes for the PR
/matter <ask-question>: Chat with your PR with Matter AI Agent
/matter remember <recommendation>: Generate AI memories for the PR

matter-code-review · 2025-05-07T10:21:36Z

Summary By MatterAI

🔄 What Changed

This PR adds support for PDF and plain text document handling in Anthropic and Bedrock providers. It introduces new interfaces for PDF content items (URL-based and base64-encoded) and plain text content items, along with transformation functions to process these file types appropriately for each provider.

🔍 Impact of the Change

This enhancement allows the system to handle PDF documents and plain text files in addition to images, expanding the types of content that can be processed by the AI models. The implementation supports both URL-based files and directly encoded file data.

📁 Total Files Changed

3 files modified
109 lines added, 3 lines deleted
Files changed: src/providers/anthropic/chatComplete.ts, src/providers/bedrock/chatComplete.ts, src/types/requestBody.ts

🧪 Test Added

Manual testing was performed with Anthropic, Vertex Anthropic, and Bedrock Anthropic providers using various test cases:

PDF URL testing
PDF base64 encoded data testing
Plain text document testing

🔒Security Vulnerabilities

No security vulnerabilities identified.

Motivation

To enhance the system's capability to process different file types, specifically PDF documents and plain text files, across multiple AI providers.

Type of Change

New feature (non-breaking change which adds functionality)

How Has This Been Tested?

Manual Testing
- Tested with Anthropic, Vertex Anthropic, and Bedrock Anthropic providers
- Tested PDF URL, PDF base64, and plain text document scenarios

Screenshots (if applicable)

N/A

Checklist

My code follows the style guidelines of this project
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
My changes generate no new warnings
I have added tests that prove my fix is effective or that my feature works
New and existing unit tests pass locally with my changes

Related Issues

N/A

Sequence Diagram

sequenceDiagram
    participant Client
    participant API
    participant AnthropicProvider
    participant BedrockProvider
    
    Client->>API: POST /chat/completions
    Note over Client,API: Request with file content (PDF/text)
    
    API->>AnthropicProvider: transformAndAppendFileContentItem(item, transformedMessage)
    Note over AnthropicProvider: Process file content based on type
    
    alt File URL provided
        AnthropicProvider-->>AnthropicProvider: Create document with URL source
    else File data provided
        AnthropicProvider-->>AnthropicProvider: Determine content type (text/base64)
        AnthropicProvider-->>AnthropicProvider: Create document with data source
    end
    
    API->>BedrockProvider: getMessageContent(message)
    Note over BedrockProvider: Process file content for Bedrock
    
    alt File URL provided
        BedrockProvider-->>BedrockProvider: Create document with s3Location source
    else File data provided
        BedrockProvider-->>BedrockProvider: Create document with bytes source
    end
    
    alt Anthropic Provider Selected
        API-->>AnthropicProvider: Process request with file content
        AnthropicProvider-->>Client: Return response
    else Bedrock Provider Selected
        API-->>BedrockProvider: Process request with file content
        BedrockProvider-->>Client: Return response
    end

narengogi added 2 commits April 30, 2025 19:19

add support for anthropic pdf

2eed829

file content part in request body for bedrock

646d3b9

narengogi requested a review from VisargD April 30, 2025 14:27

narengogi changed the title ~~add support for anthropic pdf~~ anthropic pdf Apr 30, 2025

changes per comments

bd40d81

VisargD approved these changes May 7, 2025

View reviewed changes

Merge branch 'main' into chore/anthropic-pdf

f80c462

VisargD merged commit cd9de4a into Portkey-AI:main May 7, 2025
1 check passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

anthropic pdf #1074

anthropic pdf #1074

narengogi commented Apr 30, 2025 •

edited

Loading

VisargD commented May 5, 2025

narengogi commented May 5, 2025

matter-code-review bot commented May 6, 2025

matter-code-review bot commented May 7, 2025

PR Review Skipped

Command List

matter-code-review bot commented May 7, 2025

anthropic pdf #1074

anthropic pdf #1074

Conversation

narengogi commented Apr 30, 2025 • edited Loading

Author Description

Summary By MatterAI

🔄 What Changed

🔍 Impact of the Change

📁 Total Files Changed

🧪 Test Added

🔒 Security Vulnerabilities

Type of Change

How Has This Been Tested?

Screenshots (if applicable)

Checklist

Related Issues

Quality Recommendations

Sequence Diagram

VisargD commented May 5, 2025

narengogi commented May 5, 2025

matter-code-review bot commented May 6, 2025

Summary By MatterAI

🔄 What Changed

🔍 Impact of the Change

📁 Total Files Changed

🧪 Test Added

🔒Security Vulnerabilities

Type of Change

How Has This Been Tested?

Checklist

Sequence Diagram

matter-code-review bot commented May 7, 2025

PR Review Skipped

Command List

matter-code-review bot commented May 7, 2025

Summary By MatterAI

🔄 What Changed

🔍 Impact of the Change

📁 Total Files Changed

🧪 Test Added

🔒Security Vulnerabilities

Motivation

Type of Change

How Has This Been Tested?

Screenshots (if applicable)

Checklist

Related Issues

Sequence Diagram

narengogi commented Apr 30, 2025 •

edited

Loading