Build list of models from CAPI endpoint on a scheduled action run

[dfosco]

The list of models is currently static on the HTML. We should make an authenticated request from an actions workflow to fetch the updated list of models (and their free/paid tiers) and run it on a recurring schedule, since this repo is unlikely to get a lot of updates.

PRD for Copilot Agent

Missing Piece

The details about integrating with CAPI (Copilot API) are missing from this PRD, and unknown at the time. Whatever first shot solution is built using this PRD, will have to mock the responses from the Copilot API and build the workflow as an initial PoC until those details are known to the repository or author of this change

Current Static Model Implementation

The static model list is currently implemented in the HTML file at lines 265-274:

<select id="modelSelect" class="form-select width-full mb-2">
  <option value="gpt-4o">GPT-4o (Default)</option>
  <option value="claude-3.5-sonnet">Claude 3.5 Sonnet</option>
  <option value="o3-mini">o3-mini</option>
  <option value="gemini-2.0-flash-001">Gemini 2.0 Flash</option>
  <option value="o1">o1 (Pro)</option>
  <option value="claude-3.7-sonnet">Claude 3.7 Sonnet (Pro)</option>
  <option value="claude-3.7-sonnet-thought">Claude 3.7 Sonnet Thinking (Pro)</option>
  <option value="gpt-4.5-preview">GPT-4.5 (Pro+)</option>
</select>

JavaScript Code Handling Model Selection

The JavaScript code that handles model selection and URL generation:

// Model selection event listener (line 422)
elements.modelSelect.addEventListener('change', updateURL);

// URL update function (lines 428-432)
function updateURL() {
    const encodedPrompt = encodeURIComponent(elements.promptInput.value.trim());
    const model = elements.modelSelect.value;
    elements.urlOutput.value = `https://github.com/copilot?model=${model}&prompt=${encodedPrompt}`;
}

Repository Structure and Context

The repository is a simple static site with:

• index.html - Main application file containing all HTML, CSS, and JavaScript
• Static assets (SVG images for buttons)
• Standard GitHub repository files (README, LICENSE, etc.)
• No existing build system or automation
• No package.json or other dependency management files

Key Technical Considerations

1. Current Architecture: Single-file vanilla HTML/JS application with no build process
2. Model Data Structure: Currently just value/display text pairs, needs to support tier information (free/paid)
3. Static Hosting: Appears to be designed for GitHub Pages or similar static hosting
4. No Existing CI/CD: No .github/workflows directory exists currently

Files That Need Modification

1. index.html - The model select dropdown and JavaScript code
2. New: models.json - Data file to store the dynamic model list
3. New: .github/workflows/update-models.yml - GitHub Actions workflow
4. New: scripts/update-models.js - Script to fetch and update model data

The solution will need to:

• Create a GitHub Actions workflow that runs on a schedule
• Fetch model data from an authenticated API endpoint
• Generate a models.json file with the updated data
• Modify the HTML to load models dynamically from the JSON file
• Maintain the current UI/UX while supporting tier indicators

This represents a significant architectural change from a purely static site to one with automated content updates via GitHub Actions.I'll analyze the repository context and write a detailed technical PRD for implementing dynamic model fetching. Let me first examine the current HTML structure to understand how models are currently handled. Now let me check if there are any existing GitHub Actions workflows in the repository: Let me also search for any existing data files or configuration that might contain model information: Based on my analysis of the repository, I can now write a comprehensive technical PRD for implementing dynamic model fetching. Here's the detailed document:

Technical Product Requirements Document: Dynamic Model Fetching

Executive Summary

The Let's Copilot web application currently has a hardcoded list of GitHub Copilot models in the HTML file. This creates maintenance overhead and potential inconsistencies as new models are added or existing models are modified. This PRD outlines the implementation of an automated system to fetch and update the model list dynamically using GitHub Actions.

Problem Statement

Current State

• Model list is statically defined in index.html (lines 234-241)
• Contains 8 hardcoded models with their tiers (Default, Pro, Pro+)
• Manual updates required when GitHub adds new Copilot models
• Risk of outdated model information affecting user experience
• No indication of model availability or pricing tiers

Pain Points

1. Manual Maintenance: Requires developer intervention for each model update
2. Staleness Risk: Model list may become outdated between manual updates
3. Inconsistency: No single source of truth for model information
4. Scalability: Manual process doesn't scale with frequent model updates

Objectives

Primary Goals

1. Automate model list updates using GitHub Actions
2. Maintain current user experience while improving data freshness
3. Establish a reliable, recurring update mechanism
4. Ensure backward compatibility with existing URL generation

Success Metrics

• Model list updates automatically within 24 hours of GitHub Copilot changes
• Zero manual intervention required for model updates
• Maintain 100% uptime during model list updates
• Preserve existing functionality and user workflows

Technical Requirements

1. Data Source and API Integration

GitHub Copilot Models API

• Requirement: Identify and integrate with GitHub's official Copilot models API
• Fallback: If no public API exists, implement web scraping of GitHub Copilot documentation
• Data Points Needed:
  • Model identifier (e.g., gpt-4o, claude-3.5-sonnet)
  • Display name (e.g., GPT-4o (Default))
  • Availability tier (Default, Pro, Pro+)
  • Model status (active, deprecated, beta)

Authentication

• GitHub Token: Use GITHUB_TOKEN with appropriate permissions
• Scope: Minimal required permissions for API access
• Storage: Secure token storage in GitHub Secrets

2. GitHub Actions Workflow

Workflow Triggers

• Schedule: Daily execution at 00:00 UTC using cron expression
• Manual Trigger: workflow_dispatch for immediate updates
• On-demand: Triggered by repository maintainers when needed

Workflow Steps

1. Fetch Models: Query GitHub Copilot API for current model list
2. Transform Data: Convert API response to required HTML format
3. Update File: Modify index.html with new model options
4. Commit Changes: Auto-commit if changes detected
5. Notification: Report success/failure status

Workflow File Location

.github/workflows/update-models.yml

3. Data Structure and Format

Model Data Schema

{
  "models": [
    {
      "id": "gpt-4o",
      "displayName": "GPT-4o",
      "tier": "default",
      "status": "active",
      "isDefault": true
    },
    {
      "id": "claude-3.5-sonnet", 
      "displayName": "Claude 3.5 Sonnet",
      "tier": "default",
      "status": "active",
      "isDefault": false
    }
  ]
}

HTML Generation

• Generate <option> elements with proper value attributes
• Maintain tier indicators in display names
• Preserve default model selection
• Handle model deprecation gracefully

4. File Management

Target File

• Path: index.html
• Section: Model selection <select> element (lines 233-242)
• Preservation: Maintain surrounding HTML structure

Update Strategy

• Pattern Matching: Use HTML comments to mark update region
• Atomic Updates: Replace entire model section atomically
• Backup: Preserve previous version in case of rollback need

5. Error Handling and Resilience

API Failure Scenarios

• Network Issues: Retry mechanism with exponential backoff
• Rate Limiting: Respect API rate limits and retry after delay
• Authentication Failure: Clear error messaging and notification
• Malformed Response: Validate API response structure

Fallback Mechanisms

• Static Fallback: Maintain current model list if API unavailable
• Partial Updates: Update only successfully fetched models
• Rollback: Ability to revert to previous working state

Monitoring and Alerting

• Workflow Status: Monitor GitHub Actions success/failure
• Log Analysis: Comprehensive logging for debugging
• Notification: Alert maintainers on repeated failures

Implementation Plan

Phase 1: Foundation (Week 1)

1. Research: Investigate GitHub Copilot API availability and documentation
2. Authentication: Set up GitHub token with required permissions
3. Workflow Creation: Create basic GitHub Actions workflow file
4. Testing: Implement dry-run mode for safe testing

Phase 2: Core Implementation (Week 2)

1. API Integration: Implement model fetching logic
2. Data Processing: Transform API response to HTML format
3. File Updates: Implement HTML modification logic
4. Error Handling: Add basic error handling and retries

Phase 3: Enhancement (Week 3)

1. Robust Error Handling: Comprehensive error scenarios
2. Monitoring: Add logging and notification systems
3. Documentation: Update README with new automation details
4. Testing: Thorough testing of all scenarios

Phase 4: Deployment (Week 4)

1. Production Deployment: Enable automated workflow
2. Monitoring Setup: Monitor initial runs for issues
3. Documentation: Final documentation updates
4. Handoff: Knowledge transfer to maintainers

Technical Architecture

System Components

1. GitHub Actions Runner

• Environment: Ubuntu latest runner
• Dependencies: Node.js for API requests and HTML manipulation
• Tools: curl, jq, or Node.js for API interaction

2. Model Fetcher Service

// Pseudo-code structure
class ModelFetcher {
  async fetchModels() {
    // API call to GitHub Copilot
  }
  
  transformToHTML(models) {
    // Convert to HTML options
  }
  
  updateIndexFile(htmlContent) {
    // Update index.html
  }
}

3. File Update Mechanism

• Parser: HTML parsing to locate model section
• Generator: Generate new option elements
• Writer: Atomic file write operation

Data Flow

1. Trigger: Scheduled or manual workflow trigger
2. Authentication: Authenticate with GitHub API
3. Fetch: Retrieve current model list from API
4. Transform: Convert to HTML option format
5. Update: Modify index.html file
6. Commit: Auto-commit changes if detected
7. Notify: Report status to maintainers

Security Considerations

Authentication and Authorization

• Minimal Permissions: Use least-privilege principle for GitHub token
• Token Rotation: Regular rotation of authentication tokens
• Scope Limitation: Restrict token scope to required operations only

Code Security

• Input Validation: Validate all API responses
• Injection Prevention: Sanitize data before HTML insertion
• Dependency Security: Use trusted, updated dependencies

Repository Security

• Branch Protection: Protect main branch from direct pushes
• Review Process: Optional: Require review for model updates
• Audit Trail: Maintain complete audit trail of changes

Monitoring and Maintenance

Success Metrics

• Update Frequency: Models updated within 24 hours of changes
• Success Rate: >95% workflow success rate
• Data Accuracy: 100% accuracy of model information
• Uptime: Zero user-facing downtime during updates

Monitoring Requirements

• Workflow Status: Daily workflow execution monitoring
• Model Accuracy: Periodic validation of model list accuracy
• Performance: Monitor API response times and workflow duration
• Error Tracking: Track and analyze failure patterns

Maintenance Tasks

• Token Renewal: Regular authentication token updates
• Dependency Updates: Keep workflow dependencies current
• Documentation Updates: Maintain accurate documentation
• Performance Optimization: Optimize workflow execution time

Risk Assessment

High-Risk Scenarios

1. API Deprecation: GitHub changes or removes Copilot API

   • Mitigation: Fallback to web scraping or manual updates

2. Authentication Failure: Token expiry or permission changes

   • Mitigation: Automated token validation and renewal

3. Data Corruption: Malformed API response corrupts HTML

   • Mitigation: Response validation and atomic updates

Medium-Risk Scenarios

1. Rate Limiting: API rate limits exceeded

   • Mitigation: Implement proper rate limiting and retries

2. Workflow Failures: GitHub Actions service issues

   • Mitigation: Multiple retry attempts and manual fallback

Low-Risk Scenarios

1. Minor Data Changes: Small formatting differences
   • Mitigation: Flexible parsing and generation logic

Success Criteria

Functional Requirements

• ✅ Models automatically update from authoritative source
• ✅ Existing functionality preserved during updates
• ✅ Updates complete without manual intervention
• ✅ Error handling prevents data corruption

Non-Functional Requirements

• ✅ Workflow executes reliably on schedule
• ✅ Updates complete within 5 minutes
• ✅ Zero impact on user experience during updates
• ✅ Comprehensive logging for troubleshooting

Acceptance Criteria

1. Automated Updates: Model list updates automatically when GitHub adds new models
2. Data Integrity: All model information remains accurate and properly formatted
3. Backward Compatibility: Existing URLs continue to work correctly
4. Error Resilience: System handles API failures gracefully without breaking
5. Monitoring: Clear visibility into workflow status and any issues

Dependencies and Assumptions

External Dependencies

• GitHub Copilot API: Availability of public API or documentation
• GitHub Actions: Continued availability of GitHub Actions service
• Authentication: Stable GitHub token authentication mechanism

Technical Assumptions

• API Stability: GitHub Copilot API structure remains relatively stable
• HTML Structure: Current HTML structure for model selection preserved
• Browser Compatibility: Model updates don't break browser compatibility

Business Assumptions

• Model Frequency: New models added infrequently enough for daily updates
• Maintenance Bandwidth: Team has capacity for initial setup and monitoring
• User Expectation: Users expect model list to be current and accurate

This PRD provides a comprehensive technical roadmap for implementing dynamic model fetching that addresses the current limitations while maintaining the simplicity and reliability of the Let's Copilot application.