IGG - Idea Generator Generator

📚 Overview

The IGG (Idea Generator Generator) project generates creative ideas using Markov chains trained on CSV data. It provides both a static web interface and an MCP (Model Context Protocol) server for integration with AI tools like Claude Code.

Key Features

• 🔄 Automatic CSV → Markov Model Processing: Upload CSV files and get trained models
• 🌐 Static Web Interface: Browse and generate ideas from models
• 🔌 MCP Server Integration: Use with Claude Code and other AI tools
• ☁️ AWS CDK Infrastructure: Fully managed serverless deployment
• 📊 Model Caching: Efficient local and remote model storage

🏗️ Architecture

┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   Web Frontend  │    │   MCP Server    │    │ AWS CDK Stack   │
│                 │    │                 │    │                 │
│ - Static Site   │    │ - API Gateway   │    │ - S3 Buckets    │
│ - Model Browse  │    │ - Basic Auth    │    │ - API Gateway   │
│ - Idea Gen      │    │ - Lambda Funcs  │    │ - Lambda Funcs  │
│                 │    │ - Custom Domain │    │ - SSL Certs     │
└─────────────────┘    └─────────────────┘    └─────────────────┘

📂 Project Structure

├── src/                                  # Python source code
│   ├── mcp_server.py                     # MCP server entry point
│   ├── mcp_markov_models.py              # MCP Markov logic  
│   ├── model_processor.py                # Lambda CSV processor
│   └── generate_markov_models.py         # Standalone utility
├── cdk/                                  # AWS CDK infrastructure
│   ├── app.py                            # CDK application
│   ├── custom_constructs/                # Reusable CDK components
│   │   ├── mcp_server_construct.py       # MCP server infrastructure
│   │   ├── static_site_construct.py      # Static site infrastructure
│   │   └── model_processor_construct.py  # Lambda processor infrastructure
│   └── stacks/                           # CDK stack definitions
│       ├── mcp_stack.py                  # MCP server infrastructure
│       └── static_site_stack.py          # Static site infrastructure
├── test/                                 # Unit tests
├── web/                                  # Frontend web interface
│   ├── index.html                        # Static site frontend
│   ├── samples/                          # Sample data files
│   └── script/                           # Frontend JavaScript
├── models/cache/                         # Local model cache
├── lambda-layer/                         # Lambda dependencies layer
└── layerator.py                          # Lambda layer builder script

🚀 Quick Start

Prerequisites

• Python 3.12+ with Pipenv
• Node.js and AWS CDK CLI (for infrastructure)
• AWS CLI configured (for deployment)

1. Local MCP Server Setup

# Clone and install dependencies
git clone <repository-url>
cd igg
pipenv install

# Run MCP server locally
pipenv run python src/mcp_server.py

2. MCP Integration with Claude Code

After deploying the CDK stack, configure your MCP client using the deployed API Gateway endpoint:

{
  "mcpServers": {
    "igg-markov": {
      "url": "https://mcp.yourdomain.com/",
      "headers": {
        "Authorization": "Basic <base64-encoded-credentials>"
      }
    }
  }
}

Getting the MCP endpoint URL and credentials:

# Deploy and get outputs
cd cdk && pipenv run cdk deploy IggMcpStack

# The deployment will output:
# - McpApiGatewayUrl: Direct API Gateway URL (works immediately)
# - McpCustomDomainUrl: Custom domain URL (requires DNS setup)
# - McpAuthSecretArn: Secret ARN for credentials

# Get credentials from AWS Secrets Manager
aws secretsmanager get-secret-value \
  --secret-id <McpAuthSecretArn-from-output> \
  --query SecretString --output text

URL Options:

• Quick setup: Use McpApiGatewayUrl directly (no DNS required)
• Custom domain: Use McpCustomDomainUrl after setting up DNS CNAME record

3. AWS Infrastructure Deployment

# Build Lambda layer with heavy dependencies (pandas, nltk)
pipenv run python layerator.py

# Configure domains in config.json
cd cdk
cp config.json.example config.json
# Edit config.json with your domains and certificates

# Deploy infrastructure
pipenv run cdk deploy --all

# Set up DNS records as shown in deployment outputs

🔧 Configuration

CDK Configuration (cdk/config.json)

{
  "mcp": {
    "domain": "mcp.yourdomain.com",
    "certificateDomain": "mcp.yourdomain.com"
  },
  "static_site": {
    "domain": "static.yourdomain.com", 
    "certificate_arn": "arn:aws:acm:...",
    "bucket_name": "your-bucket-name"
  }
}

Environment Variables

• IGG_BASE_URL: Override default model endpoint for MCP server
• BUCKET_NAME: S3 bucket name (set by CDK)
• INDEX_FILE: Index file name (default: index.json)

🎯 Usage Examples

MCP Tools Available

1. List Models

{
  "tool": "list_models"
}

2. Generate Ideas

{
  "tool": "generate_ideas",
  "arguments": {
    "model_name": "samples/sample.json",
    "count": 5
  }
}

3. Template-based Generation

{
  "tool": "generate_with_template",
  "arguments": {
    "model_name": "samples/sample.json",
    "template": "A $1 solution for $2 professionals",
    "count": 3
  }
}

CSV Processing Workflow

1. Upload CSV: Place CSV file in S3 bucket (any path structure)
2. Auto-Processing: Lambda automatically converts CSV → Markov JSON
3. Index Update: index.json updated with model metadata
4. Access Models: Available via API Gateway and MCP server

🧪 Testing

Run the complete test suite:

pipenv run pytest test/ -v

Test specific modules:

pipenv run pytest test/test_model_processor.py -v
pipenv run pytest test/test_generate_markov_models.py -v

🏗️ Development

Adding New Models

1. Upload CSV data to S3 bucket
2. Models automatically generated and indexed
3. Available immediately via MCP and web interface

Lambda Layer Management

The model processor uses a Lambda layer for heavy dependencies (pandas, nltk, numpy). The layer is built for Python 3.12:

# Rebuild layer when dependencies change
pipenv run python layerator.py

# Deploy updated layer
pipenv run cdk deploy IggStaticSiteStack

Extending MCP Tools

1. Add new methods to mcp_markov_models.py
2. Register tools in mcp_server.py
3. Add corresponding tests

CDK Infrastructure Changes

1. Modify constructs in cdk/custom_constructs/
2. Update stack definitions in cdk/stacks/
3. Test with pipenv run cdk diff
4. Deploy with pipenv run cdk deploy

📊 Monitoring & Debugging

CloudWatch Logs

• Lambda function logs: /aws/lambda/ModelProcessor
• API Gateway logs: Available in CloudWatch

Local Debugging

# Test model processor locally
pipenv run python -c "
import model_processor
result = model_processor.process_csv('col1,col2\nval1,val2', 'test.csv')
print(result)
"

🔒 Security

• All AWS resources use minimal IAM permissions
• API Gateway secured with custom authorizers (MCP stack)
• HTTPS/TLS 1.2+ enforced on all endpoints
• S3 bucket access restricted to API Gateway
• Authentication secrets managed via AWS Secrets Manager

🚢 Deployment

Environment-specific Deployments

# Development
pipenv run cdk deploy --context environment=dev

# Production  
pipenv run cdk deploy --context environment=prod

CI/CD Integration

# Validate infrastructure
pipenv run cdk synth
pipenv run cdk diff

# Deploy specific stacks
pipenv run cdk deploy IggMcpStack
pipenv run cdk deploy IggStaticSiteStack

🧹 Cleanup

Remove all AWS resources:

pipenv run cdk destroy --all

📄 License

This project is licensed under the Apache License 2.0. See the LICENSE file for details.

🤝 Contributing

1. Fork the repository
2. Create a feature branch: git checkout -b feature/amazing-feature
3. Run tests: pipenv run pytest
4. Commit changes: git commit -m 'Add amazing feature'
5. Push to branch: git push origin feature/amazing-feature
6. Open a Pull Request

📧 Support

• Issues: GitHub Issues
• Discussions: GitHub Discussions
• Documentation: See individual component READMEs in subdirectories