🔍 AI-Powered Code Search System (Language Agnostic)

Transform how you navigate codebases with intelligent, natural language search that actually understands your code, regardless of the (functional) programming language.

This system revolutionizes code exploration by combining AI-powered semantic understanding with lightning-fast vector search. Instead of hunting through files with keyword searches, you can ask questions like "How do we handle user authentication?" or "Show me database connection patterns" and get precise, context-aware results across your entire codebase. Designed to be language agnostic, it intelligently processes and extracts structured components like classes, interfaces, enums, and their methods from various programming languages.

Built with Clean Architecture principles, the system provides multi-dimensional search across files, classes/interfaces, and individual methods, each optimized with specialized AI-generated embeddings for maximum relevance and speed.

---

🔄 The Intelligence Pipeline: From Code to Understanding

Phase 1: Intelligent Indexing - Complete Process Breakdown

The system transforms your raw codebase into a searchable knowledge graph through a sophisticated 4-stage pipeline that creates three distinct types of optimized embeddings for maximum search precision:

graph TD
    A[Source Files] --> B[File Discovery]
    B --> C[Content Loading]
    C --> D[File Embedding Generation]
    D --> E[AI Structure Analysis]
    E --> F[Member Extraction]
    F --> G[Member Embedding Generation]
    G --> H[Method Extraction]
    H --> I[Method Embedding Generation]
    I --> J[Vector Storage]

    K[GPT-4 Analysis] --> E
    L[OpenAI Embeddings] --> D
    L --> G
    L --> I

Loading

🔍 Stage 1: File Discovery & Content Processing

File Discovery Process:

# Supported file types with language detection
supported_extensions = ['.py', '.cs', '.ts', '.js', '.java', '.cpp']

# Recursive directory scanning
for file_path in project_directory.rglob('*'):
    if file_path.suffix in supported_extensions:
        content = read_with_utf8_encoding(file_path)
        generate_sha256_hash(content)  # For change detection

Smart Resume Capability:

• Generates SHA-256 content hashes for each file
• Compares against existing index to skip unchanged files
• Typically skips 90%+ of files on subsequent indexing runs
• Dramatically reduces re-indexing time for large projects

🧮 Stage 2: File-Level Embedding Generation

Each file gets a specialized embedding optimized for file-level search:

Embedding Strategy for Files:

# OPTIMIZED: Focused summary with key classes
filename = extract_filename_without_extension(file_path)
embedding_text = f"File {filename}"

# Add key classes info if available from parsing
if parsed_members:
    class_descriptions = []
    for member in top_3_members:
        class_descriptions.append(f"{member.name} {member.type}")
    embedding_text += f": contains {', '.join(class_descriptions)}"
else:
    # Fallback: use first 200 chars for context
    content_preview = file_content[:200].replace('\n', ' ').strip()
    embedding_text += f": {content_preview}"

# Generate 3072-dimensional embedding vector
file_embedding = await openai_embedding_service.generate(embedding_text)

Example File Embedding Texts:

"File UserService: contains UserService class, DatabaseConnection class"
"File AuthenticationManager: contains AuthManager class, TokenValidator class, LoginHandler class"
"File EmailValidator: validates email addresses and formats"

🏗️ Stage 3: AI-Powered Code Structure Analysis

GPT-4 Code Parsing Process:

# AI parsing prompt optimized for structured extraction
parsing_prompt = f"""
Analyze this {language} code and extract all classes, interfaces, and enums.
Return JSON with exact structure:
{{
  "members": [
    {{
      "type": "class|interface|enum",
      "name": "MemberName",
      "summary": "Clear description of purpose",
      "methods": [
        {{"name": "methodName", "summary": "Method description"}}
      ]
    }}
  ]
}}

Source Code:
{file_content}
"""

# GPT-4 analysis via Semantic Kernel
structured_result = await gpt4_service.analyze(parsing_prompt)

AI Extraction Process:

1. Language Detection: Automatically identifies programming language from file extension
2. Structure Analysis: Identifies classes, interfaces, enums, and their relationships
3. Method Discovery: Extracts all public methods and functions with signatures
4. Summary Generation: Creates concise, human-readable descriptions for each element
5. JSON Validation: Ensures structured output meets schema requirements

🎯 Stage 4: Member-Level Embedding Generation

Each class, interface, and enum gets a name-first embedding optimized for architectural search:

Embedding Strategy for Members:

# NAME-FIRST approach for better search relevance
member_embedding_text = f"{member.name} {member.type}"

# Add concise description from AI summary
if member.summary:
    # Take first sentence or first 100 chars
    summary_part = member.summary.split('.')[0][:100]
    member_embedding_text += f" - {summary_part}"

# Add method context for richer understanding
if member.methods:
    method_count = len(member.methods)
    key_methods = [method.name for method in member.methods[:2]]
    member_embedding_text += f" with {method_count} methods including {', '.join(key_methods)}"

# Generate embedding
member_embedding = await openai_embedding_service.generate(member_embedding_text)

Example Member Embedding Texts:

"UserService class - Service for managing user operations with 5 methods including GetUserByIdAsync, CreateUserAsync"
"IUserRepository interface - Repository interface for user data access with 4 methods including GetByIdAsync, CreateAsync"
"UserRole enum - Enumeration defining user permission levels"

⚙️ Stage 5: Method-Level Embedding Generation

Each method gets a context-aware embedding optimized for implementation search:

Embedding Strategy for Methods:

# Method name with class context
method_embedding_text = f"{method.name} method"

# Add parent class context for disambiguation
if parent_member:
    method_embedding_text += f" in {parent_member.name}"

# Add concise purpose description
if method.summary:
    # First sentence or first 80 chars
    summary_part = method.summary.split('.')[0][:80]
    method_embedding_text += f" - {summary_part}"

# Generate embedding
method_embedding = await openai_embedding_service.generate(method_embedding_text)

Example Method Embedding Texts:

"GetUserByIdAsync method in UserService - Retrieves a user by their unique identifier"
"ValidateEmail method in EmailValidator - Validates email format using regex pattern"
"CreateAsync method in UserRepository - Creates a new user record in database"

💾 Stage 6: Multi-Index Vector Storage

The system stores embeddings in three separate optimized vector stores:

# Separate vector stores for each embedding type
stores = {
    'files': SimpleVectorStore(workspace + '/files'),
    'members': SimpleVectorStore(workspace + '/members'),
    'methods': SimpleVectorStore(workspace + '/methods')
}

# Each store optimized for its data type
await stores['files'].store_file_index(file_with_embedding)
await stores['members'].store_code_member(member_with_embedding)
await stores['methods'].store_code_method(method_with_embedding)

Storage Optimization:

• Custom Vector Store: Built on numpy/pickle for Python 3.13 compatibility
• Memory Efficiency: Lazy loading with intelligent caching
• Change Detection: Content hash-based duplicate prevention
• Resume Capability: Only processes changed files on re-indexing

---

Phase 2: Multi-Index Semantic Search Engine

The search engine operates across three parallel vector databases simultaneously, each optimized for different granularities of code search:

graph TD
    A[Natural Language Query] --> B[Query Embedding Generation]
    B --> C[Parallel Multi-Index Search]

    C --> D[File Index Search]
    C --> E[Member Index Search]
    C --> F[Method Index Search]

    D --> G[File Results]
    E --> H[Member Results]
    F --> I[Method Results]

    G --> J[Result Aggregation]
    H --> J
    I --> J

    J --> K[Similarity Scoring]
    K --> L[Cross-Index Ranking]
    L --> M[Unified Result Set]

Loading

🎯 Multi-Index Search Strategy

Parallel Search Execution:

async def search_by_embedding(self, embedding: List[float], config: SearchConfiguration):
    # Execute searches across all three indexes simultaneously
    search_tasks = []

    if config.use_files:
        search_tasks.append(
            self.vector_store.search_files(
                embedding,
                limit=config.max_results_per_type,
                threshold=config.similarity_threshold
            )
        )

    if config.use_members:
        search_tasks.append(
            self.vector_store.search_members(
                embedding,
                limit=config.max_results_per_type,
                threshold=config.similarity_threshold
            )
        )

    if config.use_methods:
        search_tasks.append(
            self.vector_store.search_methods(
                embedding,
                limit=config.max_results_per_type,
                threshold=config.similarity_threshold
            )
        )

    # Execute all searches in parallel
    results = await asyncio.gather(*search_tasks)

    # Combine and rank results
    return self._aggregate_and_rank_results(results)

📊 Advanced Similarity Scoring

Cosine Similarity Calculation:

def cosine_similarity(query_vector: List[float], code_vector: List[float]) -> float:
    """
    Calculate semantic similarity between query and code embeddings
    Returns: 0.0 (no similarity) to 1.0 (identical meaning)
    """
    q_array = np.array(query_vector)
    c_array = np.array(code_vector)

    dot_product = np.dot(q_array, c_array)
    q_norm = np.linalg.norm(q_array)
    c_norm = np.linalg.norm(c_array)

    return dot_product / (q_norm * c_norm)

Multi-Index Result Aggregation:

def aggregate_results(file_results, member_results, method_results):
    """
    Combine results from all three indexes with intelligent ranking
    """
    all_results = []

    # Add context metadata to each result type
    for result in file_results:
        result.context_type = "file"
        result.search_scope = "entire file content"

    for result in member_results:
        result.context_type = "member"
        result.search_scope = "class/interface definition"

    for result in method_results:
        result.context_type = "method"
        result.search_scope = "individual method implementation"

    # Combine and sort by similarity score
    all_results.extend(file_results + member_results + method_results)
    return sorted(all_results, key=lambda x: x.score, reverse=True)

🔧 Search Configuration Flexibility

Granular Search Control:

# Search only at file level (architectural overview)
config = SearchConfiguration(
    use_files=True, use_members=False, use_methods=False,
    similarity_threshold=0.7, max_results_per_type=10
)

# Search only classes/interfaces (structural patterns)
config = SearchConfiguration(
    use_files=False, use_members=True, use_methods=False,
    similarity_threshold=0.6, max_results_per_type=15
)

# Search only methods (implementation details)
config = SearchConfiguration(
    use_files=False, use_members=False, use_methods=True,
    similarity_threshold=0.8, max_results_per_type=5
)

# Full multi-dimensional search (comprehensive results)
config = SearchConfiguration(
    use_files=True, use_members=True, use_methods=True,
    similarity_threshold=0.5, max_results_per_type=8
)

Threshold Tuning Guide:

• 0.8-1.0: High precision, very specific matches
• 0.6-0.8: Balanced precision and recall
• 0.3-0.6: High recall, broader matches
• 0.1-0.3: Exploratory search, maximum coverage

---

💡 Example Searches

Multi-Index Search Results Explained

When you search for "user authentication", the system searches across all three indexes and returns different types of results:

🔍 Query: "user authentication"
📊 Found 12 results across 3 dimensions in 287ms

📄 FILE: AuthenticationService.cs (score: 0.892)
   📁 Services/Authentication/
   🔍 Scope: Entire file focused on auth functionality
   📝 Contains: AuthenticationService class, TokenValidator class

🏗️ CLASS: UserAuthenticationManager (score: 0.847)
   📁 Auth/Manager.cs
   🔍 Scope: Class definition and structure
   📝 Manages user authentication with 6 methods including AuthenticateAsync, ValidateCredentials

⚙️ METHOD: AuthenticateUserAsync (score: 0.823)
   📁 Services/UserService.cs → UserService
   🔍 Scope: Specific method implementation
   📝 Authenticates user credentials against active directory

Natural Language Queries with Multi-Index Results

# Architectural exploration
"How do we handle database connections?"
→ FILE: DatabaseService.cs (entire connection architecture)
→ CLASS: IConnectionManager (connection interface definition)
→ METHOD: GetConnectionAsync() (specific connection logic)

# Pattern discovery across granularities
"async repository methods"
→ FILE: UserRepository.cs (complete repository implementation)
→ CLASS: IAsyncRepository<T> (repository interface pattern)
→ METHOD: GetByIdAsync() (individual async method)

# Error handling analysis
"exception handling patterns"
→ FILE: GlobalExceptionHandler.cs (application-wide error handling)
→ CLASS: CustomException (exception class definitions)
→ METHOD: HandleExceptionAsync() (specific error handling logic)

Search Scope Optimization

# File-level search (architectural understanding)
search --files-only "authentication system"
→ Returns: Complete authentication modules and services

# Class-level search (structural patterns)
search --classes-only "repository pattern"
→ Returns: Repository interfaces and implementations

# Method-level search (implementation details)
search --methods-only "email validation"
→ Returns: Specific validation method implementations

---

🎯 Key Features

Feature	Description	Technical Implementation
🤖 AI-Powered Understanding	GPT-4 analyzes code structure and meaning	Semantic Kernel + OpenAI API integration
⚡ Lightning-Fast Multi-Index Search	Sub-second search across three specialized indexes	Custom vector stores with numpy optimizations
🎯 Three-Dimensional Results	Search files, classes, and methods simultaneously	Specialized embeddings for each code granularity
🔄 Smart Resume Indexing	Only processes changed files on re-indexing	SHA-256 content hashing with change detection
📊 Configurable Precision	Adjust similarity thresholds and result scope	Granular search configuration per index type
🗂️ Intelligent Caching	Avoids redundant API calls and computations	Multi-level caching (embeddings + results)

Multi-Index Search Capabilities

🔤 Cross-Index Text Search: Natural language queries across all three dimensions

search_service.search_by_text("user authentication", SearchConfiguration(
    use_files=True, use_members=True, use_methods=True
))

🔄 Similar Code Search: Find patterns across different code granularities

search_service.search_similar_code("async def get_user(id):", config)
# Returns: Similar files, classes with async methods, matching method signatures

🎯 Selective Index Search: Focus on specific code dimensions

# Architecture-focused search (files + classes only)
config = SearchConfiguration(use_files=True, use_members=True, use_methods=False)

# Implementation-focused search (methods only)
config = SearchConfiguration(use_files=False, use_members=False, use_methods=True)

📝 Context-Enhanced Search: Improve accuracy with additional context

search_service.search_with_context("validation", "user input validation", config)
# Searches across files (validation modules), classes (validator classes), methods (validate methods)

---

🏗️ Architecture

The system implements Clean Architecture principles with clear separation of concerns and dependency inversion:

src/code_search/
├── 🏛️ domain/                    # Core Business Logic
│   ├── models/                   # FileIndex, CodeMember, SearchResult
│   ├── interfaces/               # Service contracts (ISearchService, IVectorStore)
│   └── exceptions/               # Domain-specific exceptions
│
├── 🎯 application/               # Use Cases & Orchestration
│   ├── services/                 # SearchService, CodeIndexer
│   └── commands/                 # CQRS command handlers
│
├── 🔧 infrastructure/            # External Dependencies
│   ├── ai/                       # OpenAI + Semantic Kernel integrations
│   ├── storage/                  # Multi-index vector stores + file operations
│   ├── parsing/                  # Code structure analysis
│   └── configuration/            # Settings + dependency injection
│
└── 🖥️ presentation/              # User Interfaces
    ├── cli/                      # Command-line interface
    ├── api/                      # REST API (planned)
    └── web/                      # Web interface (planned)

Core Components Deep Dive

🏛️ Domain Layer

• Models: Immutable data structures with built-in validation and hash generation
• Interfaces: Service contracts enabling dependency inversion and testing
• Business Rules: Embedded in domain objects (e.g., content hash generation)

🎯 Application Layer

• SearchService: Orchestrates multi-dimensional search across three vector indexes
• CodeIndexer: Manages the complete indexing pipeline with three-stage embedding generation
• Command Handlers: CQRS pattern for clean separation of read/write operations

🔧 Infrastructure Layer

• AI Services: OpenAI embedding generation + GPT-4 code parsing via Semantic Kernel
• Multi-Index Vector Storage: Three specialized vector stores optimized for different code granularities
• Configuration: Flexible JSON + environment variable configuration system

---

⚙️ Technical Implementation

Multi-Index AI Integration Architecture

graph TD
    A[OpenAI Embeddings<br/>text-embedding-3-large] --> B[Semantic Kernel<br/>Orchestration]
    B --> C[GPT-4 Code Analysis<br/>gpt-4o]
    C --> D[Three-Dimensional<br/>Structured Extraction]
    D --> E[Specialized Vector Storage]

    E --> F[Files Index]
    E --> G[Members Index]
    E --> H[Methods Index]

    style A fill:#e1f5fe
    style B fill:#f3e5f5
    style C fill:#e8f5e8
    style D fill:#fff3e0
    style E fill:#fce4ec
    style F fill:#e0f2f1
    style G fill:#f1f8e9
    style H fill:#fff8e1

Loading

Multi-Index Embedding Strategy: Each code element type uses specialized embedding text optimized for its search context:

# FILE EMBEDDINGS: Architectural overview and key components
f"File {filename}: contains {primary_classes} with {core_functionality}"
# Example: "File UserService: contains UserService class, DatabaseConnection class"

# MEMBER EMBEDDINGS: Name-first with structural context
f"{class_name} {class_type} - {purpose_summary} with {method_count} methods including {key_methods}"
# Example: "UserService class - manages user operations with 5 methods including GetUserAsync, CreateUser"

# METHOD EMBEDDINGS: Implementation-focused with class context
f"{method_name} method in {class_name} - {specific_functionality}"
# Example: "GetUserAsync method in UserService - retrieves user by ID from database"

Multi-Index Vector Storage Performance

• Storage Architecture: Three separate numpy-based vector stores for optimal search performance
• Index Specialization: Each index optimized for its code granularity (files vs classes vs methods)
• Search Algorithm: Parallel cosine similarity across all active indexes with intelligent result merging
• Memory Management: Lazy loading per index with cross-index caching
• Scalability: Handles 10,000+ code elements per index with sub-second search times

Resume & Multi-Index Caching System

# Smart change detection across all three indexes
for each_file in project:
    if file_hash_unchanged:
        skip_all_embeddings()  # Skip file, member, AND method embeddings
    else:
        regenerate_file_embedding()

        for each_member in file:
            if member_hash_unchanged:
                skip_member_and_method_embeddings()
            else:
                regenerate_member_embedding()

                for each_method in member:
                    if method_hash_unchanged:
                        skip_method_embedding()
                    else:
                        regenerate_method_embedding()

---

⚙️ Configuration

Multi-Index Search Configuration

Control search behavior across all three vector indexes:

SearchConfiguration(
    # Index selection
    use_files=True,              # Search file-level embeddings
    use_members=True,            # Search class/interface embeddings
    use_methods=True,            # Search method-level embeddings

    # Result control per index
    max_results_per_type=10,     # Limit results from each index
    similarity_threshold=0.7,    # Minimum similarity score across all indexes

    # Advanced options
    cross_index_ranking=True,    # Rank results across all indexes
    preserve_index_context=True  # Include index type in results
)

Index-Specific Configuration Examples:

# Architecture exploration (files + classes)
architectural_config = SearchConfiguration(
    use_files=True, use_members=True, use_methods=False,
    similarity_threshold=0.6, max_results_per_type=15
)

# Implementation deep-dive (methods only)
implementation_config = SearchConfiguration(
    use_files=False, use_members=False, use_methods=True,
    similarity_threshold=0.8, max_results_per_type=20
)

# Comprehensive analysis (all indexes with balanced results)
comprehensive_config = SearchConfiguration(
    use_files=True, use_members=True, use_methods=True,
    similarity_threshold=0.5, max_results_per_type=8
)

Flexible Configuration System

The system supports multiple configuration sources with priority: Environment Variables > JSON File > Defaults

JSON Configuration (config/app_config.json):

{
  "vector_store_path": "./vectordb_workspace",
  "embedding_model": "text-embedding-3-large",
  "ai_model": "gpt-4o",
  "similarity_threshold": 0.3,
  "max_search_results": 10,
  "supported_extensions": [".py", ".cs", ".ts", ".js", ".java", ".cpp"],
  "batch_size": 100,
  "cache_size": 1000
}

Environment Variable Override:

export VECTOR_STORE_PATH="./my_vectordb"
export SIMILARITY_THRESHOLD="0.7"  # Higher precision across all indexes
export MAX_SEARCH_RESULTS="20"     # More results per index

---

🚀 Usage Examples

Multi-Index Search Patterns

Comprehensive Multi-Dimensional Search:

# Search across all three dimensions simultaneously
results = await search_handler.handle_text_search(
    SearchCodeCommand(
        query="user authentication",
        use_files=True,      # Find authentication modules
        use_members=True,    # Find auth classes and interfaces
        use_methods=True,    # Find specific auth methods
        max_results_per_type=5,
        similarity_threshold=0.6
    )
)

# Results contain mixed types with context
for result in results.all_results:
    print(f"{result.type.upper()}: {result.name} (score: {result.score:.3f})")
    print(f"  Context: {result.search_scope}")
    print(f"  Location: {result.file_path}")
    if result.type == "method":
        print(f"  Parent: {result.member_name}")

Index-Specific Search Strategies:

# 1. Architectural Discovery (Files + Classes)
architectural_search = SearchCodeCommand(
    query="database access layer",
    use_files=True,       # Database service files
    use_members=True,     # Repository classes, DB context classes
    use_methods=False,    # Skip method-level details
    similarity_threshold=0.7
)

# 2. Implementation Analysis (Methods Only)
implementation_search = SearchCodeCommand(
    query="async database queries",
    use_files=False,      # Skip file-level results
    use_members=False,    # Skip class-level results
    use_methods=True,     # Focus on method implementations
    similarity_threshold=0.8,
    max_results_per_type=15
)

# 3. Pattern Recognition (Classes + Methods)
pattern_search = SearchCodeCommand(
    query="repository pattern implementation",
    use_files=False,      # Skip files (too broad)
    use_members=True,     # Repository classes and interfaces
    use_methods=True,     # Repository method patterns
    similarity_threshold=0.65
)

Advanced Multi-Index Workflows:

async def analyze_authentication_system():
    """
    Multi-stage analysis using different index combinations
    """

    # Stage 1: Find authentication modules (file-level)
    modules = await search_handler.handle_text_search(
        SearchCodeCommand(
            query="authentication system",
            use_files=True, use_members=False, use_methods=False
        )
    )

    # Stage 2: Analyze authentication classes (member-level)
    classes = await search_handler.handle_text_search(
        SearchCodeCommand(
            query="user authentication validation",
            use_files=False, use_members=True, use_methods=False
        )
    )

    # Stage 3: Find specific auth methods (method-level)
    methods = await search_handler.handle_text_search(
        SearchCodeCommand(
            query="authenticate user credentials",
            use_files=False, use_members=False, use_methods=True
        )
    )

    return {
        'architecture': modules.file_results,
        'classes': classes.member_results,
        'implementations': methods.method_results
    }

Performance Optimization Across Indexes

# Multi-index cache management
search_service.clear_embedding_cache()  # Clear shared embedding cache

# Index-specific statistics
stats = await vector_store.get_statistics()
print(f"Files indexed: {stats['total_files']}")
print(f"Classes indexed: {stats['total_members']}")
print(f"Methods indexed: {stats['total_methods']}")

# Optimize search performance per index
fast_config = SearchConfiguration(
    max_results_per_type=3,    # Fewer results per index
    similarity_threshold=0.8   # Higher threshold for faster search
)

---

📊 Performance Characteristics

Multi-Index Performance Metrics

• Indexing Speed: 1-5 files/second creating 3 embeddings per code element
• Search Speed: 100-500ms across all three indexes simultaneously
• Memory Usage: ~3MB per 100 indexed files (across all three indexes)
• Storage Efficiency: ~36KB total per code element (12KB × 3 indexes)
• Resume Efficiency: Skips 90%+ unchanged embeddings across all indexes
• Cache Hit Rate: 80%+ for repeated searches with shared embedding caching

Index-Specific Performance

Index Type	Elements/File	Search Speed	Memory/Element	Use Case
Files	1 per file	50-100ms	~12KB	Architectural exploration
Members	1-10 per file	75-150ms	~12KB	Structural pattern discovery
Methods	5-50 per file	150-300ms	~12KB	Implementation analysis

Scalability Testing Results:

• 10,000 files: 30,000 members, 150,000 methods - Search time: ~400ms
• 1,000 files: 3,000 members, 15,000 methods - Search time: ~120ms
• 100 files: 300 members, 1,500 methods - Search time: ~50ms

---

Built with ❤️ for developers who believe code search should be as intelligent and multi-dimensional as the code itself.