Merge pull request #47 from codelion/fix-bugs

Track and persist training history per class

Author: codelion

.github/workflows/test.yml

@@ -0,0 +1,37 @@
+name: Run Tests
+
+on:
+  push:
+    branches: [ main ]
+  pull_request:
+    branches: [ main ]
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    
+    steps:
+    - uses: actions/checkout@v3
+    
+    - name: Set up Python 3.12
+      uses: actions/setup-python@v4
+      with:
+        python-version: '3.12'
+    
+    - name: Install dependencies
+      run: |
+        python -m pip install --upgrade pip
+        pip install -e .
+        pip install pytest pytest-cov psutil
+    
+    - name: Run tests
+      run: |
+        pytest tests/ -v --cov=adaptive_classifier --cov-report=xml --cov-report=term
+    
+    - name: Upload coverage to Codecov
+      uses: codecov/codecov-action@v3
+      with:
+        file: ./coverage.xml
+        flags: unittests
+        name: codecov-umbrella
+        fail_ci_if_error: false

README.md

@@ -388,6 +388,55 @@ This real-world evaluation demonstrates that adaptive classification can signifi
 - [RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models](https://arxiv.org/abs/2401.00396)
 - [LettuceDetect: A Hallucination Detection Framework for RAG Applications](https://arxiv.org/abs/2502.17125)
 
+## Order Dependency in Online Learning
+
+When using the adaptive classifier for true online learning (adding examples incrementally), be aware that the order in which examples are added can affect predictions. This is inherent to incremental neural network training.
+
+### The Challenge
+
+```python
+# These two scenarios may produce slightly different models:
+
+# Scenario 1
+classifier.add_examples(["fish example"], ["aquatic"])
+classifier.add_examples(["bird example"], ["aerial"])
+
+# Scenario 2  
+classifier.add_examples(["bird example"], ["aerial"])
+classifier.add_examples(["fish example"], ["aquatic"])
+```
+
+While we've implemented sorted label ID assignment to minimize this effect, the neural network component still learns incrementally, which can lead to order-dependent behavior.
+
+### Solution: Prototype-Only Predictions
+
+For applications requiring strict order independence, you can configure the classifier to rely solely on prototype-based predictions:
+
+```python
+# Configure to use only prototypes (order-independent)
+config = {
+    'prototype_weight': 1.0,  # Use only prototypes
+    'neural_weight': 0.0      # Disable neural network contribution
+}
+
+classifier = AdaptiveClassifier("bert-base-uncased", config=config)
+```
+
+With this configuration:
+- Predictions are based solely on similarity to class prototypes (mean embeddings)
+- Results are completely order-independent
+- Trade-off: May have slightly lower accuracy than the hybrid approach
+
+### Best Practices
+
+1. **For maximum consistency**: Use prototype-only configuration
+2. **For maximum accuracy**: Accept some order dependency with the default hybrid approach
+3. **For production systems**: Consider batching updates and retraining periodically if strict consistency is required
+4. **Model selection matters**: Some models (e.g., `google-bert/bert-large-cased`) may produce poor embeddings for single words. For better results with short inputs, consider:
+   - `bert-base-uncased`
+   - `sentence-transformers/all-MiniLM-L6-v2`
+   - Or any model specifically trained for semantic similarity
+
 ## Citation
 
 If you use this library in your research, please cite:

setup.py

@@ -15,7 +15,7 @@
 
 setup(
     name="adaptive-classifier",
-    version="0.0.14",
+    version="0.0.15",
     author="codelion",
     author_email="codelion@okyasoft.com",
     description="A flexible, adaptive classification system for dynamic text classification",

src/adaptive_classifier/classifier.py

@@ -66,6 +66,7 @@ def __init__(
 
         # Statistics
         self.train_steps = 0
+        self.training_history = {}  # Track cumulative training examples per class
 
         # Strategic classification components
         self.strategic_cost_function = None
@@ -87,19 +88,24 @@ def add_examples(self, texts: List[str], labels: List[str]):
         new_classes = set(labels) - set(self.label_to_id.keys())
         is_adding_new_classes = len(new_classes) > 0
 
-        # Update label mappings
-        for label in new_classes:
+        # Update label mappings - sort new classes alphabetically for consistent IDs
+        for label in sorted(new_classes):
             idx = len(self.label_to_id)
             self.label_to_id[label] = idx
             self.id_to_label[idx] = label
 
         # Get embeddings for all texts
         embeddings = self._get_embeddings(texts)
 
-        # Add examples to memory
+        # Add examples to memory and update training history
         for text, embedding, label in zip(texts, embeddings, labels):
             example = Example(text, label, embedding)
             self.memory.add_example(example, label)
+            
+            # Update training history
+            if label not in self.training_history:
+                self.training_history[label] = 0
+            self.training_history[label] += 1
 
         # Special handling for new classes
         if is_adding_new_classes:
@@ -118,6 +124,9 @@ def add_examples(self, texts: List[str], labels: List[str]):
             # Strategic training step if enabled
             if self.strategic_mode and self.train_steps % self.config.strategic_training_frequency == 0:
                 self._perform_strategic_training()
+        
+        # Ensure FAISS index is up to date after adding examples
+        self.memory._rebuild_index()
 
     def _train_new_classes(self, old_head: Optional[nn.Module], new_classes: Set[str]):
         """Train the model with focus on new classes while preserving old class knowledge."""
@@ -317,17 +326,21 @@ def _predict_regular(self, text: str, k: int = 5) -> List[Tuple[str, float]]:
         # Combine predictions with adjusted weights
         combined_scores = {}
 
-        # Use neural predictions more for recent classes
+        # Use training history to determine weights
         for label, score in proto_preds:
-            if label in self.memory.examples and len(self.memory.examples[label]) < 10:
-                # For newer classes (fewer examples), trust neural predictions more
+            # Check training history instead of current storage
+            trained_examples = self.training_history.get(label, 0)
+            if trained_examples < 10:
+                # For newer classes (fewer training examples), trust neural predictions more
                 weight = 0.3  # Lower prototype weight for new classes
             else:
                 weight = 0.7  # Higher prototype weight for established classes
             combined_scores[label] = score * weight
 
         for label, score in head_preds:
-            if label in self.memory.examples and len(self.memory.examples[label]) < 10:
+            # Use training history for neural weights too
+            trained_examples = self.training_history.get(label, 0)
+            if trained_examples < 10:
                 weight = 0.7  # Higher neural weight for new classes
             else:
                 weight = 0.3  # Lower neural weight for established classes
@@ -414,6 +427,7 @@ def _save_pretrained(
             'label_to_id': self.label_to_id,
             'id_to_label': {str(k): v for k, v in self.id_to_label.items()},
             'train_steps': self.train_steps,
+            'training_history': self.training_history,  # Save cumulative training counts
             'config': self.config.to_dict()
         }
 
@@ -569,6 +583,9 @@ def _from_pretrained(
             int(k): v for k, v in config_dict['id_to_label'].items()
         }
         classifier.train_steps = config_dict['train_steps']
+        
+        # Restore training history with backward compatibility
+        classifier.training_history = config_dict.get('training_history', {})
 
         # Load tensors
         tensors = load_file(model_path / "model.safetensors")
@@ -600,6 +617,13 @@ def _from_pretrained(
             classifier._initialize_adaptive_head()
             classifier.adaptive_head.load_state_dict(adaptive_head_params)
 
+        # Backward compatibility: estimate training history if not present
+        if not classifier.training_history:
+            for label, examples in saved_examples.items():
+                # Estimate based on saved examples (default saves 5, typical training uses 100+)
+                # Using 20x multiplier as reasonable estimate
+                classifier.training_history[label] = len(examples) * 20
+
         return classifier
 
     def _generate_model_card(self) -> str:
@@ -754,20 +778,15 @@ def _initialize_adaptive_head(self):
         ).to(self.device)
 
     def _get_embeddings(self, texts: List[str]) -> List[torch.Tensor]:
-        """Get embeddings for input texts with improved caching."""
-        # Sort texts for consistent tokenization
-        sorted_indices = list(range(len(texts)))
-        sorted_indices.sort(key=lambda i: texts[i])
-        sorted_texts = [texts[i] for i in sorted_indices]
-        
+        """Get embeddings for input texts."""
         # Temporarily set model to eval mode
         was_training = self.model.training
         self.model.eval()
 
         # Get embeddings
         with torch.no_grad():
             inputs = self.tokenizer(
-                sorted_texts,
+                texts,
                 max_length=self.config.max_length,
                 truncation=True,
                 padding=True,
@@ -784,12 +803,8 @@ def _get_embeddings(self, texts: List[str]) -> List[torch.Tensor]:
         if was_training:
             self.model.train()
 
-        # Restore original order
-        original_order = [0] * len(texts)
-        for i, idx in enumerate(sorted_indices):
-            original_order[idx] = embeddings[i].cpu()
-        
-        return original_order
+        # Return embeddings as list
+        return [emb.cpu() for emb in embeddings]
 
     def get_example_statistics(self) -> Dict[str, Any]:
         """Get statistics about stored examples and model state."""