draft

.gitignore

@@ -1,6 +1,7 @@
 # Ignore sensitive configuration files and credentials
 .env
 config.toml
+docker-compose.override.yml
 
 # Ignore virtual environments
 venv/

Dockerfile

@@ -1,7 +1,7 @@
 # Dockerfile to create a clean, lightweight Docker Image for the Service Quality Oracle
 
 # Use Python 3.9 slim as the base image for a lightweight container
-FROM python:3.9-slim
+FROM python:3.11-slim
 
 # Add metadata labels
 LABEL description="Service Quality Oracle" \

ELIGIBILITY_CRITERIA.md

@@ -1,42 +1,59 @@
+This document defines the requirements an Indexer must meet to be eligible for indexing rewards. It includes the current active criteria, a schedule of any upcoming changes, and a log of all historical requirements. The goal is to provide a transparent and predictable set of standards for all network participants.
+
+---
+
 # Upcoming Eligibility Criteria
 
-We will announce changes to the eligibility criteria in the table below. Once the change goes live then it will be reflected in the eligibility criteria section of this document.
+**We will announce changes to the eligibility criteria in the table below.** Once the change goes live, it will be reflected in the [Active Eligibility Criteria](https://github.com/graphprotocol/service-quality-oracle/blob/main/ELIGIBILITY_CRITERIA.md#active-eligibility-criteria) section of this document.
 
-| Upcoming Requirement | Justification | Date Updated/Introduced (YYYY-MM-DD)|
-|----------------------|---------------|-------------------------------------|
-| **Requirement 1:** | This is a placeholder for future criteria, watch this space to stay informed. We will also announce any upcoming requirements via our existing official channels. | YYYY-MM-DD |
+| Upcoming Requirement | Justification | Date Requirement Will Be Updated/Introduced (YYYY-MM-DD) |
+|----------------------|---------------|----------------------------------------------------------|
+| **Example Requirement:** | This is a placeholder for future criteria. Watch this space to stay informed. We will also announce upcoming requirements via official channels. | `YYYY-MM-DD` |
 
 > **Note**:
->
-> When announcing new eligibility criteria we will allow a window for indexers to prepare their infrastructure before any new/updated criteria goes live, refer to the `Date Updated/Introduced (YYYY-MM-DD)` column to see when upcoming eligibility criteria will merge.
+> We will typically allow a 14 day window after announcing a change before it goes live.
+
+---
+
+# Active Eligibility Criteria
 
-# Eligibility Criteria
+The following criteria are used to identify indexers that should be eligible to receive indexing rewards.
 
-The Service Quality Oracle determines which indexers are eligible to receive indexing rewards using a threshold rewards algorithm that operates by checking indexers meet the following criteria:
+- **Days Online Requirement:** Indexers must be active for **5+ days** in a given **28 day** period for rewards eligibility.
+- **Daily Query Requirement:** To be active, an indexer must serve at least **1 qualifying query** on **10 different subgraphs**.
+- **Query Quality Requirements:** A qualifying query is one that simultaneously meets **all** of the following criteria:
+  - Query Response HTTP Status: **200 OK**.
+  - Query Response Latency: **< 5,000 ms**.
+  - Query Freshness: **< 50,000 blocks** behind chainhead.
+  - Subgraph Curation Signal: **≥ 500 GRT**.
 
-1. Indexers must be online for 5+ days in a given 28 day rolling period.
-    1. To be online an indexer must serve at least 1 qualifying query on 10 different subgraphs
-        1. A qualifying query is one where:
-            1. The query response HTTP status was 200 OK, indicating query success.
-            2. The query response latency was <5,000 ms.
-            3. The query was served <50,000 blocks behind chainhead.
-            4. The subgraph had at least 500 GRT in curation signal at the time that the query was served.
+Eligibility for indexing rewards is typically refreshed daily via the ServiceQualityOracle contract.
 
 > **Note**:
-> 
-> All four quality criteria must be satisfied simultaneously for a query to count towards the daily requirement.
-> 
-> The above query criteria must be satisfied on 10+ subgraphs per day, for 5+ days in any given 28 day rolling window.
->
-> Issuance eligibility is refreshed daily via the ServiceQualityOracle contract.
->
-> Once an indexer has qualified for issuance via the ServiceQualityOracle contract, they can claim indexing rewards from the protocol for the duration of the qualification period (default is 14 days), even if the requirements change.
-
-
-
-| Requirement | Justification | Date Updated/Introduced (YYYY-MM-DD)|
-|-------------|---------------|-------------------------------------|
-| **Query Status:** The query must have a `200 OK` HTTP response status indicating query success | Indexer infrastructure needs to be capable of serving successful queries to benefit data consumers. | TBD (at genesis of the SQO) |
-| **Query Latency:** The query response must be delivered to the gateway in `< 5,000 ms` | Fast query responses are important to data consumers. | TBD (at genesis of the SQO) |
-| **Query Freshness:** The query must be served from a subgraph that is `< 50,000 blocks` behind chainhead | Data needs to be fresh to be useful to data consumers. | TBD (at genesis of the SQO) |
-| **Subgraph Signal:** The subgraph needs to have `≥ 500 GRT` in curation signal at the time when the query was served. | Indexers are encouraged to serve data on subgraphs that have curation signal. This also creates an economic barrier against those that prefer to game the system. | TBD (at genesis of the SQO) |
+> Once an indexer has successfully qualified for indexing rewards by satisfying the active eligibility criteria, and a corresponding transaction has been submitted on chain by an authorized Oracle into the ServiceQualityOracle contract, the now eligible indexer can continue claiming indexing rewards from the protocol for the duration of the qualification period (default is 14 days), even if the active eligibility criteria change.
+
+---
+
+# Eligibility Requirements Changelog
+
+This table tracks changes to the indexing rewards eligibility requirements over time.
+
+| Requirement Category | Requirement Details | Effective Date (YYYY-MM-DD) | Change Type | Justification | Notes |
+|----------------------|---------------------|-----------------------------|-------------|---------------|-------|
+| **Indexer Activity** | Indexers must be active for **5+ days** in a given **28 day** period for indexing rewards eligibility. | TBD | Initial | Encourages indexers to familiarize themselves with infrastructure maintenance and ongoing operations. | Planned for Service Quality Oracle launch |
+| **Query Qualification** | Indexers must serve **≥1 qualifying query** on **≥10 different subgraphs** in a day for the day to count towards the **Indexer Activity** requirement. | TBD | Initial | Encourages indexers to become familiar with the process of syncing a range of subgraphs. | Planned for Service Quality Oracle launch |
+| **Query Response Quality** | *•* Query Response HTTP Status: **200 OK**<br>*•* Query Response Latency: **< 5,000 ms**<br>*•* Query Freshness: **< 50,000 blocks** behind chainhead.<br>*•* Subgraph Curation Signal: **≥ 500 GRT**. | TBD | Initial | *•* Indexer infrastructure needs to serve successful queries to benefit data consumers.<br>*•* Fast query responses are important to data consumers.<br>*•* Encourages indexers to sync to chainhead.<br>*•* Creates a barrier against gaming eligibility requirements. | Planned for Service Quality Oracle launch |
+
+---
+
+# Future Example Changes
+
+The following are examples of how future changes could be recorded in the changelog above:
+
+| Requirement Category | Requirement Details | Effective Date (YYYY-MM-DD) | Change Type | Justification | Notes |
+|----------------------|---------------------|------------------------------|------------|---------------|-------|
+| **Indexer Activity** | Indexers must be active for **10+ days** in a given **28 day** period for indexing rewards eligibility. | YYYY-MM-DD | **Updated** | Gradually increase activity requirements to encourage more consistent indexer participation in the network. | **Indexer Activity** increased to **10+ days** in a given 28 day period from **5+ days** in a given 28 day period |
+| **Query Response Quality** | *•* Query Response HTTP Status: **200 OK**<br>*•* Query Response Latency: **< 1,000 ms**<br>*•* Query Freshness: **< 50,000 blocks** behind chainhead.<br>*•* Subgraph Curation Signal: **≥ 500 GRT**. | YYYY-MM-DD | **Updated** | Ensure that indexer infrastructure is capable of sub-second query responses to help improve data consumer experience through faster responses. | **Query Response Latency** tightened to **< 1,000 ms** from **< 5,000 ms**. Other query response quality requirements unchanged. |
+| **Query Volume** | Indexers must have served **500+ queries** in the last 28 days. | YYYY-MM-DD | **New** | Encourage indexers to participate in the network to a greater degree, while still ensuring that achieving indexing rewards eligibility is feasible for all active indexers. | New requirement added. |
+
+---

README.md

@@ -49,17 +49,21 @@ Please refer to the [ELIGIBILITY_CRITERIA.md](./ELIGIBILITY_CRITERIA.md) file to
 
 The application follows a clear data flow, managed by a daily scheduler:
 
-1.  **Scheduler (`scheduler.py`)**: This is the main entry point. It runs on a schedule (e.g., daily), manages the application lifecycle, and triggers the oracle run. It is also responsible for catching up on any missed runs.
+1. **Scheduler (`scheduler.py`)**: This is the main entry point. It runs on a schedule (e.g., daily), manages the application lifecycle, and triggers the oracle run. It is also responsible for catching up on any missed runs.
 
-2.  **Orchestrator (`service_quality_oracle.py`)**: For each run, this module orchestrates the end-to-end process by coordinating the other components.
+2. **Orchestrator (`service_quality_oracle.py`)**: For each run, this module orchestrates the end-to-end process by coordinating the other components.
 
-3.  **Data Fetching (`bigquery_provider.py`)**: The orchestrator calls this provider to execute a configurable SQL query against Google BigQuery, fetching the raw indexer performance data.
+3. **Data Fetching (`bigquery_provider.py`)**: The orchestrator calls this provider to execute a configurable SQL query against Google BigQuery, fetching the raw indexer performance data.
 
-4.  **Data Processing (`eligibility_pipeline.py`)**: The raw data is passed to this module, which processes it, filters for eligible and ineligible indexers, and generates CSV artifacts for auditing and record-keeping.
+4. **Data Processing (`eligibility_pipeline.py`)**: The raw data is passed to this module, which processes it, filters for eligible and ineligible indexers, and generates CSV artifacts for auditing and record-keeping.
 
-5.  **Blockchain Submission (`blockchain_client.py`)**: The orchestrator takes the final list of eligible indexers and passes it to this client, which handles the complexities of batching, signing, and sending the transaction to the blockchain via RPC providers with built-in failover.
+5. **Blockchain Submission (`blockchain_client.py`)**: The orchestrator takes the final list of eligible indexers and passes it to this client, which handles the complexities of batching, signing, and sending the transaction to the blockchain via RPC providers with built-in failover.
 
-6.  **Notifications (`slack_notifier.py`)**: Throughout the process, status updates (success, failure, warnings) are sent to Slack.
+6. **Notifications (`slack_notifier.py`)**: Throughout the process, status updates (success, failure, warnings) are sent to Slack.
+
+## Architecture
+
+For a more detailed explanation of key architectural decisions, such as the RPC provider failover and circuit breaker logic, please see the [Technical Design Document](./docs/technical-design.md).
 
 ## CI/CD Pipeline
 
@@ -128,7 +132,6 @@ bandit -r src/
 ## TODO List (only outstanding TODOs)
 
 ### 1. Testing
-- [ ] Create integration tests for the entire pipeline
 - [ ] Security review of code and dependencies
 
 ### 2. Documentation

docker-compose.yml

@@ -51,7 +51,7 @@ services:
           memory: 512M
 
     # Restart policy
-    restart: unless-stopped
+    restart: "on-failure"
 
     # Healthcheck to ensure the container is running
     healthcheck:

docs/technical-design.md

@@ -0,0 +1,56 @@
+# Technical Design & Architecture
+
+This document outlines key architectural decisions and data flows within the Service Quality Oracle.
+
+## RPC Provider Failover and Circuit Breaker Logic
+
+The application is designed to be resilient to transient network issues and RPC provider failures. It uses a multi-layered approach involving internal retries, provider rotation, and an application-level circuit breaker to prevent catastrophic failures and infinite restart loops.
+
+The following diagram illustrates the sequence of events when all RPC providers fail, leading to a single recorded failure by the circuit breaker.
+
+```mermaid
+sequenceDiagram
+    # Setup column titles
+    participant main_oracle as service_quality_oracle.py
+    participant blockchain_client as blockchain_client.py
+    participant circuit_breaker as circuit_breaker.py
+    participant slack_notifier as slack_notifier.py
+
+    # Attempt function call
+    main_oracle->>blockchain_client: batch_allow_indexers_issuance_eligibility()
+
+    # Describe failure loop inside the blockchain_client module
+    activate blockchain_client
+    alt RPC Loop (for each provider)
+
+        # Attempt RPC call 
+        blockchain_client->>blockchain_client: _execute_rpc_call() with provider A
+        note right of blockchain_client: Fails after 5 retries
+
+        # Log failure
+        blockchain_client-->>blockchain_client: raises ConnectionError
+        note right of blockchain_client: Catches error, logs rotation
+
+        # Retry RPC call
+        blockchain_client->>blockchain_client: _execute_rpc_call() with provider B
+        note right of blockchain_client: Fails after 5 retries
+
+        # Log final failure
+        blockchain_client-->>blockchain_client: raises ConnectionError
+        note right of blockchain_client: All providers tried and failed
+    end
+
+    # Raise error back to main_oracle oracle and exit blockchain_client module
+    blockchain_client-->>main_oracle: raises Final ConnectionError
+    deactivate blockchain_client
+
+    # Take note of the failure in the circuit breaker, which can break the restart loop if triggered enough times in a short duration
+    main_oracle->>circuit_breaker: record_failure()
+
+    # Notify of the RPC failure in slack
+    main_oracle->>slack_notifier: send_failure_notification()
+
+    # Document restart process
+    note right of main_oracle: sys.exit(1)
+    note right of main_oracle: Docker will restart. CircuitBreaker can halt via sys.exit(0) 
+```

requirements.txt

@@ -2,43 +2,42 @@
 # Requires Python 3.9+
 
 # Configuration management
-tomli==2.2.1  # TOML support for Python < 3.11
+tomli==2.2.1
 
 # Scheduling and resilience
 schedule==1.2.2
 pytz==2025.2
 tenacity==8.5.0
 
 # Google Cloud BigQuery for data processing
-google-cloud-bigquery==3.26.0
-bigframes==1.42.0
+google-cloud-bigquery==3.34.0
+bigframes==2.8.0
 
 # Data processing and validation
 pandas==2.2.3
 pandera==0.20.4
-numpy>=2.0.0  # Added as pandas dependency
+numpy==2.3.1
 
 # Blockchain integration - Latest compatible versions
 web3==7.12.0
-eth-account>=0.13.0
-eth-typing>=5.2.0
+eth-account==0.13.7
+eth-typing==5.2.1
 
 # GraphQL and subgraph integration (for future subgraph functionality)
 gql==3.5.2
 
 # HTTP and API
 requests==2.32.3
-aiohttp>=3.9.0  # For async HTTP requests (used by web3)
+aiohttp==3.12.13
 
 # Development/Testing
-pytest>=8.0.0
-pytest-cov>=6.0.0
-pytest-mock>=3.0.0
-pytest-snapshot>=0.9.0
-mypy>=1.0.0
-types-pytz  # Type stubs for pytz
-types-requests  # Type stubs for requests
+pytest==8.4.1
+pytest-cov==6.2.1
+pytest-mock==3.14.1
+pytest-snapshot==0.9.0
+mypy==1.16.1
+types-pytz==2025.2.0.20250516
+types-requests==2.32.4.20250611
 
 # Linting and formatting
-ruff>=0.6.0
-pip==25.1
+ruff==0.12.0