Running into an error while creating embeddings for the knowledge base (PgVector & GeminiEmbedder)

[Anan7Codes]

Hey Team,

I am running into this error

sqlalchemy.exc.IntegrityError: (psycopg.errors.NotNullViolation) null value in column "id" of relation "csv_documents" violates not-null constraint
DETAIL:  Failing row contains (null, null, {}, {}, null, null, null, 2025-06-03 16:28:07.593434+00, null, null).
[SQL: INSERT INTO ai.csv_documents DEFAULT VALUES]
(Background on this error at: https://sqlalche.me/e/20/gkpj)

which is strange because the first 5 rows are correctly embedded and inserted and then suddenly this happens. Also I believe the id should be auto-generated so not sure how I run into this.

Code below:

# Path to your CSV file
csv_path = Path("files/sales_report.csv")

# Set up PgVector database (update db_url as needed)
vector_db = PgVector(
    table_name="csv_documents",
    db_url="postgresql+psycopg://ai:ai@localhost:5532/ai",  # Update with your DB credentials
    embedder=GeminiEmbedder(),
)

# Create the knowledge base
knowledge_base = CSVKnowledgeBase(
    path=csv_path,
    vector_db=vector_db,
)

[Anan7Codes]

Okay looks like after 5 times, I get this error from Gemini. Thoughts on how I can maybe delay?

ERROR    Error processing document 'sales_report': 429 RESOURCE_EXHAUSTED. {'error': {'code': 429, 'message': 'Resource has been exhausted (e.g. check quota).',      
         'status': 'RESOURCE_EXHAUSTED'}}    

[monali7-d]

Hey @Anan7Codes
thanks for reaching out and supporting Agno. I’ve shared this with the team, we’re working through all requests one by one and will get back to you soon.
If it’s urgent, please let us know. We appreciate your patience!

[Anan7Codes]

It's not very urgent. Thank you.

[ysolanky]

Hi @Anan7Codes ! This is not a rate-limit error. Rather an exhaustion of credits error. Please either use an alternative embedder or you would need to update your Google Gemini account

[Anan7Codes]

Weird cause when I ran this custom code, it inserted around 64 rows. The behaviour seemed more like rate limit tbf.

import time
from agno.embedder.google import GeminiEmbedder

class DelayedGeminiEmbedder(GeminiEmbedder):
    # def embed(self, *args, **kwargs):
    #     result = super().embed(*args, **kwargs)
    #     time.sleep(30)  # Wait for 30 seconds after each embedding
    #     return result

    # async def async_embed(self, *args, **kwargs):
    #     result = await super().async_embed(*args, **kwargs)
    #     import asyncio
    #     await asyncio.sleep(30)  # Wait for 30 seconds after each embedding (async)
    #     return result

    def get_embedding(self, text: str):
        result = super().get_embedding(text)
        time.sleep(30)
        return result

    def get_embedding_and_usage(self, text: str):
        result = super().get_embedding_and_usage(text)
        time.sleep(30)
        return result