SQL Databases as docstore

[matthias]

Are there any plans to be able to use SQL databases (in addition to KV stores) for docstore

Would be awesome to be able to use Supabase / PostgreSQL as a unified storage for Emebddings (VectorStores) and the docstore index.

[dosubot[bot]]

🤖

Based on the information available, LlamaIndex does indeed support SQL databases for docstore. This is evident from the SQLStructStoreIndex class, which is an index that uses a SQL database under the hood. The class requires a SQLDatabase object to be passed during initialization. Here's a snippet of the __init__ method:

def __init__(
        self,
        nodes: Optional[Sequence[BaseNode]] = None,
        index_struct: Optional[SQLStructTable] = None,
        service_context: Optional[ServiceContext] = None,
        sql_database: Optional[SQLDatabase] = None,
        table_name: Optional[str] = None,
        table: Optional[Table] = None,
        ref_doc_id_column: Optional[str] = None,
        sql_context_container: Optional[SQLContextContainer] = None,
        **kwargs: Any,
    ) -> None:
        """Initialize params."""
        if sql_database is None:
            raise ValueError("sql_database must be specified")
        self.sql_database = sql_database

The SQLDatabase class is a wrapper around the SQLAlchemy engine to interact with a SQL database. It provides methods to execute SQL commands, insert data into tables, and retrieve information about the database schema. You can find more details in the source code at llama_index/indices/struct_store/sql.py and llama_index/utilities/sql_wrapper.py.

As for the support of PostgreSQL as a unified storage for Embeddings (VectorStores) and the docstore index, there is a solved issue titled Add postgres vector database support which indicates that Postgres vector database support has been added to LlamaIndex. You can find the documentation and usage examples at this link: Postgres Vector Store.

I hope this information is helpful. If you have further questions or need more clarification, feel free to ask.

Sources

• Add postgres vector database support
• llama_index/indices/struct_store/sql.py
• llama_index/utilities/sql_wrapper.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[logan-markewich]

@matthias I would gladly welcome a PR for this! The KVStore abstractions are really pretty simple, so this integration would be an easy PR :)