Adding Workspace Support to LinghtRAG Server

[danielaskdd]

Background

• Support multiple document sets(Workspace) at one server
• Create and destroy document set easy and quick
• Temporary or private document set
• Single file RAG
• Ollama models emulation for multiple document sets
• Act as a Tool(MCP Server) to AI agent with multiple document sets

Challenge

• API Compatible
• Minimize UI changes at transition
• Method of storage isolation for each storage implentation
• Cache isolation
• Parallel document processing for difference document sets
• LLM sessions reserved for query to avoid frontend blockage

[ParisNeo]

Hi there.
Good idea.
This would mean a substantial upgrades.
And it won't be stable right away, so I suggest creating a new branch on which we can start this work.

We will need alot of changes.
We need to add a new storage to store user credentials and make a robust and secure architecture.

[niallam22]

Alternative Approach: API Key-Bound Collections

Background

Utilize the existing workspace field already present in database tables (I have only checked postgres)
Map collections to API keys rather than user credentials
Maintain backend service model with simpler authentication requirements

Benefits

Minimal schema changes (only needs a collections-to-API-key mapping table)
No need for user credential management within LightRAG
Preserves LightRAG as a backend service that can be integrated with any authentication system

Implementation Strategy

Create a lightweight collection management layer that maps collection IDs to workspaces
Extend API endpoints to accept collection ID as a parameter
Add validation to ensure collections are only accessible by their associated API key

Challenges Addressed

API Compatible: Extends existing API with minimal breaking changes
Minimize UI Changes: UI only needs to pass collection ID parameter
Storage Isolation: Uses existing workspace field in all tables

This approach would require significantly less architectural change than implementing a full user credential system, while still providing robust multi-document set capabilities. It maintains LightRAG as a flexible backend service that can be integrated with various frontend authentication systems rather than building that directly into the core.

[danielaskdd]

Support rapid creation of temporary Workspaces of previously indexed files, by utilizing LLM caching, and maintain document chunks. This means the creation process will be quick.

[waiwaiwai27]

may be adding users and groups.
different userr could be in different groups. using groups to access different workspace.

[georgewangchn]

All storage systems, including vector databases and graph databases, should include a kb_id attribute. This field must be passed in all storage and query operations to ensure data isolation across different knowledge bases.

[xuhewei98]

Consider adding an OCR module similar to the one in RAGFlow to parse non-text data such as images and tables

[danielaskdd]

LightRAG already support using Docling as file extractor.

[BireleyX]

I hope this feature gets added soon! I'm currently running 2 instances of Lightrag to separate common data from user-specific data.
And my user-specific data is limited to one-user at a time for now.

Would it be feasible to implement an interim solution by adding a file_path filter in QueryParam? I think this will be much easier than using the ID list as filter. I can easily separate the source file folders for my common files and per-user files. When a query is made, I can place the root path as filter in the lightrag.query method to only include context from documents with file_paths containing the filter string.
I think this can meet the intended use of data separation for the meantime.

[danielaskdd]

LightRAG heavily relies on entity and relation summaries to answer queries, which cannot be filtered by file_path.

[bastianwegge]

Huge thanks to the team around LightRAG so far, these features and especially the single-doc-RAG sound very promising.

We have implemented a multi-topic (multiple instances of LightRAG) similar to the one described here using the existing LightRAG functionality. I have pasted a snippet in issue 1513.

IMHO LightRAGs concern should NOT be user-auth and IMHO the LightRAG web-ui is already a great start to kick new projects off of it. Addressing Cache isolation would bring huge benefits, as this is the main reason I'm back here searching discussions / issues.