Feature/kg builder

.github/workflows/pr-e2e-tests.yaml

@@ -13,6 +13,8 @@ concurrency:
 jobs:
   e2e-tests:
     runs-on: ubuntu-latest
+    strategy:
+      max-parallel: 1
     strategy:
       matrix:
         python-version: ['3.8', '3.12']

CHANGELOG.md

@@ -4,6 +4,9 @@
 
 ### Added
 - Add optional custom_prompt arg to the Text2CypherRetriever class.
+- Introduced support for Component/Pipeline flexible architecture.
+- Added new components for knowledge graph construction, including text splitters, schema builders, entity-relation extractors, and Neo4j writers.
+- Implemented end-to-end tests for the new knowledge graph builder pipeline.
 
 ### Changed
 - `GraphRAG.search` method first parameter has been renamed `query_text` (was `query`) for consistency with the retrievers interface.

docs/source/api.rst

@@ -3,14 +3,74 @@
 API Documentation
 #################
 
+.. _components-section:
+
+**********
+Components
+**********
+
+KGWriter
+========
+
+.. autoclass:: neo4j_genai.components.kg_writer.KGWriter
+    :members: run
+
+Neo4jWriter
+===========
+
+.. autoclass:: neo4j_genai.components.kg_writer.Neo4jWriter
+    :members: run
+
+TextSplitter
+============
+
+.. autoclass:: neo4j_genai.components.text_splitters.base.TextSplitter
+    :members: run
+
+LangChainTextSplitterAdapter
+============================
+
+.. autoclass:: neo4j_genai.components.text_splitters.langchain.LangChainTextSplitterAdapter
+    :members: run
+
+LlamaIndexTextSplitterAdapter
+=============================
+
+.. autoclass:: neo4j_genai.components.text_splitters.llamaindex.LlamaIndexTextSplitterAdapter
+    :members: run
+
+TextChunkEmbedder
+=================
+
+.. autoclass:: neo4j_genai.components.embedder.TextChunkEmbedder
+    :members: run
+
+SchemaBuilder
+=============
+
+.. autoclass:: neo4j_genai.components.schema.SchemaBuilder
+    :members: run
+
+EntityRelationExtractor
+=======================
+
+.. autoclass:: neo4j_genai.components.entity_relation_extractor.EntityRelationExtractor
+    :members: run
+
+LLMEntityRelationExtractor
+==========================
+
+.. autoclass:: neo4j_genai.components.entity_relation_extractor.LLMEntityRelationExtractor
+    :members: run
+
 .. _retrievers-section:
 
 **********
 Retrievers
 **********
 
 RetrieverInterface
-===================
+==================
 
 .. autoclass:: neo4j_genai.retrievers.base.Retriever
     :members:
@@ -70,39 +130,39 @@ PineconeNeo4jRetriever
     :members: search
 
 
-**********
+********
 Embedder
-**********
+********
 
 .. autoclass:: neo4j_genai.embedder.Embedder
     :members:
 
 SentenceTransformerEmbeddings
 ================================
 
-.. autoclass:: neo4j_genai.embeddings.SentenceTransformerEmbeddings
+.. autoclass:: neo4j_genai.embeddings.sentence_transformers.SentenceTransformerEmbeddings
     :members:
 
 **********
 Generation
 **********
 
 LLMInterface
-======================
+============
 
 .. autoclass:: neo4j_genai.llm.LLMInterface
     :members:
 
 
 OpenAILLM
-======================
+=========
 
 .. autoclass:: neo4j_genai.llm.OpenAILLM
     :members:
 
 
 PromptTemplate
-======================
+==============
 
 .. autoclass:: neo4j_genai.generation.prompts.PromptTemplate
     :members:
@@ -125,6 +185,8 @@ Database Interaction
 
 .. autofunction:: neo4j_genai.indexes.upsert_vector
 
+.. autofunction:: neo4j_genai.indexes.upsert_vector_on_relationship
+
 
 ******
 Errors
@@ -157,6 +219,12 @@ Errors
 
   * :class:`neo4j_genai.exceptions.LLMGenerationError`
 
+  * :class:`neo4j_genai.pipeline.exceptions.PipelineDefinitionError`
+
+  * :class:`neo4j_genai.pipeline.exceptions.PipelineMissingDependencyError`
+
+  * :class:`neo4j_genai.pipeline.exceptions.PipelineStatusUpdateError`
+
 
 Neo4jGenAiError
 ===============
@@ -222,7 +290,7 @@ Neo4jVersionError
 
 
 Text2CypherRetrievalError
-==========================
+=========================
 
 .. autoclass:: neo4j_genai.exceptions.Text2CypherRetrievalError
    :show-inheritance:
@@ -236,21 +304,42 @@ SchemaFetchError
 
 
 RagInitializationError
-==========================
+======================
 
 .. autoclass:: neo4j_genai.exceptions.RagInitializationError
    :show-inheritance:
 
 
 PromptMissingInputError
-==========================
+=======================
 
 .. autoclass:: neo4j_genai.exceptions.PromptMissingInputError
    :show-inheritance:
 
 
 LLMGenerationError
-==========================
+==================
 
 .. autoclass:: neo4j_genai.exceptions.LLMGenerationError
    :show-inheritance:
+
+
+PipelineDefinitionError
+=======================
+
+.. autoclass:: neo4j_genai.pipeline.exceptions.PipelineDefinitionError
+   :show-inheritance:
+
+
+PipelineMissingDependencyError
+==============================
+
+.. autoclass:: neo4j_genai.pipeline.exceptions.PipelineMissingDependencyError
+   :show-inheritance:
+
+
+PipelineStatusUpdateError
+=========================
+
+.. autoclass:: neo4j_genai.pipeline.exceptions.PipelineStatusUpdateError
+   :show-inheritance:

docs/source/index.rst

@@ -30,7 +30,8 @@ Python versions supported:
 Topics
 ******
 
-+ :ref:`user-guide`
++ :ref:`user-guide-rag`
++ :ref:`user-guide-pipeline`
 + :ref:`api-documentation`
 + :ref:`types-documentation`
 
@@ -39,7 +40,8 @@ Topics
     :caption: Contents:
     :hidden:
 
-    user_guide.rst
+    user_guide_rag.rst
+    user_guide_pipeline.rst
     api.rst
     types.rst
 

docs/source/types.rst

@@ -5,30 +5,80 @@ Types
 *****
 
 RawSearchResult
-==================
+===============
 
 .. autoclass:: neo4j_genai.types.RawSearchResult
 
 
 RetrieverResult
-==================
+===============
 
 .. autoclass:: neo4j_genai.types.RetrieverResult
 
 
 RetrieverResultItem
-====================
+===================
 
 .. autoclass:: neo4j_genai.types.RetrieverResultItem
 
 
 LLMResponse
-====================
+===========
 
 .. autoclass:: neo4j_genai.llm.types.LLMResponse
 
 
 RagResultModel
-====================
+==============
 
 .. autoclass:: neo4j_genai.generation.types.RagResultModel
+
+TextChunk
+=========
+
+.. autoclass:: neo4j_genai.components.types.TextChunk
+
+TextChunks
+==========
+
+.. autoclass:: neo4j_genai.components.types.TextChunks
+
+Neo4jNode
+=========
+
+.. autoclass:: neo4j_genai.components.types.Neo4jNode
+
+Neo4jRelationship
+=================
+
+.. autoclass:: neo4j_genai.components.types.Neo4jRelationship
+
+Neo4jGraph
+==========
+
+.. autoclass:: neo4j_genai.components.types.Neo4jGraph
+
+KGWriterModel
+=============
+
+.. autoclass:: neo4j_genai.components.kg_writer.KGWriterModel
+
+SchemaProperty
+==============
+
+.. autoclass:: neo4j_genai.components.schema.SchemaProperty
+
+SchemaEntity
+============
+
+.. autoclass:: neo4j_genai.components.schema.SchemaEntity
+
+SchemaRelation
+==============
+
+.. autoclass:: neo4j_genai.components.schema.SchemaEntity
+
+SchemaConfig
+============
+
+.. autoclass:: neo4j_genai.components.schema.SchemaConfig