LLMGraphTransformer genertate empty relationships list.

[nicholasvan]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

import os

from langchain_community.graphs import Neo4jGraph

os.environ["NEO4J_URI"] = "bolt://localhost:7687"
os.environ["NEO4J_USERNAME"] = "neo4j"
os.environ["NEO4J_PASSWORD"] = "password"

graph = Neo4jGraph()

from langchain_experimental.graph_transformers import LLMGraphTransformer
from langchain_ollama import ChatOllama

llm = ChatOllama(model="llama3.1", temperature=0)
llm_transformer = LLMGraphTransformer(llm=llm)

from langchain_core.documents import Document

text = """
Marie Curie, born in 1867, was a Polish and naturalised-French physicist and chemist who conducted pioneering research on radioactivity.
She was the first woman to win a Nobel Prize, the first person to win a Nobel Prize twice, and the only person to win a Nobel Prize in two scientific fields.
Her husband, Pierre Curie, was a co-winner of her first Nobel Prize, making them the first-ever married couple to win the Nobel Prize and launching the Curie family legacy of five Nobel Prizes.
She was, in 1906, the first woman to become a professor at the University of Paris.
"""
documents = [Document(page_content=text)]
graph_documents = llm_transformer.convert_to_graph_documents(documents)
print(f"Nodes:{graph_documents[0].nodes}")
print(f"Relationships:{graph_documents[0].relationships}")

Description

As I'm following the 'How to do' chapter: https://python.langchain.com/docs/how_to/graph_constructing

BTW, I'm using ollama with llama3.1:8b to replace ChatGPT.

The expected output should be:

Nodes:[Node(id='Marie Curie', type='Person'), Node(id='Pierre Curie', type='Person'), Node(id='University Of Paris', type='Organization')]
Relationships:[Relationship(source=Node(id='Marie Curie', type='Person'), target=Node(id='Pierre Curie', type='Person'), type='MARRIED'), Relationship(source=Node(id='Marie Curie', type='Person'), target=Node(id='University Of Paris', type='Organization'), type='PROFESSOR')]

But this is what I got:

Nodes:[Node(id='Marie Curie', type='Person', properties={}), Node(id='Pierre Curie', type='Person', properties={}), Node(id='University Of Paris', type='Organization', properties={})]
Relationships:[]

The relationships is empty...So I decided to debug. And got the AIMessage:

AIMessage(content='', additional_kwargs={}, response_metadata={'model': 'llama3.1', 'created_at': '2024-10-04T06:34:55.398352Z', 'message': {'role': 'assistant', 'content': '', 'tool_calls': [{'function': {'name': 'DynamicGraph', 'arguments': {'nodes': '[{"id": "Marie Curie", "type": "person"}, {"id": "Pierre Curie", "type": "person"}, {"id": "University of Paris", "type": "organization"}]', 'relationships': '[{"source": "Marie Curie", "target": "Pierre Curie", "type": "spouse"}, {"source": "Marie Curie", "target": "University of Paris", "type": "professor"}, {"source": "Marie Curie", "target": "Nobel Prize", "type": "winner"}]'}}}]}, 'done_reason': 'stop', 'done': True, 'total_duration': 2966214800, 'load_duration': 16829100, 'prompt_eval_count': 727, 'prompt_eval_duration': 278482000, 'eval_count': 148, 'eval_duration': 2668183000}, id='run-aff1f6ea-79a3-4510-8ced-5ee284398d8f-0', tool_calls=[{'name': 'DynamicGraph', 'args': {'nodes': '[{"id": "Marie Curie", "type": "person"}, {"id": "Pierre Curie", "type": "person"}, {"id": "University of Paris", "type": "organization"}]', 'relationships': '[{"source": "Marie Curie", "target": "Pierre Curie", "type": "spouse"}, {"source": "Marie Curie", "target": "University of Paris", "type": "professor"}, {"source": "Marie Curie", "target": "Nobel Prize", "type": "winner"}]'}, 'id': '07324c7f-82de-404e-a5e3-c1239b576d18', 'type': 'tool_call'}], usage_metadata={'input_tokens': 727, 'output_tokens': 148, 'total_tokens': 875})

This means the graph information is correctly extracted, the relationship is not empty at first.

def _convert_to_graph_document(
    raw_schema: Dict[Any, Any],
) -> Tuple[List[Node], List[Relationship]]:
    # If there are validation errors
    if not raw_schema["parsed"]:
        try:
            try:  # OpenAI type response
                argument_json = json.loads(
                    raw_schema["raw"].additional_kwargs["tool_calls"][0]["function"][
                        "arguments"
                    ]
                )
            except Exception:  # Google type response
                try:
                    argument_json = json.loads(
                        raw_schema["raw"].additional_kwargs["function_call"][
                            "arguments"
                        ]
                    )
                except Exception:  # Ollama type response
                    argument_json = raw_schema["raw"].tool_calls[0]["args"]
                    if isinstance(argument_json["nodes"], str):
                        argument_json["nodes"] = json.loads(argument_json["nodes"])
                    if isinstance(argument_json["relationships"], str):
                        argument_json["relationships"] = json.loads(
                            argument_json["relationships"]
                        )

            nodes, relationships = _parse_and_clean_json(argument_json)

the argument_json here is

{
   "nodes":[
      {
         "id":"Marie Curie",
         "type":"person"
      },
      {
         "id":"Pierre Curie",
         "type":"person"
      },
      {
         "id":"University of Paris",
         "type":"organization"
      }
   ],
   "relationships":[
      {
         "source":"Marie Curie",
         "target":"Pierre Curie",
         "type":"spouse"
      },
      {
         "source":"Marie Curie",
         "target":"University of Paris",
         "type":"professor"
      },
      {
         "source":"Marie Curie",
         "target":"Nobel Prize",
         "type":"winner"
      }
   ]
}

And the function _parse_and_clean_json require key is not match with the input. So the relationships is empty List.

   ...
    relationships = []
    for rel in argument_json["relationships"]:
        # Mandatory props
        if (
            not rel.get("source_node_id")
            or not rel.get("target_node_id")
            or not rel.get("type")
        ):
            continue
   ...

And my langsmith shows this error

2 validation errors for DynamicGraph
nodes
  Input should be a valid list [type=list_type, input_value='[{"id": "Marie Curie", "...type": "organization"}]', input_type=str]
    For further information visit https://errors.pydantic.dev/2.9/v/list_type
relationships
  Input should be a valid list [type=list_type, input_value='[{"source": "Marie Curie...ze", "type": "winner"}]', input_type=str]
    For further information visit https://errors.pydantic.dev/2.9/v/list_typeTraceback (most recent call last):


  File "F:\conda\envs\torch\Lib\site-packages\langchain_core\runnables\base.py", line 1923, in _call_with_config
    context.run(


  File "F:\conda\envs\torch\Lib\site-packages\langchain_core\runnables\config.py", line 396, in call_func_with_variable_args
    return func(input, **kwargs)  # type: ignore[call-arg]
           ^^^^^^^^^^^^^^^^^^^^^


  File "F:\conda\envs\torch\Lib\site-packages\langchain_core\runnables\passthrough.py", line 481, in _invoke
    **self.mapper.invoke(
      ^^^^^^^^^^^^^^^^^^^


  File "F:\conda\envs\torch\Lib\site-packages\langchain_core\runnables\base.py", line 3723, in invoke
    output = {key: future.result() for key, future in zip(steps, futures)}
                   ^^^^^^^^^^^^^^^


  File "F:\conda\envs\torch\Lib\concurrent\futures\_base.py", line 456, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^


  File "F:\conda\envs\torch\Lib\concurrent\futures\_base.py", line 401, in __get_result
    raise self._exception


  File "F:\conda\envs\torch\Lib\concurrent\futures\thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^


  File "F:\conda\envs\torch\Lib\site-packages\langchain_core\runnables\base.py", line 3707, in _invoke_step
    return context.run(
           ^^^^^^^^^^^^


  File "F:\conda\envs\torch\Lib\site-packages\langchain_core\runnables\base.py", line 3022, in invoke
    input = context.run(step.invoke, input, config)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^


  File "F:\conda\envs\torch\Lib\site-packages\langchain_core\output_parsers\base.py", line 185, in invoke
    return self._call_with_config(
           ^^^^^^^^^^^^^^^^^^^^^^^


  File "F:\conda\envs\torch\Lib\site-packages\langchain_core\runnables\base.py", line 1923, in _call_with_config
    context.run(


  File "F:\conda\envs\torch\Lib\site-packages\langchain_core\runnables\config.py", line 396, in call_func_with_variable_args
    return func(input, **kwargs)  # type: ignore[call-arg]
           ^^^^^^^^^^^^^^^^^^^^^


  File "F:\conda\envs\torch\Lib\site-packages\langchain_core\output_parsers\base.py", line 186, in <lambda>
    lambda inner_input: self.parse_result(
                        ^^^^^^^^^^^^^^^^^^


  File "F:\conda\envs\torch\Lib\site-packages\langchain_core\output_parsers\openai_tools.py", line 297, in parse_result
    raise e


  File "F:\conda\envs\torch\Lib\site-packages\langchain_core\output_parsers\openai_tools.py", line 292, in parse_result
    pydantic_objects.append(name_dict[res["type"]](**res["args"]))
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^


  File "F:\conda\envs\torch\Lib\site-packages\pydantic\main.py", line 212, in __init__
    validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^


pydantic_core._pydantic_core.ValidationError: 2 validation errors for DynamicGraph
nodes
  Input should be a valid list [type=list_type, input_value='[{"id": "Marie Curie", "...type": "organization"}]', input_type=str]
    For further information visit https://errors.pydantic.dev/2.9/v/list_type
relationships
  Input should be a valid list [type=list_type, input_value='[{"source": "Marie Curie...ze", "type": "winner"}]', input_type=str]
    For further information visit https://errors.pydantic.dev/2.9/v/list_type

Could any one help me?

System Info

System Information

OS: Windows
OS Version: 10.0.26120
Python Version: 3.12.5 | packaged by Anaconda, Inc. | (main, Sep 12 2024, 18:18:29) [MSC v.1929 64 bit (AMD64)]

Package Information

langchain_core: 0.3.8
langchain: 0.3.2
langchain_community: 0.3.1
langsmith: 0.1.129
langchain_chroma: 0.1.4
langchain_experimental: 0.3.2
langchain_ollama: 0.2.0
langchain_text_splitters: 0.3.0
langgraph: 0.2.21

Optional packages not installed

langserve

Other Dependencies

aiohttp: 3.10.5
async-timeout: Installed. No version info available.
chromadb: 0.5.7
dataclasses-json: 0.6.7
fastapi: 0.115.0
httpx: 0.27.2
jsonpatch: 1.33
langgraph-checkpoint: 1.0.9
numpy: 1.26.4
ollama: 0.3.3
orjson: 3.10.7
packaging: 24.1
pydantic: 2.9.2
pydantic-settings: 2.5.2
PyYAML: 6.0.2
requests: 2.32.3
SQLAlchemy: 2.0.34
tenacity: 8.5.0
typing-extensions: 4.12.2

[dosubot[bot]]

It seems like you're encountering an issue with the LLMGraphTransformer generating an empty relationships list. This might be due to a mismatch in the expected keys for relationships in the _parse_and_clean_json function. The function expects source_node_id, target_node_id, and type keys, but your input JSON uses source, target, and type.

To resolve this, you can modify the _parse_and_clean_json function to match the keys in your input JSON. Alternatively, you can preprocess your JSON to rename the keys to source_node_id and target_node_id before passing it to the function.

Additionally, there is a similar issue discussed in the LangChain repository where the LLMGraphTransformer returned nothing when using langchain_openai.ChatOpenAi with a base URL. The solution involved modifying the initialization and response processing logic to handle the structured output correctly ^[1].

You might want to check if similar adjustments are needed for your setup with ChatOllama.

To continue talking to Dosu, mention @dosu.

[nicholasvan]

Thanks，Are there any alternatives that don't require modifying the LangChain source code?