[Bug]: LLM creates JSON with individual keys/field_names and cant be used

[WhiteSockedDancer]

🔍 Bug Summary

Using the standard prompt some documents entice the llm to build a json with other keys then title, correspondend and so on.

📖 Description

Even adding more information to the prompt to add only the provided keys doesnt change it for these documents.

Model: gwen2.5:14b

My last tried prompt:

You are a personalized document analyzer. Your task is to analyze documents and extract relevant information.

Analyze the document content and extract only the following information into a structured JSON object:

1. title: Create a concise, meaningful title for the document
2. correspondent: Identify the sender/institution but do not include addresses
3. tags: Select up to 4 relevant thematic tags from existing tags
4. document_date: Extract the document date (format: YYYY-MM-DD)
5. document_type: Determine a precise type that classifies the document (e.g. Invoice, Contract, Employer, Information and so on)
6. language: Determine the document language (e.g. "de" or "en")

Important rules for the analysis:

Only create information for the mentioned keys in the JSON object: title, correspondent, tags, document_date, document_type, language
Do not translate the keys.

For tags:
- FIRST check the existing tags before suggesting new ones
- Use only relevant categories
- Maximum 4 tags per document, less if sufficient (at least 1)
- Use only the most important information for tag selection

For the title:
- Short and concise, NO ADDRESSES
- Contains the most important identification features
- For invoices/orders, mention invoice/order number if available
- The output language is always German! IMPORTANT!

For the correspondent:
- FIRST check the existing senders or institutions before suggesting new ones
- Identify the sender or institution
- When generating the correspondent, always create the shortest possible form of the company name (e.g. "Amazon" instead of "Amazon EU SARL, German branch")

For the document type:
- FIRST check the existing document types before suggesting new ones

For the document date:
- Extract the date of the document
- Use the format YYYY-MM-DD
- If multiple dates are present, use the most relevant one

For the language:
- Determine the document language
- Use language codes like "de" for German or "en" for English
- If the language is not clear, use "und" as a placeholder

Probably structured output for ollama could fix this:
#381

🔄 Steps to Reproduce

1. use some complex german document that can be parsed very well? Didnt understand which documents do this..

✅ Expected Behavior

1. always output the same structured json

❌ Actual Behavior

2. LLM puts out a json that cant be used by paperless-ai

📜 Docker Logs

Prompt Token Count: 3236

Expected Response Tokens: 1024

Dynamic calculated num_ctx: 4260

Extracted JSON String: {

  "rechnung": {

    "nummer": "<redacted>",

    "rechnungsdatum": "23.01.2025",

    "zahlungstermin": "28.02.2025",

    "betrag": "22,20 €",

    "behandlungsart": "Ambulante Behandlung",

    "verschreibung": {

      "arzt": "<redacted>",

      "adresse": "<redacted>"

    },

    "patient": {

      "name": "<redacted>",

      "geburtsdatum": "<redacted>",

      "nummer": "<redacted>"

    }

Error parsing JSON from response: Expected ',' or '}' after property value in JSON at position 438 (line 16 column 6)

Attempting to sanitize the JSON...

Final JSON parsing failed after sanitization.

This happens when the JSON structure is too complex or invalid.

That indicates an issue with the generated JSON string by Ollama.

Switch to OpenAI for better results or fine tune your prompt.

No tags or correspondent found in response from Ollama for Document.

Please review your prompt or switch to OpenAI for better results.

📜 Paperless-ngx Logs

🖼️ Screenshots of your settings page

No response

🖥️ Desktop Environment

Windows

💻 OS Version

No response

🌐 Browser

None

🔢 Browser Version

No response

🌐 Mobile Browser

No response

📝 Additional Information

• I have checked existing issues and this is not a duplicate
• I am sure that this problem is affecting everyone, not only me
• I have tried debugging this issue on my own
• I can provide a fix and submit a PR

📌 Extra Notes

After looking into this pull request (#381) i wondered why the prompt is also written into https://github.com/clusterzx/paperless-ai/blob/main/config/config.js

In which way is the Prompt Description (maybe also misleading named) from the settings page used?

[clusterzx]

This is not an issue by paperless and the solution or reason is in the logs you posted.

That's completely up to the LLM you use.

Closed as not a bug.

[cmorlok]

It might be not a bug, but the majority of LLMs are not usable in Ollama when not using the structured output feature.

[WhiteSockedDancer]

First of, thank you for the quite well support and active commenting here! That is not always a given!

Maybe you can convert this into a discussion. I feel like this could also be a issue for other ollama users and maybe others can profit with similar experiences (even if this is not code-related).

Will the mentioned PR get merged or do i have to get another solution (If this even clears my issue)?

Would you mind explaining my question:
After looking into this pull request (#381) i wondered why the prompt is also written into https://github.com/clusterzx/paperless-ai/blob/main/config/config.js

In which way is the Prompt Description from the settings page (vs. the hard coded prompt in config.js) used?

[cmorlok]

In which way is the Prompt Description from the settings page (vs. the hard coded prompt in config.js) used?

The hard coded prompt in config.js is always added to whatever the user enters in the settings and can't be overwritten (except for the new custom prompt feature in the web hook)