How to use local DeepSeek R1

[TomSeestern]

TLDR: Works good, can recommend

I just used DeepSeek-R1-Distill-Llama-8B-Q4_K_M.gguf for analyzing 100 documents locally, and I'm fairly impressed by both speed and accuracy:

• Took about 10s per document on a RTX 2060
• Happy with the results
  • Was able to process 95 of 100 document presented
  • Decent titles, sometimes only "Invoice", might depend on my prompt
  • Good use of existing Correspondence
  • Mediocre use of existing Tags, created lots of new ones and mixed languages
  • Excellent parsing of dates, even being able to form something Greek "25 Σεπ. 02 -> 25.09.2002

How to

If you want to try it yourself, here are the settings I used:

Settings: Paperless Assistant

AI Configuration

AI Provider
-> Custom
Base URL (local IP of my pc, not localhost!, note the /v1 at the end if the URL!)
-> http://192.168.7.99:1234/v1
API Key (just a random '0' as LM Studio doesn't need a key)
-> 0
Model
-> deepseek-r1-distill-llama-8b

Prompt

You are a personalized document analyzer. Your task is to analyze documents and extract relevant information.

Analyze the document content and extract the following information into a structured JSON object:

1. title: Create a concise, meaningful title for the document
2. correspondent: Identify the sender/institution but do not include addresses
3. tags: Select up to 4 relevant thematic tags
4. document_date: Extract the document date (format: YYYY-MM-DD)
5. language: Determine the document language (e.g. "de" or "en")
      
Important rules for the analysis:

For tags:
- FIRST check the existing tags before suggesting new ones
- Use only relevant categories
- Maximum 4 tags per document, less if sufficient (at least 1)
- Avoid generic or too specific tags
- Use only the most important information for tag creation
- The output language is the one used in the document! IMPORTANT!

For the title:
- Short and concise, NO ADDRESSES
- Contains the most important identification features
- For invoices/orders, mention invoice/order number if available
- The output language is the one used in the document! IMPORTANT!

For the correspondent:
- Identify the sender or institution
- When generating the correspondent, always create the shortest possible form of the company name (e.g. "Amazon" instead of "Amazon EU SARL, German branch")

For the document date:
- Extract the date of the document
- Use the format YYYY-MM-DD
- If multiple dates are present, use the most relevant one

For the language:
- Determine the document language
- Use language codes like "de" for German or "en" for English
- If the language is not clear, use "und" as a placeholder

Settings LM Studio

I used LM Studio 0.3.9 running on a RTX 2060

• Switch mode on the bottom left to "Developer" and go to the tab

My Server settings:

Inference settings

• Make sure to enable structured output and use my template below
• Make sure to save the preset.

Important: You have to use "structured output" in order to get the right output that paperless-ai can understand!
Here is the Schema I used, paste it into the "Structured Output" setting in LM Studio as shown in the screenshot above.

{
  "type": "object",
  "properties": {
    "title": {
      "type": "string",
      "description": "The title of the document."
    },
    "correspondent": {
      "type": "string",
      "description": "The name of the correspondent related to the document."
    },
    "tags": {
      "type": "array",
      "items": {
        "type": "string"
      },
      "description": "A list of tags associated with the document."
    },
    "document_type": {
      "type": "string",
      "description": "The type of document, e.g., 'invoice', 'report', 'email'."
    },
    "document_date": {
      "type": "string",
      "format": "date",
      "description": "The date of the document in YYYY-MM-DD format."
    },
    "language": {
      "type": "string",
      "description": "The language of the document in ISO 639-1 format."
    }
  },
  "required": [
    "title",
    "document_type",
    "correspondent",
    "tags",
    "document_date",
    "language"
  ]
}

Load Settings

• Had some big files, so I cranked up the context length to 13k

[mamema]

hmm, yeah, deepseek the new kid on the block....
..but you wrote for example: Mediocre use of existing Tags, created lots of new ones and mixed languages
So.... can you compare ito to gemma2:latest, as i haven't those issues with it you are describing?

deepseek is great.... for jailbreak prompts as it lacks every safety measure around. :-)

[TomSeestern]

A bit of Prompt tuning helped with the Language stuff, but I think the Tag problem is related to #232 . I tried putting all my Tags as part of the prompt and that somewhat helped. So that should be a problem/fix with any model in paperless-ai.
I will try gemma2 next!

[mamema]

putting all my tags as part of the prompt, would break every prompting, as i have +/- 450 tags. This whole "put all tags in prompt" i still don't understand to use and/or cannot use because of prompt size.

[cmorlok]

Thanks for figuring out how to use DeepSeek. I have created #381 to add support for DeepSeek (and others) on Ollama by adding support for structured output.

However, I am not happy with the results. I have tried the same document a couple of times. 2 out of 10 times the results are garbage, values are english (instead of german) and even the date is in a wrong format (document says 31.01.2025, DeepSeek returns 3101-02-25). In 2 out of 10 times it is OK, but not good, and in the remaining 6 times, it is perfect. No idea why, it is always the exact same prompt. Similar things happen with other models as well, but never as much as with DeepSeek. Might be a problem that all of my documents are german.

[mamema]

no i guess thats AI like it is today. Even ChatGPT is like that sometimes

[cmorlok]

I am thinking about adding validation step, asking the AI whether the answer fulfills the required structure and content and if not, try again

[haldi4803]

• Mediocre use of existing Tags, created lots of new ones and mixed languages

I can Confirm this. Holy crap Google Gemini was already freespirited in that regard, but this is WAY over the top.