Is it possible to specify document metadata using YAML pipeline?

[lambdaofgod]

I'm using this pipeline for document search.

The outputs of API /query endpoint are confusing, in returned results I don't see any field that lets me correlate passages with the whole document:

{
  "_index": "document",
  "_type": "_doc",
  "_id": "1b93d0dff9095a44cf715c39ef9dd21a",
  "_version": 1,
  "_score": 1,
  "_source": {
    "content": "... ",
    "content_type": "text",
    "id_hash_keys": [
      "content"
    ],
    "embedding": [],
    "_split_id": 1,
    "_split_overlap": [...]
  },
  "fields": {
    "_split_id": [
      1
    ],
    "_split_overlap.range": [
      0,
      339,
      616,
      804
    ],
    "content_type": [
      "text"
    ],
    "id_hash_keys": [
      "content"
    ],
    "embedding": [... ],
    "_split_overlap.doc_id": [
      "1c10a1ee2e935b693980b949a577a443",
      "22b8e7c938a966433b7978048ba77d74"
    ],
    "content": [...]
}

Is it possible to add separate indexing step that will index the whole documents, not only splits?

I can't find anything about using multiple indexers in documentation or pipeline specifications.

Is it possible to add something that will let me find the whole document based on passages?

[bilgeyucel]

Hey @lambdaofgod , how do you index your files to the DocumentStore? If you're using the /file-upload endpoint, there is a field named meta to add metadata to each file as you upload them.

[ZanSara]

Actually, returned documents should already have a meta field which contains the original filename into the name field. It's added by the Preprocessor. @lambdaofgod can you maybe check directly into the document store to see if such information is stored? This way we can at least tell if it's an indexing or a query problem.

[lambdaofgod]

Ok, I've tried this meta field and it works.
It might be a good idea to document this or show some example in swagger.