Limit the number of results returned by _field_caps

[rdettai]

Is your feature request related to a problem? Please describe.
When using json fields (or dynamic mappings), the number of fields can be arbitrarily large.

Describe the solution you'd like
We would like for the field_caps endpoint to return only K results, prioritizing fields that appear in many documents. This doesn't need to be an exact Top K, we typically just want to prune a long tail of field names with random values that appear only in 1 or a few documents.

Describe alternatives you've considered
We could use pattern matching to filter out some field names, but using an approx. TopK is easier to use.

Implementation propositions

• fields that are part of the doc mapping can always be returned, their number doesn't grow with the dataset and they are usually "important"
• we would need an approximative count, for each field, of the number of docs where it appears. We need it at least for fields dynamically created in a json field.

It seems that getting that count would be quite costly for fast fields as it would require deserializing every column:

• a) we could amortize this cost by storing the count in the split_fields section of the split. This section is currently extracted from the Tantivy index in the packager. It would still be fairly costly
  • for splits with fast: true on JSON fields, it would mean deserializing all that data
  • if the split has many fields (e.g millions) it could be even worse as we would need to read all this fragmented data
• b) we could build the list of fields with their count at index time
  • Store List of Fields in Segment tantivy#2279 proposes to store the field list in the Tantivy index
  • we could also build the list of FieldMetadata (with the counts) at index time but without storing it in the index (i.e SegmentIndexWriter.add_document_track_fields(&mut self, document: D, fields_metadata: &mut HashMap<K, FieldMetadata>)

[fulmicoton-dd]

It might be interesting to add a (high) cap on the packager code as well and log a warning when it is hit, to avoid storing too much metadata. I am thinking num docs in segment / 10.

As you spotted, getting access to the number of docs containing each field is not expensive for the inverted index.
We get it from the term info (which is read anyway).
https://github.com/quickwit-oss/tantivy/blob/main/src/index/inverted_index_reader.rs#L84

it would mean deserializing all that data

Getting access to the number of docs containing each field for fast field goes as follows.
If the cardinality is optional, then getting the actual count requires deserializing the block metadata.. (or code-wise, opening the ColumnIndex). (I don't think deserializing all of the data is necessary). If this is not too expensive, we could do this. If it is too expensive, we could write adhoc code to just compute the num docs with non null values or use the column index size as a proxy, or whatever creative solution we can come up with. (?)

For multivalued column, it seems like in V2 we shield it with an optional index, which suggests we can fallback to the optional solution.

[rdettai]

As you spotted, getting access to the number of docs containing each field is not expensive for the inverted index.

Actually, I initially went a bit fast on this one. It is not that simple. The term info contains the frequency by term, but I don't see how you can derive a field level frequency from it. I assume we could record a special token that means "the field is present", but AFAIK we don't do that. If we did, we wouldn't need the FIELD_PRESENCE_FIELD in Quickwit.

We could use the FIELD_PRESENCE_FIELD, but besides the fact that it's not always enabled, it would require to go forth and back between Quickwit and Tantivy, which would prevent pushing down the TopK to Tantivy.

[fulmicoton-dd]

Ah yes you are right. Rephrasing for later reader.

TermInfo contains doc_freq, which is the number of documents containing a term. But a term is field:value.
Here what we want is the number of doc that contain the field in general. Deriving it is a rather expensive operation.

The goal is really to limit the number of fields and make sure we keep the most important ones.
Of course it is nicer if we use something that is easy to interpret to make the cut, but if it is too difficult, we can probably use something simpler like the amount of bytes dedicated to it, etc.

[rdettai]

I propose to merge #5843 as a quick and dirty workaround for now.