Execute text search only on a subset of documents to reduce execution time

[lbhm]

I am working on a project that uses full-text search as part of a larger query language, and recently, I started to consider migrating from Lucene to Tantivy for the text search component.

Since full-text search is only part of our query language, we often encounter the scenario where users write a generic keyword query (say *a*), but we already know from other (non-text) operators in the query that eventually, only a small subset of documents is relevant.

Therefore, my question is: Can I run queries only on a subset of documents based on their document ID in tantivy?

Based on my initial documentation search, I was wondering if the FilterCollector would be relevant here. However, I could not find much information on what I can pass as a TPredicate to the filter.

As a follow-up question: Would using a FilterCollector in combination with a subsequent TopDocs collector for the documents that are not filtered out have a performance benefit?

[PSeitz]

Yes intersection queries work like that generally (e.g. name:*a* AND id:1000), but it depends on the exact performance characteristics of the query if it is actually beneficial.
Related PR which tries to address this: #2538

If you have your own query language, you probably want to check what the existing query language in tantivy does.

[lbhm]

Thank you for the quick response @PSeitz.

I have two follow-up questions:

• Does tantivy have a query optimizer that recognizes that certain parts of an AND query are faster to execute than others and rewrites the query accordingly? Or do I have to write the query in a specific way? I searched through the documentation of the Query classes but coulnd't find a reference.
• Are there any plans to publish a new release soon? Ideally, I would like to use tantivy via the Python bindings so to benefit from the PR you mentioned, a new release would be great.

[PSeitz]

• There's no query optimizer currently, if you have an example for a potential rewrite we can open an issue for it
• Yes a release is planned, but it's unclear if the PR will be part of it, and if there would be a benefit for your use case

[lbhm]

I am not familiar enough with tantivy's internal approach to query execution (yet), so I'm not sure to what extent my conceptual idea is already implemented and/or doable in tantivy.

From a high level, I think of the following:
If the user issues a query like name:[a TO c}* AND id:1000 or name:[a TO c}* AND id: IN [1000, 1001, 1002], I see the following optimization potential:

• The system could detect that we have a conjunctive query and that an integer lookup is faster than doing (possibly complex) string comparisons.
• Therefore, it reorders the subqueries and executes the second subquery of my two examples first.
• For the second subquery, the system skips all documents not returned by the first subquery.

Abstractly, there are two optimization techniques here (again, I haven't checked to which extent they are implemented):

• Subqueries of conjunctive queries can skip all documents not returned by the prior subqueries of that conjunction.
• Based on that insight, the execution order of subqueries could be reordered to prioritize subqueries that are
  • fastest to execute or
  • have the highest selectivity (i.e., lowest number of results)

What do you think about that? The first optimization technique is probably much easier to implement and would leave it to the user to order their subqueries according to how efficient they expect them to be.

[PSeitz]

DocSets are advanced simultaneously, so such an optimization doesn't apply. The optimizations are more about the costs of DocSet creation and the cost of advance and skip.

[lbhm]

I see, then I'll close this issue. Thank you for all your explanations!