Docs: vector search concept (#17102)

Co-authored-by: anton-bobkov <anton-bobkov@ydb.tech>

Author: azevaykin

ydb/docs/en/core/concepts/_includes/secondary_indexes.md

@@ -33,9 +33,9 @@ Currently, a unique index cannot be added to an existing table.
 
 ## Vector Index
 
-[Vector Index](vector_indexes.md) is a special type of secondary index.
+[Vector Index](../../dev/vector-indexes.md) is a special type of secondary index.
 
-Unlike secondary indexes, which optimize equality or range searches, vector indexes allow similarity searches based on distance or similarity functions.
+Unlike secondary indexes, which optimize equality or range searches, vector indexes allow [vector search](../vector_search.md) based on distance or similarity functions.
 
 ### Creating a Secondary Index Online {#index-add}
 

ydb/docs/en/core/concepts/_includes/vector_indexes.md

@@ -0,0 +1,14 @@
+Approximate methods do not perform a complete enumeration of the initial data. This allows significantly speeding up the search process, although it might lead to some reduction in the quality of the results.
+
+[Scalar Quantization](../../yql/reference/udf/list/knn.md#approximate-vector-search-scalar-quantization) is a method of reducing vector dimensionality, where a set of coordinates is mapped into a space of smaller dimensions.
+
+{{ ydb-short-name }} supports vector searching for vector types `Float`, `Int8`, `Uint8`, and `Bit`. Consequently, it is possible to apply scalar quantization to transform data from `Float` to any of these types.
+
+Scalar quantization reduces the time required for reading and writing data by decreasing the number of bytes. For example, when quantizing from `Float` to `Bit`, each vector is reduced by 32 times.
+
+[Approximate vector search without an index](../../yql/reference/udf/list/knn.md#approximate-vector-search-examples) uses a very simple additional data structure - a set of vectors with other quantization. This allows the use of a simple search algorithm: first, a rough preliminary search is performed on the compressed vectors, followed by refining the results on the original vectors.
+
+Main advantages:
+
+* Full support for [transactions](../../concepts/glossary.md#transactions), including in strict consistency mode.
+* Instant application of data modification operations: insertion, update, deletion.

ydb/docs/en/core/concepts/_includes/vector_search_approximate.md

@@ -0,0 +1,9 @@
+The foundation of the exact method is the calculation of the distance from the query vector to all the vectors in the dataset. This algorithm, also known as the naive approach or brute force method, has a runtime of `O(dn)`, where `n` is the number of vectors in the dataset, and `d` is their dimensionality.
+
+[Exact vector search](../../yql/reference/udf/list/knn.md#exact-vector-search-examples) is best utilized if the complete enumeration of the vectors occurs within acceptable time limits. This includes cases where they can be pre-filtered based on some condition, such as a user identifier. In such instances, the exact method may perform faster than the current implementation of [vector indexes](../../dev/vector-indexes.md).
+
+Main advantages:
+
+* No need for additional data structures, such as specialized [vector indexes](../../concepts/glossary.md#vector-index).
+* Full support for [transactions](../../concepts/glossary.md#transactions), including in strict consistency mode.
+* Instant execution of data modification operations: insertion, update, deletion.

ydb/docs/en/core/concepts/_includes/vector_search_exact.md

@@ -100,7 +100,7 @@ Technically, tablets are [actors](#actor) with a persistent state reliably saved
 
 Together, these mechanisms allow {{ ydb-short-name }} to provide [strict consistency](https://en.wikipedia.org/wiki/Consistency_model#Strict_consistency).
 
-The implementation of distributed transactions is covered in a separate article [{#T}](../contributor/datashard-distributed-txs.md), while below there's a list of several [related terms](#distributed-transaction-implementation).
+The implementation of distributed transactions is covered in a separate article [{#T}](../contributor/datashard-distributed-txs.md), while below there's a list of several [related terms](#deterministic-transactions).
 
 ### Interactive transactions {#interactive-transaction}
 
@@ -162,9 +162,10 @@ A special type of **secondary index** is singled out separately - [vector index]
 
 #### Vector Index {#vector-index}
 
-A **vector index** is an additional data structure used to speed up the [nearest neighbor search](https://en.wikipedia.org/wiki/Nearest_neighbor_search), typically when the data is too large for the [index-less approach](../yql/reference/udf/list/knn.md) to handle the load. Unlike the primary index, vector indexes are managed independently of the underlying table data. Thus, a table can have multiple vector indexes for different scenarios. For more information about using vector indexes in {{ ydb-short-name }}, see [{#T}](vector_indexes.md).
+**Vector index** is an additional data structure used to speed up the [vector search](vector_search.md) when there is a large amount of data, and the [exact vector search without an index](../yql/reference/udf/list/knn.md) does not perform satisfactorily. The capabilities of {{ ydb-short-name }} regarding vector indexes are described in a separate article [{#T}](../dev/vector-indexes.md).
+
+**Vector index** is distinct from a [secondary index](#secondary-index) as it solves other tasks.
 
-A **vector index** is allocated separately from the [secondary index] (#secondary-index), as it solves other tasks.
 
 #### Column family {#column-family}
 

ydb/docs/en/core/concepts/glossary.md

@@ -14,8 +14,8 @@ items:
   href: transactions.md
 - name: Secondary indexes
   href: secondary_indexes.md
-- name: Vector indexes
-  href: vector_indexes.md
+- name: Vector search
+  href: vector_search.md
 - name: Change Data Capture (CDC)
   href: cdc.md
   when: feature_changefeed

ydb/docs/en/core/concepts/toc_i.yaml

@@ -0,0 +1,48 @@
+# Vector search
+
+## Concept of vector search
+
+**Vector search**, also known as [nearest neighbor search](https://en.wikipedia.org/wiki/Nearest_neighbor_search) (NN), is an optimization problem where the goal is to find the nearest vector (or a set of vectors) in a given dataset relative to a specified query vector. The proximity between vectors is determined using distance or similarity metrics.
+
+Vector search is actively used in the following areas:
+
+* recommendation systems
+* semantic search
+* search for similar images
+* anomaly detection
+* classification systems
+
+In addition, **vector search** in {{ ydb-short-name }} is widely applied in machine learning (ML) and artificial intelligence (AI) tasks. It is particularly useful in Retrieval-Augmented Generation (RAG) approaches, which utilize vector search to retrieve relevant information from large volumes of data, significantly enhancing the quality of generative models.
+
+Methods for solving vector search tasks can be divided into three major categories:
+
+* [exact methods](#vector-search-exact)
+* [approximate methods without index](#vector-search-approximate)
+* [approximate methods with index](#vector-search-index)
+
+The choice of a method depends on the number of vectors and the nature of the workload. Exact methods search slowly but update quickly, whereas indexes do the opposite.
+
+## Exact vector search {#vector-search-exact}
+
+{% include [vector_search_exact.md](_includes/vector_search_exact.md) %}
+
+Learn more about [exact vector search](../yql/reference/udf/list/knn.md#exact-vector-search-examples).
+
+## Approximate vector search without index {#vector-search-approximate}
+
+{% include [vector_search_approximate.md](_includes/vector_search_approximate.md) %}
+
+Learn more about [approximate vector search without index](../yql/reference/udf/list/knn.md#approximate-vector-search-examples).
+
+## Approximate vector search with index {#vector-search-index}
+
+When the data volume significantly increases, non-index approaches cease to work within acceptable time limits. In such cases, additional data structures are necessary such as [vector indexes](../dev/vector-indexes.md), which accelerate the search process.
+
+Main advantage:
+
+* ability to work with a large number of vectors
+
+Disadvantages:
+
+* index construction may take considerable time
+* in the current version, data modification operations such as insertion, update, and deletion are not supported