|
| 1 | +:sectnums: |
| 2 | +:sectnumlevels: 5 |
| 3 | + |
| 4 | += pgvector |
| 5 | + |
| 6 | +== Overview |
| 7 | + |
| 8 | +The vector database is an important component of Generative Artificial Intelligence (GenAI). As a significant extension of PostgreSQL, pgvector not only supports vector calculations of up to 16000 dimensions but also provides powerful vector operations and indexing capabilities, enabling PostgreSQL to directly transform into an efficient vector database. IvorySQL, being developed based on PostgreSQL, inherits the seamless integration capability with pgvector extension, thereby offering users a wider range of data processing and analysis options. Additionally, in Oracle compatibility mode, the pgvector extension is also available, providing great convenience for Oracle users to use vector databases, allowing for easy migration and management of data and achieving more efficient business operations. |
| 9 | + |
| 10 | + |
| 11 | +== Principles |
| 12 | + |
| 13 | +PGVector has two indexing algorithms, IVFFLAT and HNSW. |
| 14 | + |
| 15 | +=== IVFFLAT |
| 16 | + |
| 17 | +The working principle of IVFFLAT is to cluster similar vectors into regions and build an inverted index mapping each region to its vectors. This allows queries to focus on a subset of the data, enabling fast searches. By adjusting the parameters of lists and probes, IVFFLAT can balance the speed and accuracy of the dataset, enabling PostgreSQL to perform rapid semantic similarity searches on complex data. Through simple queries, applications can find the nearest neighbors to a query vector among millions of high-dimensional vectors. For tasks such as natural language processing and information retrieval, IVFFLAT provides an effective solution. |
| 18 | + |
| 19 | +When building an IVFFLAT index, you need to decide how many lists to include in the index. Each list represents a "center" which are computed using the k-means algorithm. Once all centers are determined, IVFFLAT determines which center each vector is closest to and adds it to the index. When querying vector data, you can decide how many centers to check, which is determined by the ivfflat.probes parameter. This results in a trade-off between ANN performance/recall: the more centers accessed, the more accurate the results, but at the expense of performance. |
| 20 | + |
| 21 | +=== HNSW |
| 22 | + |
| 23 | + |
| 24 | +HNSW (Hierarchical Navigating Small World) is a graph-based indexing algorithm consisting of multiple layers of neighborhood graphs, hence the name "hierarchical" NSW method. It constructs multiple layers of navigation graphs for a given graph according to certain rules, with the upper layers of the graph being sparser and the distances between nodes farther apart; and the lower layers of the graph being denser and the distances between nodes closer together. HNSW algorithm is a classic trade-off between space and time, as it achieves high search quality and speed, but at the cost of significant memory overhead. This is because it not only requires storing all vectors in memory but also maintaining the structure of the graph, which also needs to be stored. |
| 25 | + |
| 26 | +== Installation |
| 27 | +[TIP] |
| 28 | +==== |
| 29 | +The IvorySQL 3.0(above version) has been installed in the environment, and the installation path is /usr/local/ivorysql/ivorysql-3 |
| 30 | +==== |
| 31 | + |
| 32 | +=== Source Code Installation |
| 33 | + |
| 34 | +** Setting PG_CONFIG |
| 35 | +``` |
| 36 | +export PG_CONFIG=/usr/local/ivorysql/ivorysql-3/bin/pg_config |
| 37 | +``` |
| 38 | + |
| 39 | +** Pull pg_vector source code |
| 40 | +``` |
| 41 | +git clone --branch v0.6.2 https://github.com/pgvector/pgvector.git |
| 42 | +``` |
| 43 | + |
| 44 | +** Install pgvector |
| 45 | +``` |
| 46 | +cd pgvector |
| 47 | + |
| 48 | +sudo --preserve-env=PG_CONFIG make |
| 49 | +sudo --preserve-env=PG_CONFIG make install |
| 50 | +``` |
| 51 | + |
| 52 | +** Create pgvector extension |
| 53 | +``` |
| 54 | +[ivorysql@localhost ivorysql-3]$ psql |
| 55 | +psql (16.2) |
| 56 | +Type "help" for help. |
| 57 | + |
| 58 | +ivorysql=# create extension vector; |
| 59 | +CREATE EXTENSION |
| 60 | +``` |
| 61 | +Now, pgvector is installed completely. |
| 62 | +For more usage cases, please refer to https://github.com/pgvector/pgvector?tab=readme-ov-file#getting-started[pgvector文档] |
| 63 | + |
| 64 | +== Oracle Compatible |
| 65 | +In IvorySQL's Oracle compatibility mode, the pgvector extension can also work correctly. |
| 66 | +[TIP] |
| 67 | +We suggest users to test using port 1521, using the command: psql -p 1521. |
| 68 | + |
| 69 | +=== Data Type |
| 70 | + |
| 71 | +``` |
| 72 | +ivorysql=# CREATE TABLE items5 (id bigserial PRIMARY KEY, name varchar2(20), num number(20), embedding bit(3)); |
| 73 | +CREATE TABLE |
| 74 | +ivorysql=# INSERT INTO items5 (name, num, embedding) VALUES ('1st oracle data',0, '000'), ('2nd oracle data', 111, '111'); |
| 75 | +INSERT 0 2 |
| 76 | +ivorysql=# SELECT * FROM items5 ORDER BY bit_count(embedding # '101') LIMIT 5; |
| 77 | + id | name | num | embedding |
| 78 | +----+-----------------+-----+----------- |
| 79 | + 2 | 2nd oracle data | 111 | 111 |
| 80 | + 1 | 1st oracle data | 0 | 000 |
| 81 | +``` |
| 82 | +
|
| 83 | +=== Anonymous Block |
| 84 | +
|
| 85 | +``` |
| 86 | +ivorysql=# declare |
| 87 | +i vector(3) := '[1,2,3]'; |
| 88 | +begin |
| 89 | +raise notice '%', i; |
| 90 | +end; |
| 91 | +ivorysql-# / |
| 92 | +NOTICE: [1,2,3] |
| 93 | +DO |
| 94 | +``` |
| 95 | +
|
| 96 | +=== PROCEDURE |
| 97 | +``` |
| 98 | +ivorysql=# CREATE OR REPLACE PROCEDURE ora_procedure() |
| 99 | +AS |
| 100 | +p vector(3) := '[4,5,6]'; |
| 101 | +begin |
| 102 | +raise notice '%', p; |
| 103 | +end; |
| 104 | +/ |
| 105 | +CREATE PROCEDURE |
| 106 | +ivorysql=# call ora_procedure(); |
| 107 | +NOTICE: [4,5,6] |
| 108 | +CALL |
| 109 | +``` |
| 110 | +
|
| 111 | +==== FUNCTION |
| 112 | +``` |
| 113 | +ivorysql=# CREATE OR REPLACE FUNCTION AddVector(a vector(3), b vector(3)) |
| 114 | +RETURN vector(3) |
| 115 | +IS |
| 116 | +BEGIN |
| 117 | +RETURN a + b; |
| 118 | +END; |
| 119 | +/ |
| 120 | +CREATE FUNCTION |
| 121 | +ivorysql=# SELECT AddVector('[1,2,3]','[4,5,6]') FROM DUAL; |
| 122 | + addvector |
| 123 | +---------------- |
| 124 | + [5,7,9] |
| 125 | +(1 row) |
| 126 | +``` |
0 commit comments