Skip to content

Commit be874f5

Browse files
DOC-4837 added remaining code examples
1 parent 48c40d4 commit be874f5

File tree

1 file changed

+69
-61
lines changed

1 file changed

+69
-61
lines changed

content/develop/clients/jedis/vecsearch.md

Lines changed: 69 additions & 61 deletions
Original file line numberDiff line numberDiff line change
@@ -28,10 +28,9 @@ similarity of an embedding generated from some query text with embeddings stored
2828
or JSON fields, Redis can retrieve documents that closely match the query in terms
2929
of their meaning.
3030

31-
In the example below, we use the
32-
[`sentence-transformers`](https://pypi.org/project/sentence-transformers/)
33-
library to generate vector embeddings to store and index with
34-
Redis Query Engine.
31+
In the example below, we use the [HuggingFace](https://huggingface.co/) model
32+
[`all-mpnet-base-v2`](https://huggingface.co/sentence-transformers/all-mpnet-base-v2)
33+
to generate the vector embeddings to store and index with Redis Query Engine.
3534

3635
## Initialize
3736

@@ -152,43 +151,67 @@ jedis.ftCreate("vector_idx",
152151
);
153152
```
154153

155-
## Define some helper methods
154+
## Define a helper method
155+
156+
The embedding model represents the vectors as an array of `long` integer values,
157+
but Redis Query Engine expects the vector components to be `float` values.
158+
Also, when you store vectors in a hash object, you must encode the vector
159+
array as a `byte` string. To simplify this situation, we declare a helper
160+
method `longsToFloatsByteString()` that takes the `long` array that the
161+
embedding model returns, converts it to an array of `float` values, and
162+
then encodes the `float` array as a `byte` string:
156163

164+
```java
165+
public static byte[] longsToFloatsByteString(long[] input) {
166+
float[] floats = new float[input.length];
167+
for (int i = 0; i < input.length; i++) {
168+
floats[i] = input[i];
169+
}
157170

171+
byte[] bytes = new byte[Float.BYTES * floats.length];
172+
ByteBuffer.wrap(bytes).order(ByteOrder.LITTLE_ENDIAN).asFloatBuffer().put(floats);
173+
return bytes;
174+
}
175+
```
158176

159177
## Add data
160178

161179
You can now supply the data objects, which will be indexed automatically
162180
when you add them with [`hset()`]({{< relref "/commands/hset" >}}), as long as
163181
you use the `doc:` prefix specified in the index definition.
164182

165-
Use the `model.encode()` method of `SentenceTransformer`
183+
Use the `encode()` method of the `sentenceTokenizer` object
166184
as shown below to create the embedding that represents the `content` field.
167-
The `astype()` option that follows the `model.encode()` call specifies that
168-
we want a vector of `float32` values. The `tobytes()` option encodes the
169-
vector components together as a single binary string rather than the
170-
default Python list of `float` values.
171-
Use the binary string representation when you are indexing hash objects
172-
(as we are here), but use the default list of `float` for JSON objects.
185+
The `getIds()` method that follows the `encode()` call obtains the vector
186+
of `long` values which we then convert to a `float` array stored as a `byte`
187+
string. Use the `byte` string representation when you are indexing hash
188+
objects (as we are here), but use the default list of `float` for JSON objects.
173189

174190
```java
175191
String sentence1 = "That is a very happy person";
176192
jedis.hset("doc:1", Map.of( "content", sentence1, "genre", "persons"));
177193
jedis.hset(
178194
"doc:1".getBytes(),
179195
"embedding".getBytes(),
180-
longArrayToByteArray(sentenceTokenizer.encode(sentence1).getIds())
196+
longsToFloatsByteString(sentenceTokenizer.encode(sentence1).getIds())
181197
);
182198

183199
String sentence2 = "That is a happy dog";
184200
jedis.hset("doc:2", Map.of( "content", sentence2, "genre", "pets"));
185-
jedis.hset("doc:2".getBytes(), "embedding".getBytes(), longArrayToByteArray(sentenceTokenizer.encode(sentence2).getIds()));
201+
jedis.hset(
202+
"doc:2".getBytes(),
203+
"embedding".getBytes(),
204+
longsToFloatsByteString(sentenceTokenizer.encode(sentence2).getIds())
205+
);
186206

187207
String sentence3 = "Today is a sunny day";
188208
Map<String, String> doc3 = Map.of( "content", sentence3, "genre", "weather");
189209
jedis.hset("doc:3", doc3);
190-
jedis.hset("doc:3".getBytes(), "embedding".getBytes(), longArrayToByteArray(sentenceTokenizer.encode(sentence3).getIds()));
191-
210+
jedis.hset(
211+
"doc:3".getBytes(),
212+
"embedding".getBytes(),
213+
longsToFloatsByteString(sentenceTokenizer.encode(sentence3).getIds())
214+
);
192215
```
193216

194217
## Run a query
@@ -199,58 +222,43 @@ text. Redis calculates the similarity between the query vector and each
199222
embedding vector in the index as it runs the query. It then ranks the
200223
results in order of this numeric similarity value.
201224

202-
The code below creates the query embedding using `model.encode()`, as with
203-
the indexing, and passes it as a parameter when the query executes
204-
(see
225+
The code below creates the query embedding using the `encode()` method, as with
226+
the indexing, and passes it as a parameter when the query executes (see
205227
[Vector search]({{< relref "/develop/interact/search-and-query/query/vector-search" >}})
206228
for more information about using query parameters with embeddings).
207229

208-
```python
209-
q = Query(
210-
"*=>[KNN 3 @embedding $vec AS vector_distance]"
211-
).return_field("score").dialect(2)
212-
213-
query_text = "That is a happy person"
214-
215-
res = r.ft("vector_idx").search(
216-
q, query_params={
217-
"vec": model.encode(query_text).astype(np.float32).tobytes()
218-
}
219-
)
220-
221-
print(res)
230+
```java
231+
String sentence = "That is a happy person";
232+
233+
int K = 3;
234+
Query q = new Query("*=>[KNN $K @embedding $BLOB AS score]").
235+
returnFields("content", "score").
236+
addParam("K", K).
237+
addParam(
238+
"BLOB",
239+
longsToFloatsByteString(
240+
sentenceTokenizer.encode(sentence).getIds()
241+
)
242+
).
243+
dialect(2);
244+
245+
List<Document> docs = jedis.ftSearch("vector_idx", q).getDocuments();
246+
247+
for (Document doc: docs) {
248+
System.out.println(doc);
249+
}
222250
```
223251

224252
The code is now ready to run, but note that it may take a while to complete when
225-
you run it for the first time (which happens because RedisVL must download the
226-
`all-MiniLM-L6-v2` model data before it can
253+
you run it for the first time (which happens because the tokenizer must download the
254+
`all-mpnet-base-v2` model data before it can
227255
generate the embeddings). When you run the code, it outputs the following result
228-
object (slightly formatted here for clarity):
229-
230-
```Python
231-
Result{
232-
3 total,
233-
docs: [
234-
Document {
235-
'id': 'doc:0',
236-
'payload': None,
237-
'vector_distance': '0.114169985056',
238-
'content': 'That is a very happy person'
239-
},
240-
Document {
241-
'id': 'doc:1',
242-
'payload': None,
243-
'vector_distance': '0.610845386982',
244-
'content': 'That is a happy dog'
245-
},
246-
Document {
247-
'id': 'doc:2',
248-
'payload': None,
249-
'vector_distance': '1.48624813557',
250-
'content': 'Today is a sunny day'
251-
}
252-
]
253-
}
256+
objects:
257+
258+
```
259+
id:doc:1, score: 1.0, properties:[score=9301635, content=That is a very happy person]
260+
id:doc:2, score: 1.0, properties:[score=1411344, content=That is a happy dog]
261+
id:doc:3, score: 1.0, properties:[score=67178800, content=Today is a sunny day]
254262
```
255263

256264
Note that the results are ordered according to the value of the `vector_distance`

0 commit comments

Comments
 (0)