Skip to content

Commit 71a9824

Browse files
q-andykolchfa-awsnatebower
authored
Update neural stats API for 3.1 (#10079)
* Update neural stats API for 3.1 Signed-off-by: Andy Qin <qinandy@amazon.com> * Doc review -I Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Use full processor names Signed-off-by: Andy Qin <qinandy@amazon.com> * More renamings Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Apply suggestions from code review Signed-off-by: Nathan Bower <nbower@amazon.com> --------- Signed-off-by: Andy Qin <qinandy@amazon.com> Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> Signed-off-by: Nathan Bower <nbower@amazon.com> Co-authored-by: Fanit Kolchina <kolchfa@amazon.com> Co-authored-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Co-authored-by: Nathan Bower <nbower@amazon.com>
1 parent 1f4f9b5 commit 71a9824

File tree

1 file changed

+254
-79
lines changed

1 file changed

+254
-79
lines changed

‎_vector-search/api/neural.md

Lines changed: 254 additions & 79 deletions
Original file line numberDiff line numberDiff line change
@@ -54,6 +54,12 @@ The following table lists the available query parameters. All query parameters a
5454
| :--- | :--- | :--- |
5555
| `include_metadata` | Boolean | When `true`, includes additional metadata fields for each statistic (see [Available metadata](#available-metadata)). Default is `false`. |
5656
| `flat_stat_paths` | Boolean | When `true`, flattens the JSON response structure for easier parsing. Default is `false`. |
57+
| `include_individual_nodes` | Boolean | When `true`, includes statistics for individual nodes in the `nodes` category. When `false`, excludes the `nodes` category from the response. Default is `true`. |
58+
| `include_all_nodes` | Boolean | When `true`, includes aggregated statistics across all nodes in the `all_nodes` category. When `false`, excludes the `all_nodes` category from the response. Default is `true`. |
59+
| `include_info` | Boolean | When `true`, includes cluster-wide information in the `info` category. When `false`, excludes the `info` category from the response. Default is `true`. |
60+
61+
#### Parameter interactions
62+
5763

5864
#### Example request
5965

@@ -64,96 +70,183 @@ GET /_plugins/_neural/node1,node2/stats/stat1,stat2?include_metadata=true,flat_s
6470

6571
#### Example response
6672

67-
If `include_metadata` is `false`, the response appears as follows:
73+
<details markdown="block">
74+
<summary>
75+
Response
76+
</summary>
77+
{: .text-delta}
6878

6979
```json
80+
GET /_plugins/_neural/stats/
7081
{
71-
"_nodes": {
72-
"total": 1,
73-
"successful": 1,
74-
"failed": 0
75-
},
76-
"cluster_name": "integTest",
77-
"info": {
78-
"cluster_version": "3.0.0",
79-
"processors": {
80-
"ingest": {
81-
"text_embedding_processors_in_pipelines": 0
82-
}
83-
}
84-
},
85-
"all_nodes": {
86-
"processors": {
87-
"ingest": {
88-
"text_embedding_executions": 0
89-
}
90-
}
91-
},
92-
"nodes": {
93-
"896MIkjCSnWBHy-SxB62zQ": {
94-
"processors": {
95-
"ingest": {
96-
"text_embedding_executions": 0
97-
}
98-
}
99-
}
100-
}
82+
"_nodes": {
83+
"total": 1,
84+
"successful": 1,
85+
"failed": 0
86+
},
87+
"cluster_name": "integTest",
88+
"info": {
89+
"cluster_version": "3.1.0",
90+
"processors": {
91+
"search": {
92+
"hybrid": {
93+
"comb_geometric_processors": 0,
94+
"comb_rrf_processors": 0,
95+
"norm_l2_processors": 0,
96+
"norm_minmax_processors": 0,
97+
"comb_harmonic_processors": 0,
98+
"comb_arithmetic_processors": 0,
99+
"norm_zscore_processors": 0,
100+
"rank_based_normalization_processors": 0,
101+
"normalization_processors": 0
102+
},
103+
"rerank_ml_processors": 0,
104+
"rerank_by_field_processors": 0,
105+
"neural_sparse_two_phase_processors": 0,
106+
"neural_query_enricher_processors": 0
107+
},
108+
"ingest": {
109+
"sparse_encoding_processors": 0,
110+
"skip_existing_processors": 0,
111+
"text_image_embedding_processors": 0,
112+
"text_chunking_delimiter_processors": 0,
113+
"text_embedding_processors_in_pipelines": 0,
114+
"text_chunking_fixed_token_length_processors": 0,
115+
"text_chunking_fixed_char_length_processors": 0,
116+
"text_chunking_processors": 0
117+
}
118+
}
119+
},
120+
"all_nodes": {
121+
"query": {
122+
"hybrid": {
123+
"hybrid_query_with_pagination_requests": 0,
124+
"hybrid_query_with_filter_requests": 0,
125+
"hybrid_query_with_inner_hits_requests": 0,
126+
"hybrid_query_requests": 0
127+
},
128+
"neural": {
129+
"neural_query_against_semantic_sparse_requests": 0,
130+
"neural_query_requests": 0,
131+
"neural_query_against_semantic_dense_requests": 0,
132+
"neural_query_against_knn_requests": 0
133+
},
134+
"neural_sparse": {
135+
"neural_sparse_query_requests": 0
136+
}
137+
},
138+
"semantic_highlighting": {
139+
"semantic_highlighting_request_count": 0
140+
},
141+
"processors": {
142+
"search": {
143+
"neural_sparse_two_phase_executions": 0,
144+
"hybrid": {
145+
"comb_harmonic_executions": 0,
146+
"norm_zscore_executions": 0,
147+
"comb_rrf_executions": 0,
148+
"norm_l2_executions": 0,
149+
"rank_based_normalization_processor_executions": 0,
150+
"comb_arithmetic_executions": 0,
151+
"normalization_processor_executions": 0,
152+
"comb_geometric_executions": 0,
153+
"norm_minmax_executions": 0
154+
},
155+
"rerank_by_field_executions": 0,
156+
"neural_query_enricher_executions": 0,
157+
"rerank_ml_executions": 0
158+
},
159+
"ingest": {
160+
"skip_existing_executions": 0,
161+
"text_chunking_fixed_token_length_executions": 0,
162+
"sparse_encoding_executions": 0,
163+
"text_chunking_fixed_char_length_executions": 0,
164+
"text_chunking_executions": 0,
165+
"text_embedding_executions": 0,
166+
"semantic_field_executions": 0,
167+
"semantic_field_chunking_executions": 0,
168+
"text_chunking_delimiter_executions": 0,
169+
"text_image_embedding_executions": 0
170+
}
171+
}
172+
},
173+
"nodes": {
174+
"_cONimhxS6KdedymRZr6xg": {
175+
"query": {
176+
"hybrid": {
177+
"hybrid_query_with_pagination_requests": 0,
178+
"hybrid_query_with_filter_requests": 0,
179+
"hybrid_query_with_inner_hits_requests": 0,
180+
"hybrid_query_requests": 0
181+
},
182+
"neural": {
183+
"neural_query_against_semantic_sparse_requests": 0,
184+
"neural_query_requests": 0,
185+
"neural_query_against_semantic_dense_requests": 0,
186+
"neural_query_against_knn_requests": 0
187+
},
188+
"neural_sparse": {
189+
"neural_sparse_query_requests": 0
190+
}
191+
},
192+
"semantic_highlighting": {
193+
"semantic_highlighting_request_count": 0
194+
},
195+
"processors": {
196+
"search": {
197+
"neural_sparse_two_phase_executions": 0,
198+
"hybrid": {
199+
"comb_harmonic_executions": 0,
200+
"norm_zscore_executions": 0,
201+
"comb_rrf_executions": 0,
202+
"norm_l2_executions": 0,
203+
"rank_based_normalization_processor_executions": 0,
204+
"comb_arithmetic_executions": 0,
205+
"normalization_processor_executions": 0,
206+
"comb_geometric_executions": 0,
207+
"norm_minmax_executions": 0
208+
},
209+
"rerank_by_field_executions": 0,
210+
"neural_query_enricher_executions": 0,
211+
"rerank_ml_executions": 0
212+
},
213+
"ingest": {
214+
"skip_existing_executions": 0,
215+
"text_chunking_fixed_token_length_executions": 0,
216+
"sparse_encoding_executions": 0,
217+
"text_chunking_fixed_char_length_executions": 0,
218+
"text_chunking_executions": 0,
219+
"text_embedding_executions": 0,
220+
"semantic_field_executions": 0,
221+
"semantic_field_chunking_executions": 0,
222+
"text_chunking_delimiter_executions": 0,
223+
"text_image_embedding_executions": 0
224+
}
225+
}
226+
}
227+
}
101228
}
102229
```
103230

104-
If `include_metadata` is `true`, the response appears as follows:
231+
</details>
232+
233+
If `include_metadata` is `true`, each stats object contains additional metadata:
105234

106235
```json
107236
{
108-
"_nodes": {
109-
"total": 1,
110-
"successful": 1,
111-
"failed": 0
112-
},
113-
"cluster_name": "integTest",
114-
"info": {
115-
"cluster_version": {
116-
"value": "3.0.0",
117-
"stat_type": "info_string"
237+
...,
238+
"text_embedding_executions": {
239+
"value": 0,
240+
"stat_type": "timestamped_event_counter",
241+
"trailing_interval_value": 0,
242+
"minutes_since_last_event": 29061801
118243
},
119-
"processors": {
120-
"ingest": {
121-
"text_embedding_processors_in_pipelines": {
122-
"value": 0,
123-
"stat_type": "info_counter"
124-
}
125-
}
126-
}
127-
},
128-
"all_nodes": {
129-
"processors": {
130-
"ingest": {
131-
"text_embedding_executions": {
132-
"value": 0,
133-
"stat_type": "timestamped_event_counter",
134-
"trailing_interval_value": 0,
135-
"minutes_since_last_event": 29061801
136-
}
137-
}
138-
}
139-
},
140-
"nodes": {
141-
"896MIkjCSnWBHy-SxB62zQ": {
142-
"processors": {
143-
"ingest": {
144-
"text_embedding_executions": {
145-
"value": 0,
146-
"stat_type": "timestamped_event_counter",
147-
"trailing_interval_value": 0,
148-
"minutes_since_last_event": 29061801
149-
}
150-
}
151-
}
152-
}
153-
}
244+
...
154245
}
155246
```
156247

248+
For more information, see [Available metadata](#available-metadata).
249+
157250
### Response body fields
158251

159252
The following sections describe response body fields.
@@ -175,8 +268,90 @@ The following table lists the available statistics. For statistics with paths pr
175268
| Statistic name | Category | Statistic path within category | Description |
176269
| :--- | :--- | :--- | :--- |
177270
| `cluster_version` | `info` | `cluster_version` | The version of the cluster. |
178-
| `text_embedding_processors_in_pipelines` | `info` | `processors.ingest.text_embedding_processors_in_pipelines` | The number of text embedding processors configured in ingest pipelines. |
179-
| `text_embedding_executions` | `nodes`, `all_nodes` | `processors.ingest.text_embedding_executions` | The number of text embedding processor executions per node. |
271+
272+
**Info statistics: Processors**
273+
274+
| Statistic name | Category | Statistic path within category | Description |
275+
| :--- | :--- | :--- | :--- |
276+
| `text_embedding_processors_in_pipelines` | `info` | `processors.ingest.text_embedding_processors_in_pipelines` | The number of `text_embedding` processors in ingest pipelines. |
277+
| `sparse_encoding_processors` | `info` | `processors.ingest.sparse_encoding_processors` | The number of `sparse_encoding` processors in ingest pipelines. |
278+
| `skip_existing_processors` | `info` | `processors.ingest.skip_existing_processors` | The number of processors with `skip_existing` set to `true` in ingest pipelines. |
279+
| `text_image_embedding_processors` | `info` | `processors.ingest.text_image_embedding_processors` | The number of `text_image_embedding` processors in ingest pipelines. |
280+
| `text_chunking_delimiter_processors` | `info` | `processors.ingest.text_chunking_delimiter_processors` | The number of `text_chunking` processors using the `delimiter` algorithm in ingest pipelines. |
281+
| `text_chunking_fixed_token_length_processors` | `info` | `processors.ingest.text_chunking_fixed_token_length_processors` | The number of `text_chunking` processors using the `fixed_token_length` algorithm in ingest pipelines. |
282+
| `text_chunking_fixed_char_length_processors` | `info` | `processors.ingest.text_chunking_fixed_char_length_processors` | The number of `text_chunking` processors using the `fixed_character_length` algorithm in ingest pipelines. |
283+
| `text_chunking_processors` | `info` | `processors.ingest.text_chunking_processors` | The number of `text_chunking` processors in ingest pipelines. |
284+
| `rerank_ml_processors` | `info` | `processors.search.rerank_ml_processors` | The number of `rerank` processors of the `ml_opensearch` type in search pipelines. |
285+
| `rerank_by_field_processors` | `info` | `processors.search.rerank_by_field_processors` | The number of `rerank` processors of the `by_field` type. |
286+
| `neural_sparse_two_phase_processors` | `info` | `processors.search.neural_sparse_two_phase_processors` | The number of `neural_sparse_two_phase_processor` processors in search pipelines. |
287+
| `neural_query_enricher_processors` | `info` | `processors.search.neural_query_enricher_processors` | The number of `neural_query_enricher` processors in search pipelines. |
288+
289+
**Info statistics: Hybrid processors**
290+
291+
| Statistic name | Category | Statistic path within category | Description |
292+
| :--- | :--- | :--- | :--- |
293+
| `normalization_processors` | `info` | `processors.search.hybrid.normalization_processors` | The number of `normalization-processor` processors. |
294+
| `norm_minmax_processors` | `info` | `processors.search.hybrid.norm_minmax_processors` | The number of `normalization-processor` processors with `normalization.technique` set to `min_max`. |
295+
| `norm_l2_processors` | `info` | `processors.search.hybrid.norm_l2_processors` | The number of `normalization-processor` processors with `normalization.technique` set to `l2`. |
296+
| `norm_zscore_processors` | `info` | `processors.search.hybrid.norm_zscore_processors` | The number of `normalization-processor` processors with `normalization.technique` set to `z_score`. |
297+
| `comb_arithmetic_processors` | `info` | `processors.search.hybrid.comb_arithmetic_processors` | The number of `normalization-processor` processors with `combination.technique` set to `arithmetic_mean`. |
298+
| `comb_geometric_processors` | `info` | `processors.search.hybrid.comb_geometric_processors` | The number of `normalization-processor` processors with `combination.technique` set to `geometric_mean`. |
299+
| `comb_harmonic_processors` | `info` | `processors.search.hybrid.comb_harmonic_processors` | The number of `normalization-processor` processors with `combination.technique` set to `harmonic_mean`. |
300+
| `rank_based_normalization_processors` | `info` | `processors.search.hybrid.rank_based_normalization_processors` | The number of `score-ranker-processor` processors. |
301+
| `comb_rrf_processors` | `info` | `processors.search.hybrid.comb_rrf_processors` | The number of `score-ranker-processor` processors with `combination.technique` set to `rrf`. |
302+
303+
**Node-level statistics: Processors**
304+
305+
| Statistic name | Category | Statistic path within category | Description |
306+
| :--- | :--- | :--- | :--- |
307+
| `text_embedding_executions` | `nodes`, `all_nodes` | `processors.ingest.text_embedding_executions` | The number of `text_embedding` processor executions. |
308+
| `skip_existing_executions` | `nodes`, `all_nodes` | `processors.ingest.skip_existing_executions` | The number of processor executions that have `skip_existing` set to `true`. |
309+
| `text_chunking_fixed_token_length_executions` | `nodes`, `all_nodes` | `processors.ingest.text_chunking_fixed_token_length_executions` | The number of `text_chunking` processor executions with the `fixed_token_length` algorithm. |
310+
| `sparse_encoding_executions` | `nodes`, `all_nodes` | `processors.ingest.sparse_encoding_executions` | The number of `sparse_encoding` processor executions. |
311+
| `text_chunking_fixed_char_length_executions` | `nodes`, `all_nodes` | `processors.ingest.text_chunking_fixed_char_length_executions` | The number of `text_chunking` processor executions with the `fixed_character_length` algorithm. |
312+
| `text_chunking_executions` | `nodes`, `all_nodes` | `processors.ingest.text_chunking_executions` | The number of `text_chunking` processor executions. |
313+
| `semantic_field_executions` | `nodes`, `all_nodes` | `processors.ingest.semantic_field_executions` | The number of `semantic` field system processor executions. |
314+
| `semantic_field_chunking_executions` | `nodes`, `all_nodes` | `processors.ingest.semantic_field_chunking_executions` | The number of `semantic` field system chunking processor executions. |
315+
| `text_chunking_delimiter_executions` | `nodes`, `all_nodes` | `processors.ingest.text_chunking_delimiter_executions` | The number of `text_chunking` processor executions with the `delimiter` algorithm. |
316+
| `text_image_embedding_executions` | `nodes`, `all_nodes` | `processors.ingest.text_image_embedding_executions` | The number of `text_image_embedding` processor executions. |
317+
| `neural_sparse_two_phase_executions` | `nodes`, `all_nodes` | `processors.search.neural_sparse_two_phase_executions` | The number of `neural_sparse_two_phase_processor` processor executions. |
318+
| `rerank_by_field_executions` | `nodes`, `all_nodes` | `processors.search.rerank_by_field_executions` | The number of `rerank` processor executions of the `by_field` type. |
319+
| `neural_query_enricher_executions` | `nodes`, `all_nodes` | `processors.search.neural_query_enricher_executions` | The number of `neural_query_enricher` processor executions. |
320+
| `rerank_ml_executions` | `nodes`, `all_nodes` | `processors.search.rerank_ml_executions` | The number of `rerank` processor executions of the `ml_opensearch` type. |
321+
322+
**Node-level statistics: Hybrid processors**
323+
324+
| Statistic name | Category | Statistic path within category | Description |
325+
| :--- | :--- | :--- | :--- |
326+
| `normalization_processor_executions` | `nodes`, `all_nodes` | `processors.search.hybrid.normalization_processor_executions` | The number of `normalization-processor` processor executions. |
327+
| `rank_based_normalization_processor_executions` | `nodes`, `all_nodes` | `processors.search.hybrid.rank_based_normalization_processor_executions` | The number of `score-ranker-processor` processor executions. |
328+
| `comb_harmonic_executions` | `nodes`, `all_nodes` | `processors.search.hybrid.comb_harmonic_executions` | The number of `normalization-processor` processor executions with `combination.technique` set to `harmonic_mean`. |
329+
| `norm_zscore_executions` | `nodes`, `all_nodes` | `processors.search.hybrid.norm_zscore_executions` | The number of `normalization-processor` processor executions with `normalization.technique` set to `z_score`. |
330+
| `comb_rrf_executions` | `nodes`, `all_nodes` | `processors.search.hybrid.comb_rrf_executions` | The number of `score-ranker-processor` processor executions with `combination.technique` set to `rrf`. |
331+
| `norm_l2_executions` | `nodes`, `all_nodes` | `processors.search.hybrid.norm_l2_executions` | The number of `normalization-processor` processor executions with `normalization.technique` set to `l2`. |
332+
| `comb_arithmetic_executions` | `nodes`, `all_nodes` | `processors.search.hybrid.comb_arithmetic_executions` | The number of `normalization-processor` processor executions with `combination.technique` set to `arithmetic_mean`. |
333+
| `comb_geometric_executions` | `nodes`, `all_nodes` | `processors.search.hybrid.comb_geometric_executions` | The number of `normalization-processor` processor executions with `combination.technique` set to `geometric_mean`. |
334+
| `norm_minmax_executions` | `nodes`, `all_nodes` | `processors.search.hybrid.norm_minmax_executions` | The number of `normalization-processor` processor executions with `normalization.technique` set to `min_max`. |
335+
336+
**Node-level statistics: Query**
337+
338+
| Statistic name | Category | Statistic path within category | Description |
339+
| :--- | :--- | :--- | :--- |
340+
| `hybrid_query_with_pagination_requests` | `nodes`, `all_nodes` | `query.hybrid.hybrid_query_with_pagination_requests` | The number of `hybrid` query requests with pagination. |
341+
| `hybrid_query_with_filter_requests` | `nodes`, `all_nodes` | `query.hybrid.hybrid_query_with_filter_requests` | The number of `hybrid` query requests with filters. |
342+
| `hybrid_query_with_inner_hits_requests` | `nodes`, `all_nodes` | `query.hybrid.hybrid_query_with_inner_hits_requests` | The number of `hybrid` query requests with inner hits. |
343+
| `hybrid_query_requests` | `nodes`, `all_nodes` | `query.hybrid.hybrid_query_requests` | The total number of `hybrid` query requests. |
344+
| `neural_query_against_semantic_sparse_requests` | `nodes`, `all_nodes` | `query.neural.neural_query_against_semantic_sparse_requests` | The number of `neural` query requests against semantic sparse fields. |
345+
| `neural_query_requests` | `nodes`, `all_nodes` | `query.neural.neural_query_requests` | The total number of `neural` query requests. |
346+
| `neural_query_against_semantic_dense_requests` | `nodes`, `all_nodes` | `query.neural.neural_query_against_semantic_dense_requests` | The number of `neural` query requests against semantic dense fields. |
347+
| `neural_query_against_knn_requests` | `nodes`, `all_nodes` | `query.neural.neural_query_against_knn_requests` | The number of `neural` query requests against k-NN fields. |
348+
| `neural_sparse_query_requests` | `nodes`, `all_nodes` | `query.neural_sparse.neural_sparse_query_requests` | The number of `neural_sparse` query requests. |
349+
350+
**Node-level statistics: Semantic highlighting**
351+
352+
| Statistic name | Category | Statistic path within category | Description |
353+
| :--- | :--- | :--- | :--- |
354+
| `semantic_highlighting_request_count` | `nodes`, `all_nodes` | `semantic_highlighting.semantic_highlighting_request_count` | The number of `semantic` highlighting requests. |
180355

181356
#### Available metadata
182357

0 commit comments

Comments
 (0)