Skip to content

Commit e60d422

Browse files
authored
[Docs] Improve docstring for ray data llm example (#20597)
Signed-off-by: Ricardo Decal <rdecal@anyscale.com>
1 parent 0d914c8 commit e60d422

File tree

1 file changed

+11
-9
lines changed

1 file changed

+11
-9
lines changed

examples/offline_inference/batch_llm_inference.py

Lines changed: 11 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -3,17 +3,19 @@
33
"""
44
This example shows how to use Ray Data for data parallel batch inference.
55
6-
Ray Data is a data processing framework that can handle large datasets
7-
and integrates tightly with vLLM for data-parallel inference.
8-
9-
As of Ray 2.44, Ray Data has a native integration with
10-
vLLM (under ray.data.llm).
6+
Ray Data is a data processing framework that can process very large datasets
7+
with first-class support for vLLM.
118
129
Ray Data provides functionality for:
13-
* Reading and writing to cloud storage (S3, GCS, etc.)
14-
* Automatic sharding and load-balancing across a cluster
15-
* Optimized configuration of vLLM using continuous batching
16-
* Compatible with tensor/pipeline parallel inference as well.
10+
* Reading and writing to most popular file formats and cloud object storage.
11+
* Streaming execution, so you can run inference on datasets that far exceed
12+
the aggregate RAM of the cluster.
13+
* Scale up the workload without code changes.
14+
* Automatic sharding, load-balancing, and autoscaling across a Ray cluster,
15+
with built-in fault-tolerance and retry semantics.
16+
* Continuous batching that keeps vLLM replicas saturated and maximizes GPU
17+
utilization.
18+
* Compatible with tensor/pipeline parallel inference.
1719
1820
Learn more about Ray Data's LLM integration:
1921
https://docs.ray.io/en/latest/data/working-with-llms.html

0 commit comments

Comments
 (0)