Skip to content

Commit 9ec1e30

Browse files
authored
[Misc][Doc] Add missing comment for LLM (#20285)
Signed-off-by: Lifan Shen <lifans@meta.com>
1 parent 9dae7d4 commit 9ec1e30

File tree

1 file changed

+20
-12
lines changed

1 file changed

+20
-12
lines changed

vllm/entrypoints/llm.py

Lines changed: 20 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -132,6 +132,14 @@ class LLM:
132132
hf_overrides: If a dictionary, contains arguments to be forwarded to the
133133
HuggingFace config. If a callable, it is called to update the
134134
HuggingFace config.
135+
mm_processor_kwargs: Arguments to be forwarded to the model's processor
136+
for multi-modal data, e.g., image processor. Overrides for the
137+
multi-modal processor obtained from `AutoProcessor.from_pretrained`.
138+
The available overrides depend on the model that is being run.
139+
For example, for Phi-3-Vision: `{"num_crops": 4}`.
140+
override_pooler_config: Initialize non-default pooling config or
141+
override default pooling config for the pooling model.
142+
e.g. `PoolerConfig(pooling_type="mean", normalize=False)`.
135143
compilation_config: Either an integer or a dictionary. If it is an
136144
integer, it is used as the level of compilation optimization. If it
137145
is a dictionary, it can specify the full compilation configuration.
@@ -1347,16 +1355,16 @@ def sleep(self, level: int = 1):
13471355
during the sleep period, before `wake_up` is called.
13481356
13491357
Args:
1350-
level: The sleep level. Level 1 sleep will offload the model
1351-
weights and discard the kv cache. The content of kv cache
1358+
level: The sleep level. Level 1 sleep will offload the model
1359+
weights and discard the kv cache. The content of kv cache
13521360
is forgotten. Level 1 sleep is good for sleeping and waking
1353-
up the engine to run the same model again. The model weights
1354-
are backed up in CPU memory. Please make sure there's enough
1355-
CPU memory to store the model weights. Level 2 sleep will
1356-
discard both the model weights and the kv cache. The content
1357-
of both the model weights and kv cache is forgotten. Level 2
1361+
up the engine to run the same model again. The model weights
1362+
are backed up in CPU memory. Please make sure there's enough
1363+
CPU memory to store the model weights. Level 2 sleep will
1364+
discard both the model weights and the kv cache. The content
1365+
of both the model weights and kv cache is forgotten. Level 2
13581366
sleep is good for sleeping and waking up the engine to run a
1359-
different model or update the model, where previous model
1367+
different model or update the model, where previous model
13601368
weights are not needed. It reduces CPU memory pressure.
13611369
"""
13621370
self.reset_prefix_cache()
@@ -1366,12 +1374,12 @@ def wake_up(self, tags: Optional[list[str]] = None):
13661374
"""
13671375
Wake up the engine from sleep mode. See the [sleep][] method
13681376
for more details.
1369-
1377+
13701378
Args:
1371-
tags: An optional list of tags to reallocate the engine memory
1372-
for specific memory allocations. Values must be in
1379+
tags: An optional list of tags to reallocate the engine memory
1380+
for specific memory allocations. Values must be in
13731381
`("weights", "kv_cache")`. If None, all memory is reallocated.
1374-
wake_up should be called with all tags (or None) before the
1382+
wake_up should be called with all tags (or None) before the
13751383
engine is used again.
13761384
"""
13771385
self.llm_engine.wake_up(tags)

0 commit comments

Comments
 (0)