vllm + 5090 series problem

[changtimwu]

https://www.reddit.com/r/StableDiffusion/comments/1jco4kw/rtx_5090_help_with_cuda_error_no_kernel_image_is/

[changtimwu]

on 5090

model="Qwen/Qwen3-0.6B-FP8"
vllmtag="latest"

docker run --runtime nvidia --gpus all \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    -p 18000:8000 \
    --rm \
    --ipc=host \
    vllm/vllm-openai:$vllmtag \
    --model $model

error message
session.log

no such a problem when using model="Qwen/Qwen3-0.6B"

on 6000ada(on vast.ai)

vast instance activation command

vastai create instance <OFFER_ID> --image vastai/vllm:@vastai-automatic-tag --env '-p 1111:1111 -p 8080:8080 -p 8000:8000 -p 8265:8265 -e OPEN_BUTTON_PORT=1111 -e OPEN_BUTTON_TOKEN=1 -e JUPYTER_DIR=/ -e DATA_DIRECTORY=/workspace/ -e PORTAL_CONFIG="localhost:1111:11111:/:Instance Portal|localhost:8000:18000:/docs:vLLM API|localhost:8265:28265:/:Ray Dashboard|localhost:8080:18080:/:Jupyter|localhost:8080:8080:/terminals/1:Jupyter Terminal" -e VLLM_MODEL=Qwen/Qwen3-0.6B-FP8 -e VLLM_ARGS="--max-model-len 32768  --enforce-eager --download-dir /workspace/models --host 127.0.0.1 --port 18000" -e RAY_ARGS="--head --port 6379  --dashboard-host 127.0.0.1 --dashboard-port 28265" -e RAY_ADDRESS=127.0.0.1:6379 -e USE_ALL_GPUS=true' --onstart-cmd 'entrypoint.sh' --disk 70 --jupyter --ssh --direct

succesfully deployed

SSH key propagation complete.
Refusing to start jupyter (/.launch managing)
Waiting for /etc/portal.yaml before starting instance_portal...
Waiting for /etc/portal.yaml before starting ray...
Waiting for /etc/portal.yaml before starting tunnel_manager...
Waiting for /etc/portal.yaml before starting vllm...
Waiting for /etc/portal.yaml before starting syncthing...
Skipping startup for syncthing (not in /etc/portal.yaml)
Waiting for /etc/portal.yaml before starting tensorboard...
Skipping startup for tensorboard (not in /etc/portal.yaml)
*****
*
*
* Automatic login is enabled via the 'Open' button
* Your web credentials are: vastai / ce5f41489c25f25962ad88a07e4c78fd1665825c7a3690efb093b1ec406071e8
*
* To make API requests, pass an Authorization header (Authorization: Bearer ce5f41489c25f25962ad88a07e4c78fd1665825c7a3690efb093b1ec406071e8)
*
*
*****
Starting Caddy...
{"level":"info","ts":1751882102.6283014,"msg":"using config from file","file":"/etc/Caddyfile"}
{"level":"info","ts":1751882102.6345897,"msg":"adapted config to JSON","adapter":"caddyfile"}
{"level":"info","ts":1751882102.6369965,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
{"level":"info","ts":1751882102.6374416,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc000697d80"}
{"level":"warn","ts":1751882102.6455505,"logger":"http","msg":"HTTP/3 skipped because it requires TLS","network":"tcp","addr":":1111"}
{"level":"warn","ts":1751882102.6456919,"logger":"http","msg":"HTTP/2 skipped because it requires TLS","network":"tcp","addr":":1111"}
{"level":"info","ts":1751882102.6456969,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
{"level":"warn","ts":1751882102.6457033,"logger":"http","msg":"HTTP/3 skipped because it requires TLS","network":"tcp","addr":":8000"}
{"level":"warn","ts":1751882102.6457288,"logger":"http","msg":"HTTP/2 skipped because it requires TLS","network":"tcp","addr":":8000"}
{"level":"info","ts":1751882102.6457322,"logger":"http.log","msg":"server running","name":"srv1","protocols":["h1","h2","h3"]}
{"level":"warn","ts":1751882102.6457374,"logger":"http","msg":"HTTP/3 skipped because it requires TLS","network":"tcp","addr":":8265"}
{"level":"warn","ts":1751882102.64576,"logger":"http","msg":"HTTP/2 skipped because it requires TLS","network":"tcp","addr":":8265"}
{"level":"info","ts":1751882102.6457636,"logger":"http.log","msg":"server running","name":"srv2","protocols":["h1","h2","h3"]}
{"level":"info","ts":1751882102.645988,"msg":"autosaved config (load with --resume flag)","file":"/root/.config/caddy/autosave.json"}
{"level":"info","ts":1751882102.6459942,"msg":"serving initial configuration"}
{"level":"info","ts":1751882102.6533563,"logger":"tls","msg":"cleaning storage unit","storage":"FileStorage:/root/.local/share/caddy"}
{"level":"info","ts":1751882102.6543608,"logger":"tls","msg":"finished cleaning storage units"}
2025-07-07 09:55:02,810	INFO usage_lib.py:467 -- Usage stats collection is enabled by default without user confirmation because this terminal is detected to be non-interactive. To disable this, add `--disable-usage-stats` to the command that starts the cluster, or run the following command: `ray disable-usage-stats` before starting the cluster. See https://docs.ray.io/en/master/cluster/usage-stats.html for more details.
2025-07-07 09:55:02,811	INFO scripts.py:971 -- �[37mLocal node IP�[39m: �[1m172.17.0.8�[22m
2025-07-07 09:55:04,742	WARNING utils.py:592 -- Detecting docker specified CPUs. In previous versions of Ray, CPU detection in containers was incorrect. Please ensure that Ray has enough CPUs allocated. As a temporary workaround to revert to the prior behavior, set `RAY_USE_MULTIPROCESSING_CPU_COUNT=1` as an env var before starting Ray. Set the env var: `RAY_DISABLE_DOCKER_CPU_WARNING=1` to mute this warning.
2025-07-07 09:55:04,743	WARNING utils.py:604 -- Ray currently does not support initializing Ray with fractional cpus. Your num_cpus will be truncated from 11.52 to 11.
2025-07-07 09:55:04,926	SUCC scripts.py:1007 -- �[32m--------------------�[39m
2025-07-07 09:55:04,926	SUCC scripts.py:1008 -- �[32mRay runtime started.�[39m
2025-07-07 09:55:04,927	SUCC scripts.py:1009 -- �[32m--------------------�[39m
2025-07-07 09:55:04,927	INFO scripts.py:1011 -- �[36mNext steps�[39m
2025-07-07 09:55:04,927	INFO scripts.py:1014 -- To add another node to this Ray cluster, run
2025-07-07 09:55:04,927	INFO scripts.py:1017 -- �[1m  ray start --address='172.17.0.8:6379'�[22m
2025-07-07 09:55:04,927	INFO scripts.py:1026 -- To connect to this Ray cluster:
2025-07-07 09:55:04,927	INFO scripts.py:1028 -- �[35mimport�[39m�[26m ray
2025-07-07 09:55:04,927	INFO scripts.py:1029 -- ray�[35m.�[39m�[26minit()
2025-07-07 09:55:04,927	INFO scripts.py:1041 -- To submit a Ray job using the Ray Jobs CLI:
2025-07-07 09:55:04,927	INFO scripts.py:1042 -- �[1m  RAY_ADDRESS='http://127.0.0.1:28265' ray job submit --working-dir . -- python my_script.py�[22m
2025-07-07 09:55:04,927	INFO scripts.py:1051 -- See https://docs.ray.io/en/latest/cluster/running-applications/job-submission/index.html
2025-07-07 09:55:04,927	INFO scripts.py:1055 -- for more information on submitting Ray jobs to the Ray cluster.
2025-07-07 09:55:04,927	INFO scripts.py:1060 -- To terminate the Ray runtime, run
2025-07-07 09:55:04,927	INFO scripts.py:1061 -- �[1m  ray stop�[22m
2025-07-07 09:55:04,928	INFO scripts.py:1064 -- To view the status of the cluster, use
2025-07-07 09:55:04,928	INFO scripts.py:1065 --   �[1mray status�[22m�[26m
2025-07-07 09:55:04,928	INFO scripts.py:1069 -- To monitor and debug Ray, view the dashboard at
2025-07-07 09:55:04,928	INFO scripts.py:1070 --   �[1m127.0.0.1:28265�[22m�[26m
2025-07-07 09:55:04,928	INFO scripts.py:1077 -- �[4mIf connection to the dashboard fails, check your firewall settings and network configuration.�[24m
INFO 07-07 09:55:09 [__init__.py:244] Automatically detected platform cuda.
INFO 07-07 09:55:14 [api_server.py:1287] vLLM API server version 0.9.1
INFO 07-07 09:55:15 [cli_args.py:309] non-default args: {'host': '127.0.0.1', 'port': 18000, 'model': 'Qwen/Qwen3-0.6B-FP8', 'max_model_len': 32768, 'enforce_eager': True, 'download_dir': '/workspace/models'}
INFO 07-07 09:55:22 [config.py:823] This model supports multiple tasks: {'embed', 'score', 'generate', 'classify', 'reward'}. Defaulting to 'generate'.
INFO 07-07 09:55:22 [config.py:2195] Chunked prefill is enabled with max_num_batched_tokens=2048.
WARNING 07-07 09:55:22 [cuda.py:91] To see benefits of async output processing, enable CUDA graph. Since, enforce-eager is enabled, async output processor cannot be used
WARNING 07-07 09:55:26 [env_override.py:17] NCCL_CUMEM_ENABLE is set to 0, skipping override. This may increase memory overhead with cudagraph+allreduce: https://github.com/NVIDIA/nccl/issues/1234
INFO 07-07 09:55:28 [__init__.py:244] Automatically detected platform cuda.
INFO 07-07 09:55:30 [core.py:455] Waiting for init message from front-end.
INFO 07-07 09:55:30 [core.py:70] Initializing a V1 LLM engine (v0.9.1) with config: model='Qwen/Qwen3-0.6B-FP8', speculative_config=None, tokenizer='Qwen/Qwen3-0.6B-FP8', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config={}, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=32768, download_dir='/workspace/models', load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=fp8, enforce_eager=True, kv_cache_dtype=auto,  device_config=cuda, decoding_config=DecodingConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_backend=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=Qwen/Qwen3-0.6B-FP8, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=True, use_async_output_proc=False, pooler_config=None, compilation_config={"level":0,"debug_dump_path":"","cache_dir":"","backend":"","custom_ops":[],"splitting_ops":[],"use_inductor":true,"compile_sizes":[],"inductor_compile_config":{"enable_auto_functionalized_v2":false},"inductor_passes":{},"use_cudagraph":true,"cudagraph_num_of_warmups":0,"cudagraph_capture_sizes":[],"cudagraph_copy_inputs":false,"full_cuda_graph":false,"max_capture_size":0,"local_cache_dir":null}
WARNING 07-07 09:55:31 [utils.py:2737] Methods determine_num_available_blocks,device_config,get_cache_block_size_bytes,initialize_cache not implemented in 
INFO 07-07 09:55:32 [parallel_state.py:1065] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0
INFO 07-07 09:55:32 [topk_topp_sampler.py:49] Using FlashInfer for top-p & top-k sampling.
INFO 07-07 09:55:32 [gpu_model_runner.py:1595] Starting to load model Qwen/Qwen3-0.6B-FP8...
INFO 07-07 09:55:32 [gpu_model_runner.py:1600] Loading model from scratch...
INFO 07-07 09:55:32 [cuda.py:252] Using Flash Attention backend on V1 engine.
INFO 07-07 09:55:32 [weight_utils.py:292] Using model weights format ['*.safetensors']
INFO 07-07 09:55:35 [weight_utils.py:308] Time spent downloading weights for Qwen/Qwen3-0.6B-FP8: 2.403304 seconds
INFO 07-07 09:55:35 [weight_utils.py:345] No model.safetensors.index.json found in remote.
Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  6.86it/s]
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  6.85it/s]
INFO 07-07 09:55:35 [default_loader.py:272] Loading weights took 0.17 seconds
INFO 07-07 09:55:35 [gpu_model_runner.py:1624] Model loading took 0.7101 GiB and 3.128133 seconds
WARNING 07-07 09:55:36 [fp8_utils.py:526] Using default W8A8 Block FP8 kernel config. Performance might be sub-optimal! Config file not found at /venv/main/lib/python3.12/site-packages/vllm/model_executor/layers/quantization/utils/configs/N=4096,K=1024,device_name=NVIDIA_RTX_6000_Ada_Generation,dtype=fp8_w8a8,block_shape=[128,128].json
WARNING 07-07 09:55:37 [fp8_utils.py:526] Using default W8A8 Block FP8 kernel config. Performance might be sub-optimal! Config file not found at /venv/main/lib/python3.12/site-packages/vllm/model_executor/layers/quantization/utils/configs/N=1024,K=2048,device_name=NVIDIA_RTX_6000_Ada_Generation,dtype=fp8_w8a8,block_shape=[128,128].json
WARNING 07-07 09:55:37 [fp8_utils.py:526] Using default W8A8 Block FP8 kernel config. Performance might be sub-optimal! Config file not found at /venv/main/lib/python3.12/site-packages/vllm/model_executor/layers/quantization/utils/configs/N=6144,K=1024,device_name=NVIDIA_RTX_6000_Ada_Generation,dtype=fp8_w8a8,block_shape=[128,128].json
WARNING 07-07 09:55:37 [fp8_utils.py:526] Using default W8A8 Block FP8 kernel config. Performance might be sub-optimal! Config file not found at /venv/main/lib/python3.12/site-packages/vllm/model_executor/layers/quantization/utils/configs/N=1024,K=3072,device_name=NVIDIA_RTX_6000_Ada_Generation,dtype=fp8_w8a8,block_shape=[128,128].json
/venv/main/lib/python3.12/site-packages/torch/utils/cpp_extension.py:2356: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'].
warnings.warn(
INFO 07-07 09:56:16 [gpu_worker.py:227] Available KV cache memory: 38.67 GiB
INFO 07-07 09:56:16 [kv_cache_utils.py:715] GPU KV cache size: 362,048 tokens
INFO 07-07 09:56:16 [kv_cache_utils.py:719] Maximum concurrency for 32,768 tokens per request: 11.05x
INFO 07-07 09:56:16 [core.py:171] init engine (profile, create kv cache, warmup model) took 40.57 seconds
INFO 07-07 09:56:16 [loggers.py:137] Engine 000: vllm cache_config_info with initialization after num_gpu_blocks is: 22628
INFO 07-07 09:56:17 [api_server.py:1349] Starting vLLM API server 0 on http://127.0.0.1:18000
INFO 07-07 09:56:17 [launcher.py:29] Available routes are:
INFO 07-07 09:56:17 [launcher.py:37] Route: /openapi.json, Methods: GET, HEAD
INFO 07-07 09:56:17 [launcher.py:37] Route: /docs, Methods: GET, HEAD
INFO 07-07 09:56:17 [launcher.py:37] Route: /docs/oauth2-redirect, Methods: GET, HEAD
INFO 07-07 09:56:17 [launcher.py:37] Route: /redoc, Methods: GET, HEAD
INFO 07-07 09:56:17 [launcher.py:37] Route: /health, Methods: GET
INFO 07-07 09:56:17 [launcher.py:37] Route: /load, Methods: GET
INFO 07-07 09:56:17 [launcher.py:37] Route: /ping, Methods: POST
INFO 07-07 09:56:17 [launcher.py:37] Route: /ping, Methods: GET
INFO 07-07 09:56:17 [launcher.py:37] Route: /tokenize, Methods: POST
INFO 07-07 09:56:17 [launcher.py:37] Route: /detokenize, Methods: POST
INFO 07-07 09:56:17 [launcher.py:37] Route: /v1/models, Methods: GET
INFO 07-07 09:56:17 [launcher.py:37] Route: /version, Methods: GET
INFO 07-07 09:56:17 [launcher.py:37] Route: /v1/chat/completions, Methods: POST
INFO 07-07 09:56:17 [launcher.py:37] Route: /v1/completions, Methods: POST
INFO 07-07 09:56:17 [launcher.py:37] Route: /v1/embeddings, Methods: POST
INFO 07-07 09:56:17 [launcher.py:37] Route: /pooling, Methods: POST
INFO 07-07 09:56:17 [launcher.py:37] Route: /classify, Methods: POST
INFO 07-07 09:56:17 [launcher.py:37] Route: /score, Methods: POST
INFO 07-07 09:56:17 [launcher.py:37] Route: /v1/score, Methods: POST
INFO 07-07 09:56:17 [launcher.py:37] Route: /v1/audio/transcriptions, Methods: POST
INFO 07-07 09:56:17 [launcher.py:37] Route: /rerank, Methods: POST
INFO 07-07 09:56:17 [launcher.py:37] Route: /v1/rerank, Methods: POST
INFO 07-07 09:56:17 [launcher.py:37] Route: /v2/rerank, Methods: POST
INFO 07-07 09:56:17 [launcher.py:37] Route: /invocations, Methods: POST
INFO 07-07 09:56:17 [launcher.py:37] Route: /metrics, Methods: GET
INFO:     Started server process [1023]
INFO:     Waiting for application startup.
INFO:     Application startup complete.

[changtimwu]

5090 embedding mode

server activation script

model="Qwen/Qwen3-Embedding-0.6B"
#vllmtag="v0.8.5"
vllmtag="latest"

docker run --runtime nvidia --gpus all \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    -p 18000:8000 \
    --rm \
    --ipc=host \
    vllm/vllm-openai:$vllmtag \
    --model $model \
    --task embed

client side script

host="http://localhost:18000"
#host="http://192.168.1.222:9997"

curl -X POST ${host}/v1/embeddings   -H "Content-Type: application/json"   -d '{
    "input": ["Instruct: Given a web search query, retrieve relevant passages that answer the query\nQuery: What is the capital of China?"],
    "model": "Qwen/Qwen3-Embedding-0.6B"
  }'

client side error

./testq3emb.sh

{"object":"error","message":"Engine loop is not running. Inspect the stacktrace to find the original error: RuntimeError('CUDA error: no kernel image is available for execution on the device\\nCUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\\nFor debugging consider passing CUDA_LAUNCH_BLOCKING=1\\nCompile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.\\n').","type":"BadRequestError","param":null,"code":400}{"object":"error","message":"The model `Qwen3-Embedding-0.6B` does not exist.","type":"NotFoundError","param":null,"code":404}

server side

ERROR 07-07 03:10:47 [engine.py:165]     return forward_call(*args, **kwargs)
ERROR 07-07 03:10:47 [engine.py:165]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 07-07 03:10:47 [engine.py:165]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/linear.py", line 487, in forward
ERROR 07-07 03:10:47 [engine.py:165]     output_parallel = self.quant_method.apply(self, input_, bias)
ERROR 07-07 03:10:47 [engine.py:165]                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 07-07 03:10:47 [engine.py:165]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/linear.py", line 203, in apply
ERROR 07-07 03:10:47 [engine.py:165]     return dispatch_unquantized_gemm()(x, layer.weight, bias)
ERROR 07-07 03:10:47 [engine.py:165]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 07-07 03:10:47 [engine.py:165] RuntimeError: CUDA error: no kernel image is available for execution on the device
ERROR 07-07 03:10:47 [engine.py:165] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
ERROR 07-07 03:10:47 [engine.py:165] For debugging consider passing CUDA_LAUNCH_BLOCKING=1
ERROR 07-07 03:10:47 [engine.py:165] Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
ERROR 07-07 03:10:47 [engine.py:165]

server side(vllm) error

session1.txt

[ardge-timwu]

vllm-project/vllm#16901
fixed in https://github.com/vllm-project/vllm/releases/tag/v0.9.2 , which is released in July 8th
see highlights

• NVIDIA Blackwell
  SM120: CUTLASS W8A8/FP8 kernels and related tuning, added to Dockerfile
  SM100: block‑scaled‑group GEMM, INT8/FP8 vectorization, deep‑GEMM kernels, activation‑chunking for MoE, and group‑size 64 for Machete
• Qwen 3 Embedding & Reranker ([New Model]: Support Qwen3 Embedding & Reranker  vllm-project/vllm#19260)

[ardge-timwu]

FP8 support for Blackwell in vllm v0.9.2 remains incomplete. See: https://discuss.vllm.ai/t/rtx-pro-6000-qwen3-32b-fp8/1010

[ardge-timwu]

It seems that vllm v0.9.2 can load Qwen/Qwen3-Embedding-0.6B. However, on R2R, it complains

{"object":"error","message":"Model \"Qwen/Qwen3-Embedding-0.6B\" does not support matryoshka representation, changing output dimensions will lead to poor results.","type":"BadRequestError","param":null,"code":400}

we have no idea why this model works well on xinference.

model info comparsion

on vllm

curl http://localhost:18001/v1/models
{"object":"list","data":[{"id":"Qwen/Qwen3-Embedding-0.6B","object":"model","created":1752564945,"owned_by":"vllm","root":"Qwen/Qwen3-Embedding-0.6B","parent":null,"max_model_len":32768,"permission":[{"id":"modelperm-441197926b444c3b9803523366d602ea","object":"model_permission","created":1752564945,"allow_create_engine":false,"allow_sampling":true,"allow_logprobs":true,"allow_search_indices":false,"allow_view":true,"allow_fine_tuning":false,"organization":"*","group":null,"is_blocking":false}]}]}

on xinference, it has {"dimensions":1024}

$ curl http://192.168.5.222:9997/v1/models
{"object":"list","data":[{"id":"qwen3","object":"model","created":0,"owned_by":"xinference","model_type":"LLM","address":"0.0.0.0:46153","accelerators":["0"],"model_name":"qwen3","model_lang":["en","zh"],"model_ability":["chat","reasoning","hybrid","tools"],"model_description":"Qwen3 is the latest generation of large language models in Qwen series, offering a comprehensive suite of dense and mixture-of-experts (MoE) models. Built upon extensive training, Qwen3 delivers groundbreaking advancements in reasoning, instruction-following, agent capabilities, and multilingual support","model_format":"ggufv2","model_size_in_billions":30,"model_family":"qwen3","quantization":"Q5_K_M","multimodal_projector":null,"model_hub":"huggingface","revision":null,"context_length":40960,"replica":1},{"id":"Qwen3-Embedding-0.6B","object":"model","created":0,"owned_by":"xinference","model_type":"embedding","address":"0.0.0.0:44319","accelerators":["0"],"model_name":"Qwen3-Embedding-0.6B","dimensions":1024,"max_tokens":32768,"language":["zh","en"],"model_revision":"744169034862c8eec56628663995004342e4e449","replica":1}]}

• reference: "Qwen/Qwen3-Embedding-0.6B" 不是应该是支持自定义输出维度的吗？ QwenLM/Qwen3-Embedding#88