Qwen3-embedding got list of null

[franklucky001]

System Info

Description

run with docker images ghcr.io/huggingface/text-embeddings-inference:turing-1.7.2

Information

• Docker
• The CLI directly

Tasks

• An officially supported command
• My own modifications

Reproduction

run testcase with curl curl 127.0.0.1:8080/embed -X POST -d '{"inputs":"What is Deep Learning?"}' -H 'Content-Type: application/json'

result

[[null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null]]

Expected behavior

embed inference all is ok

[vrdn-23]

Duplicate of #53
@Narsil @alvarobartt Might be worth pinning this issue or updating the README with a link to the issue, to ensure that people don't keep raising the same bug

[nikita-vanyasin]

Hello
I'm seeing same issue with Qwen3-Embedding-4B model running with :cuda-latest tag.
I've tried to disable flash attention, but not sure if it actually got disabled.

docker run --rm -d --gpus all -p 8080:80 -v hf_cache:/data --sysctl net.ipv6.conf.all.disable_ipv6=1 --shm-size="16gb" -e USE_FLASH_ATTENTION=False  --name=tei-1 ghcr.io/huggingface/text-embeddings-inference:cuda-latest --model-id Qwen/Qwen3-Embedding-4B --dtype float16

I'm running this ^ on AWS g4dn (nvidia-enabled) instance. Maybe I should have used different tag?

UPD: it is working with qwen3 0.6B model. Wondering if 16GiB RAM is not enough for 4B model and it just silently fails?

UPD2: tried running on 32GiB RAM (g4dn.2xlarge): same result.

UPD3: Tried both cuda-latest (cuda-sha-6b79f20) and cuda-1.7.3

curl http://localhost:8080/embed     -X POST     -d '{"inputs": ["lorem ipsum dolome hi"]}'     -H "Content-Type: application/json"
[[null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null ......

INFO embed{total_time="63.477809ms" tokenization_time="469.117µs" queue_time="404.755µs" inference_time="62.527476ms"}: text_embeddings_router::http::server: router/src/http/server.rs:730: Success

[nikita-vanyasin]

Hello I'm seeing same issue with Qwen3-Embedding-4B model running with :cuda-latest tag. I've tried to disable flash attention, but not sure if it actually got disabled.

docker run --rm -d --gpus all -p 8080:80 -v hf_cache:/data --sysctl net.ipv6.conf.all.disable_ipv6=1 --shm-size="16gb" -e USE_FLASH_ATTENTION=False  --name=tei-1 ghcr.io/huggingface/text-embeddings-inference:cuda-latest --model-id Qwen/Qwen3-Embedding-4B --dtype float16

I'm running this ^ on AWS g4dn (nvidia-enabled) instance. Maybe I should have used different tag?

UPD: it is working with qwen3 0.6B model. Wondering if 16GiB RAM is not enough for 4B model and it just silently fails?

UPD2: tried running on 32GiB RAM (g4dn.2xlarge): same result.

UPD3: Tried both cuda-latest (cuda-sha-6b79f20) and cuda-1.7.3

curl http://localhost:8080/embed     -X POST     -d '{"inputs": ["lorem ipsum dolome hi"]}'     -H "Content-Type: application/json"
[[null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null ......

INFO embed{total_time="63.477809ms" tokenization_time="469.117µs" queue_time="404.755µs" inference_time="62.527476ms"}: text_embeddings_router::http::server: router/src/http/server.rs:730: Success

Update: it worked with turing-latest docker image. Wondering if cuda tags are also affected by precision issue but '-e USE_FLASH_ATTENTION=False' is not handled?

[nikita-vanyasin]

Update: after running a model about ~1 hour it started to return nulls again 😱
Restarting docker container helped to get it running properly again.

[nikita-vanyasin]

Update: after running a model about ~1 hour it started to return nulls again 😱 Restarting docker container helped to get it running properly again.

Actually running for 1 hour is not required. Way to reproduce:

# start
docker run -d --rm  --gpus all -p 8080:80 -v hf_cache:/data --sysctl net.ipv6.conf.all.disable_ipv6=1 --shm-size="16gb"  --name=tei-search  ghcr.io/huggingface/text-embeddings-inference:turing-latest --model-id Qwen/Qwen3-Embedding-4B  --dtype float16
# wait for ready

# normal payload
curl http://localhost:8080/embed     -X POST     -d '{"inputs": ["lorem ipsum ds dolome hi"]}'     -H "Content-Type: application/json"
# this works fine, can be executed many times


# send some edge-case nonsense payload:
curl http://localhost:8080/embed     -X POST     -d '{"inputs": ["asdfg"]}'     -H "Content-Type: application/json"
[[null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,...

# all next payloads will return nulls, even same "lorem ipsum ds dolome hi":
curl http://localhost:8080/embed     -X POST     -d '{"inputs": ["lorem ipsum ds dolome hi"]}'     -H "Content-Type: application/json"
[[null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,...

[dstampher]

Just wanted to +1, as I have just encountered this same problem. Any work around, so that I can atleast test the Qwen3 06b embeddings locally?

[alvarobartt]

Hey @nikita-vanyasin thanks for reporting, as @vrdn-23 mentioned we've received similar reports in the past at it might be related to Flash Attention, in any case, since you mentioned that the issue seems to persist even when disabling it, we'll look into it closely (didn't experience those null ourselves), but it can be related to #663 (I'll do a patch release on Monday and ping you back since that may resolve the issue of null appearing when running on FP16 due to dtype downcasting issues)

cc @dstampher also for visibility!

[alvarobartt]

Hi @nikita-vanyasin, @dstampher and @vrdn-23, we recently released a patch for Text Embeddings Inference (TEI) as v1.7.4, so please make sure you re-run with it to see if the issue disappears (it should as per our testing, but we might have missed something).

https://github.com/huggingface/text-embeddings-inference/releases/tag/v1.7.4

[nikita-vanyasin]

Hi @alvarobartt
I've tried the above reproducer with 'ghcr.io/huggingface/text-embeddings-inference:turing-1.7.4' docker image. Same results

[dstampher]

@alvarobartt Same as @nikita-vanyasin above, also getting same result. I also tried with the USE_FLASH_ATTENTION flag set to true and false Same result with both values.

[alvarobartt]

Fair thanks for the report @nikita-vanyasin and @dstampher, as @vrdn-23 flagged above it seems related to #53, but since it seems that even when disabling Flash Attention it will still lead to the same null outputs I'll try to take an in-depth look and reproduce with what @nikita-vanyasin shared earlier in the thread!

[alvarobartt]

Also @dstampher alternatively since you asked:

Any work around, so that I can atleast test the Qwen3 06b embeddings locally?

I'd suggest in the meantime you can leverage the ONNX runtime, which works just fine on CPU, as it follows:

docker run -p 8080:80 ghcr.io/huggingface/text-embeddings-inference:1.7 --model-id Qwen/Qwen3-Embedding-0.6B --revision refs/pr/27 --max-batch-tokens 512

Or via cargo from main as:

cargo run --release --features ort,http --no-default-features -- --model-id Qwen/Qwen3-Embedding-0.6B --revision refs/pr/27 --max-batch-tokens 512

And you'll be able to leverage CPU inference running with the ONNX runtime just fine in the meantime! 🤗