Is such a tensor shape legal when ne = [x, y, z, 1] but nb = [ 1, x*y, x, x*y ] ? That is the last nb not being x*y*z

[lslusarczyk]

When running llama-bench I've met strange tensor. Please help me to understand if there is a bug in the code which created it or if such a tensors can exist and what is their purpose.

Precisely I see a permuted tensor with float2 type inside (no quatization, complex blocks etc) and the following dimensions:

ne = [96, 128, 32, 1] 
nb = [2, 6144, 192, 6144] (that is nb = [2, 96*32*2, 96*2, 96*32*2]

I would expect the last nb dimension to be 96 * 32 * 128 * 2

I see above tensor in matrix matrix multiplication. The second tensor of multiplication has the same "ne", but now its "nb" values I understand and that look valid to me:

ne = [96, 128, 32, 1]
nb = [4, 12288, 384, 1572864] (that is nb = [4, 96*32*4, 96*4, 96*32*128*4])

Is the first tensor valid or is it a bug in a code that has created it?
The code which created it and tried to multiply is:

#8  ggml_backend_graph_compute_async (backend=0x1e047a20, cgraph=0x7ffe3e6f909c) at llama/ggml/src/ggml-backend.cpp:334
334	    return backend->iface.graph_compute(backend, cgraph);
#9  ggml_backend_sched_compute_splits (sched=0x1e043580) at /home/lslusarczyk/src/llama/ggml/src/ggml-backend.cpp:1399
1399	            enum ggml_status ec = ggml_backend_graph_compute_async(split_backend, &split->graph);
#10 ggml_backend_sched_graph_compute_async (sched=0x1e043580, graph=<optimized out>) at llama/ggml/src/ggml-backend.cpp:1590
1590    return ggml_backend_sched_compute_splits(sched);
#11 llama_context::graph_compute (this=this@entry=0x1e59c9d0, gf=gf@entry=0x1e5b03e0, batched=<optimized out>) at llama/src/llama-context.cpp:1694
1694    auto status = ggml_backend_sched_graph_compute_async(sched.get(), gf);
#12 llama_context::decode (this=0x1e59c9d0, inp_batch=...) at llama/src/llama-context.cpp:1361
1361    const auto compute_status = graph_compute(gf, ubatch.n_tokens > 1);
#13 llama_decode (ctx=<optimized out>, batch=...) at llama/src/llama-context.cpp:2784
2784    const int ret = ctx->decode(batch);
#14 test_prompt (ctx=ctx@entry=0x1e59c9d0, n_prompt=512, n_batch=128, n_threads=<optimized out>) at llama/examples/llama-bench/llama-bench.cpp:1447
1447    llama_decode(ctx, llama_batch_get_one(tokens.data(), n_tokens));
#15 main (argc=<optimized out>, argv=<optimized out>) at llama/examples/llama-bench/llama-bench.cpp:1611
1611    test_prompt(ctx, t.n_prompt, t.n_batch, t.n_threads);

[ggerganov]

Both cases are valid in ggml. We allow the stride of dimension d to be arbitrary when ne[d] == 1 (see #7856)