Will "Prequant BitsAndBytes models with TP" be supported?

[congwupiece]

when seve a BitsAndBytes model with --tensor-parallel-size N
it come that
ValueError: Prequant BitsAndBytes models with TP is not supported.Please try with PP.

[aidando73]

Getting same error.

Btw, what is PP?

[jeejeelee]

@chenqianfzh What difficulties are there with this support?
@aidando73 PP stands for pipeline parallelism

[chenqianfzh]

@chenqianfzh What difficulties are there with this support? @aidando73 PP stands for pipeline parallelism

Michael and I had discussed about the lack of support of TP to prequant bnb models, in PR #8434. We agreed PP is a reasonable choice than TP.

[apohllo]

Could you please change the message? It's obvious to me what is tensor parallelizm, but seeing the message I had no idea was is PP vs. TP. This is such a minor change, but will save a lot of people I belive.

[jeejeelee]

Could you please change the message? It's obvious to me what is tensor parallelizm, but seeing the message I had no idea was is PP vs. TP. This is such a minor change, but will save a lot of people I belive.

Make sense, I am handling this now

[jeejeelee]

@apohllo See: #12320