How does the multi-alpha quantile regression training work?

[skchao74]

Hi! I am using the xgboost package (in python if this matters) to train quantile regression for multiple alphas. One can set a list of alphas in the train xgboost train function when using pin ball loss, and this would return a single model that do prediction for the quantiles at this list of alphas. I wonder how it is different from me looping over the list of alphas, and train a single xgboost model for every alpha?

Could you clarify that for the single model approach (set a list of alphas in the train function and train one model), are the prediction heads for different alphas concatenated to a shared tree structure? If so, how large is the shared tree and how can I tune it? Is there any document or paper you may be able to share for this approach?

Thank you!

[trivialfis]

I wonder how it is different from me looping over the list of alphas, and train a single xgboost model for every alpha

Other than the initialization step, no difference.

are the prediction heads for different alphas concatenated to a shared tree structure

No, but we are working on it. See #9043

[skchao74]

Thanks so much for the feedback! So basically you're saying for training with a list of alphas, there are still multiple models being trained. We did see a slight performance drop for the single model approach than the "training with one alpha at a time" approach, so based on your feedback this is solely due to the different initialization?

[trivialfis]

We did see a slight performance drop for the single model approach than the "training with one alpha at a time" approach

Could you please elaborate on that? Are you suggesting that xgb.train({"quantile_alphas": [0.95, 0.5, 0.05]}) has lower accuracy than:

for q in (0.95, 0.5, 0.05):
    xgb.train({"quantile_alphas": [q]})

? If so, could you please try #11286 ?

There's no difference for training aside from the initialization step. However, the metric calculation (pin ball) had a bug related to multi-quantiles and it's fixed in 3.0. After the fix, it should use the average of the loss across quantile targets, which may contribute to the observed difference.

So basically you're saying for training with a list of alphas, there are still multiple models being trained.

Yes. Similar to multi-class/label/target models.

In future releases, the difference in the initialization step will be removed (so, will be completely the same). #9043 is a different path that uses a single tree for multiple targets, but it will take some time.

[skchao74]

Thanks, I assume the bug fixed in 3.0 was related to this line https://github.com/dmlc/xgboost/blob/master/src/objective/quantile_obj.cu#L170, and I assume the average is over all quantiles, like mean(L_alpha) over all alphas in the list, where L_alpha is the loss function at an alpha value?

[trivialfis]

No, it's just the metric: #11279 not related to training.