Proposal to Add Penalized Brier Score (PBS) and Penalized Log Loss (PLL) Methods

[Ostovane]

Hello @jxzhangjhu ,

First, thank you for maintaining the excellent scoringrules package! I want to propose integrating the Penalized Brier Score (PBS) and Penalized Log Loss (PLL) methods from Ahmadian et al.'s work, which directly address limitations of traditional probabilistic scoring rules in classification tasks.

Key Advantages

1. Fixes Critical Flaws

   • Resolves Brier Score/Log Loss failure to guarantee proper scoring in finite-sample regimes (Proofs)
   • Introduces penalty term: `penalty = Ensuring incorrect predictions always score worse than correct ones '

2. Theoretical Guarantees

   • Strictly proper by construction
   • Consistent model selection even with imperfect models

3. Practical Benefits

   • Better guidance for early stopping and checkpointing
   • Robust to class imbalance
   • Computationally equivalent to traditional scores

Implementation Details

The reference Python implementation provides:

• Vectorized PBS/PLL calculations
• Unit tests verifying proper conditions
• Benchmarks against standard metrics

Would you consider adding these to scoringrules?

Paper: "Superior scoring rules..." (Ahmadian et al., Int. J. Approx. Reason. 2024)

Thank you for considering this contribution to improve probabilistic scoring!