use collected article headlines and fine-tune personal BERT model

[Shubxam]

• have the article_data table with headlines and inferred sentiment labels.
• review (by some means) sentiment labels to make sure they are right.
• take a good base BERT model and finetune on a local machine.

[Shubxam]

references:

• tutorial: https://blog.jetbrains.com/pycharm/2024/12/how-to-do-sentiment-analysis-with-large-language-models/#fine-tune-your-own-llm-for-sentiment-analysis
• https://huggingface.co/blog/bert-101
• https://paperswithcode.com/method/bert

[Shubxam]

You can use Encoder only model (which is the architecture of BERT models) and train it for downstream task (in our case, sentiment analysis).

• there is a newer BERT architecture (Called ModernBERT) which is even better then base BERT models.
  • HN discussion
• Could use a library like infinity and VLLM which now supports ModernBERT for high throughput inference.
• ModernBERT usually shows better performance where long context is needed (such as Long News Articles Text). For only classifying headlines, its performance should be similar to BERT.
• HF Docs

Please note that ModernBERT, as any other base model, can only do masked word prediction out-of-the-box. To be able to perform other tasks, the base model should be fine-tuned as done in these boilerplates.

[Shubxam]

Datasets:

• https://www.researchgate.net/publication/251231364_FinancialPhraseBank-v10
• https://huggingface.co/datasets/takala/financial_phrasebank
• https://huggingface.co/datasets/kdave/Indian_Financial_News