A preference question

[aslawliet]

Which one do you like more SPIN or Self-Rewarding LLM, which one do you think have the potential to surpass human experts?

[Jayant1234]

Self-rewarding! SPIN is a very neat hack but it's likely a dead end.

[lucidrains]

that's interesting, why do you think SPIN will hit a dead end? (or do you just mean the surpass human experts part)?

[Jayant1234]

SPIN assumes that the provided labels represent the ideal distribution, which is a strong and faulty assumption, limiting almost all SFT approaches. The second reason is that it's completely equivalent to DPO where provided labels are preferred over LLM output, iteratively replacing reference with previous iteration LLM. So, it's neither novel and also relies on faulty grounds. The way they came up with the approach was ingenious, but they stumbled upon a relative dead-end. Self-rewarding introduces a self-judgment angle that shows promise, and improving self-judgment is likely going to be the future of iterative fine-tuning. However, I might change my mind about SPIN. (I mean for maximizing performance with iterative finetuning without Catastrophic forgetting, so indirectly also applies to surpassing human experts too)