Mentored speculative decoding #3733

KerfuffleV2 started this conversation in Ideas

KerfuffleV2
Oct 23, 2023
Collaborator

This looks interesting: https://vivien000.github.io/blog/journal/a-provably-optimal-lossy-variant-of-speculative-decoding.html

And not really super difficult to implement on top of what already exists: https://github.com/vivien000/mentored_decoding/blob/ace0902c48e9d6aa6c7823a71d6831c3d106ff19/notebook.ipynb

Replies: 0 comments

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment