Perplexity as benchmark ?

[cmp-nct]

I'm not super happy with perplexity as a general idea of model quality assessment.
I just added a perplexity calculator into the repo, so we have one. It will be useful for tests but in the long run I'd prefer something that less strictly adheres to following a text corpus.
Not sure which alternatives are out there.