I could use guidance.

[isaaclepes]

GPT has a very difficult time using free/blank verse for poetry. Often it mistakenly rhymes anyway.

I was hoping to submit this as an eval, but it doesn't look like there's a good way to do it with the eval format. GPT does fine at telling the difference between rhyming and non-rhyming, but when asked to create a poem that doesn't rhyme, it often struggles.

If there is a good format to make evals out of this, I would be happy to produce a bunch of examples. I just don't know how to phrase the eval prompts.

[qrdlgit]

Yeah, it's pretty hilarious.

[qrdlgit]

Here's a prompt that might work in an eval.

Create a 12 line BLANK VERSE about a dog where no more than 5 end words rhyme, that is your task. After you've written the poem, evaluate it to see if it rhymes. First list all the end words, and then list all the words that rhyme. Provide a count of all those words. Finally, if more than 5 words rhyme, write 'code1234' as an indication of your failure , if it does adhere to a blank verse structure, write 'code0987' as an indication of your success.

While I think my prompt is clever, it does unfortunately conflate two things - the ability to rhyme and the ability to self-eval. I don't know if those are necessarily thematically consistent, which is something they are looking for.

They may also not prioritize the business value of writing blank verse and that would be :(

Let us know how it goes.

[qrdlgit]

Hmm, try using "Free verse" instead of "blank verse". Seems to get better results.

[isaaclepes]

Hmm, try using "Free verse" instead of "blank verse". Seems to get better results.

Nope, failed it.

[qrdlgit]

Hmmm, you're right. Sometimes I can get minimal rhyming, but not often.

I've written a modelgraded eval here which you can work from, though I'd appreciate a link back to same URL in the txt of the PR.

If you don't plan on submitting, let me know, and I'll submit it.

[isaaclepes]

Hmmm, you're right. Sometimes I can get minimal rhyming, but not often.

I've written a modelgraded eval here which you can work from, though I'd appreciate a link back to same URL in the txt of the PR.

If you don't plan on submitting, let me know, and I'll submit it.

I got caught up in other projects, so go ahead if you like.

Some notes that may be of use:
GPT can recognize non-rhyming poetry, and it can also reliably quote existing free/open verse poetry, but it fails to consistently avoid rhyming when asked to create such poetry.