Caching

[Michael-F-Ellis]

I feel like I'm being a pest, but that's not my intention. I just want to know whether the new OAI-like endpoint v1/chat/completions supports or will support caching prior text. I can't get it to work. I've submitted two issues, #4329 and #4287, that have so far not garnered a single reply. I've also tried the unofficial support forum and stackoverflow. I've read the docs and attempted to figure it out from the server.cpp code.

I understand that llama.cpp is a (marvelous) volunteer effort and no one has any obligation to provide support, but I'd greatly appreciate at least a quick "No, it's not supported" or "Yes, but you're doing it wrong" so I won't keep spinning my wheels.

Thanks,
Mike

[KerfuffleV2]

It looks like it always just turns prompt caching on. https://github.com/ggerganov/llama.cpp/blob/5f6e0c0dff1e7a89331e6b25eca9a9fd71324069/examples/server/api_like_OAI.py#L80-L84

Whether it actually works or not, I don't know. You can possibly make your own modifications there to the stuff it's sending to the server example.

[Michael-F-Ellis]

Thanks, I was actually trying to use the new endpoint that obviates the need for api_like_OAI.py

[Michael-F-Ellis]

@ggerganov has fixed it in branch gg/server-oai-cache-prompt. Works very well now. See #4329. Makes it feasible to work on large-ish docs and chats interactively with 7B models running on my Mac mini.