Llama.cpp support in ModelFusion library (JS/TS)

[lgrammel]

ModelFusion provides a JS/TS client for Llama.cpp.

It supports:

• text generation
• text streaming
• structure generation with automatic gbnf creation from json schema
• same for structure streaming
• text embeddings
• tokenization
• prompt templates for models

Here's an example for structure generation:

const structure = await generateStructure(
  llamacpp
    .CompletionTextGenerator({
      // run openhermes-2.5-mistral-7b.Q4_K_M.gguf in llama.cpp
      promptTemplate: llamacpp.prompt.ChatML,
      maxGenerationTokens: 1024,
      temperature: 0,
    })
    // automatically restrict the output to your schema using GBNF:
    .asStructureGenerationModel(jsonStructurePrompt.text()),

  zodSchema(
    z.object({
      characters: z.array(
        z.object({
          name: z.string(),
          class: z
            .string()
            .describe("Character class, e.g. warrior, mage, or thief."),
          description: z.string(),
        })
      ),
    })
  ),

  "Generate 3 character descriptions for a fantasy role playing game. "
);

[lgrammel]

Starter for Next.js chatbot: https://github.com/lgrammel/modelfusion-llamacpp-nextjs-starter