Working with Groq?

[ubervincent]

Is there a way for it to work with Groq?

I don't know if they can already but it's not immediately obvious that they work with each other.

[khankanz]

I'm trying to figure the same thing. Here's where I started. Since Groq appears to follow a format similar to openai we can take advantage of the openai implementation.

import os
from openai import AsyncOpenAI
from outlines import models
from outlines.models.openai import OpenAIConfig

client = AsyncOpenAI(
    api_key=os.environ.get("GROQ_API_KEY"),
    base_url="https://api.groq.com/openai/v1"
)
config = OpenAIConfig("llama3-70b-8192")
model = models.openai(client, config)

model("Name an expert in quantum gravity.", stop_at=["\n", "."])

I tried to run a generate.choice... task but received an error that a tokenizer needs to associated as well. Will need to dig into library's source code to figure how exactly this flow works.

Reference: https://outlines-dev.github.io/outlines/reference/#structured-generation