📋 Formats

[hejny]

For each prompt template you can define the exact format you need.

It's similar concept as JSON mode from OpenAI (its kinda super-JSON mode) but with 3 advantages:

• You can expect more formats then JSON like XML, YAML, Markdown, HTML, ICal, Vcard... they are simply addable as plugins.
• You can define schemas. Never again will it happen that you expect a list of strings with emails and LLM "creatively" creates its own structure.
• You can still use count expectations which will intelligently count only the data part, not the structure part.

Simple syntax

- EXPECT JSON

- EXPECT XML

Format + count expectations

You can combine format and count - it will only take and count the data part, not the structure part:

- EXPECT JSON
- EXPECT MIN 2 WORDS
- EXPECT MAX 3 WORDS

This will NOT pass:

{
    "firstName": "John Smith",
    "lastName": "Peter Green"
}

This will pass:

{
    "firstName": "John",
    "lastName": "Smith"
}

Format + schema

- EXPECT JSON https://promptbook.studio/sample/schemas/contacts.schema.json

Partial check

If there are any tokens you have that cannot result in a valid format (even healed). The partial result is discarded and one token is returned.

See more in discussion about 🔙 Expectation-aware generation.

Healing

When there is obvious how to make format valid from invalid, it is done without rerunning LLM.

This invalid JSON:

{
    "firstName": "John",
    "lastName": "Smith",
}

Will be healed:

{
    "firstName": "John",
    "lastName": "Smith"
}

FormatDefinition

Each format is defined by a format definition.

/**
 * A format definition is a set of functions that define how to validate, heal and convert response from LLM
 */
export type FormatDefinition<TValue extends TPartialValue, TPartialValue extends string, TSchema extends object> = {
    /**
     * The name of the format used in .ptbk.md files
     *
     * @sample "JSON"
     */
    readonly name: string_name;

    /**
     * Aliases for the name
     */
    readonly aliases?: Array<string_name>;

    /**
     * The mime type of the format (if any)
     *
     * @sample "application/json"
     */
    readonly mimeType?: string_mime_type;

    /**
     * Check if a value is fully valid
     *
     * @param value The value to check, for example "{\"foo\": true}"
     * @param schema Optional schema to do extra validation
     */
    isValid(value: string, schema?: TSchema): value is TValue;

    /**
     * Check if a first part of a value is valid
     *
     * @see https://github.com/webgptorg/promptbook/discussions/37
     *
     * @param partialValue Partial value to check, for example "{\"foo\": t"
     * @param schema Optional schema to do extra validation
     */
    canBeValid(partialValue: string, schema?: TSchema): partialValue is TPartialValue;

    /**
     * Heal a value to make it valid if possible
     *
     * Note: This make sense in context of LLMs that often returns slightly invalid values
     * @see https://github.com/webgptorg/promptbook/discussions/31
     *
     * @param value The value to heal, for example "{foo: true}"
     * @param scheme
     * @throws {Error} If the value cannot be healed
     */
    heal(value: string, scheme?: TSchema): TValue;

    /**
     * Parses just the values and removes structural information
     *
     * Note: This is useful when you want to combine format expectations with counting words, characters,...
     *
     * @param value The value to check, for example "{\"name\": "John Smith"}"
     * @param schema Optional schema
     * @example "{\"name\": "John Smith"}" -> ["John Smith"]
     */
    extractValues(value: string, schema?: TSchema): Array<string>;
}

For all code samples look here.

[hejny]

See OpenAI`s Structured Outputs vs JSON mode

[hejny]

TODO: Update FormatDefinition section

[BleedingDev]

What formats are currently supported?

YAML? XML?