diff --git a/docs/api.md b/docs/api.md index 61b9f1e..b1a89b1 100644 --- a/docs/api.md +++ b/docs/api.md @@ -6,936 +6,1182 @@ ## Classes -### API +### LiteralClient -#### Methods - -##### sendSteps() +#### Constructors -> **sendSteps**(`steps`): `Promise`\<`any`\> +##### new LiteralClient(options) -Sends a collection of steps to the GraphQL endpoint. +> **new LiteralClient**(`options`): [`LiteralClient`](api.md#literalclient) -This method constructs a GraphQL query using the provided steps, then executes the query. +Initialize a new Literal AI Client. ###### Parameters -▪ **steps**: `Step`[] +▪ **options**: `object`= `{}` + +▪ **options.apiKey?**: `string` + +The API key to use for the Literal AI API. Defaults to the LITERAL_API_KEY environment variable. + +▪ **options.apiUrl?**: `string` + +The URL of the Literal AI API. Defaults to the LITERAL_API_URL env var, or https://cloud.getliteral.ai. -An array of Step objects to be sent. +▪ **options.environment?**: [`Environment`](api.md#environment) + +The environment to use for the Literal AI API. + +▪ **options.disabled?**: `boolean` + +If set to true, no call will be made to the Literal AI API. ###### Returns -`Promise`\<`any`\> +[`LiteralClient`](api.md#literalclient) -The response from the GraphQL call. +A new LiteralClient instance. -##### getStep() +#### Properties -> **getStep**(`id`): `Promise`\<`Maybe`\<`Step`\>\> +##### api -Retrieves a step by its ID. +> **api**: `API` -This method constructs a GraphQL query to fetch a step by its unique identifier. -It then executes the query and returns the step if found. +##### openai -###### Parameters +> **openai**: (`openai`) => `object` -▪ **id**: `string` +###### Parameters -The unique identifier of the step to retrieve. +▪ **openai**: `OpenAI` ###### Returns -`Promise`\<`Maybe`\<`Step`\>\> +`object` + +> ###### assistant +> +> > **assistant**: `object` +> +> ###### assistant.syncer +> +> > **assistant.syncer**: `OpenAIAssistantSyncer` +> -A `Promise` that resolves to the step if found, or `null` if not found. +##### instrumentation -##### deleteStep() +> **instrumentation**: `object` -> **deleteStep**(`id`): `Promise`\<`string`\> +###### Type declaration -Deletes a Step from the platform by its unique identifier. +###### openai -This method constructs a GraphQL mutation to delete a step by its unique identifier. -It then executes the mutation and returns the ID of the deleted step. +> **openai**: (`options`?) => `object` + +Instrument the OpenAI client to log all generations to the Literal AI API. +Compatible with OpenAI's `chat.completions.create`, `completions.create` and `images.generate` functions. +If you want to add tags or metadata at the call level, you should use the augmented client returned by this function. +Its functions will be augmented with `literalaiTags` and `literalaiMetadata` options. ###### Parameters -▪ **id**: `string` +▪ **options?**: [`OpenAIGlobalOptions`](api.md#openaiglobaloptions) + +###### Returns + +`object` + +> ###### chat +> +> > **chat**: `object` +> +> ###### chat.completions +> +> > **chat.completions**: `object` +> +> ###### chat.completions.create +> +> > **chat.completions.create**: (`this`, `body`, `callOptions`?) => `Promise`\<`Stream`\<`ChatCompletionChunk`\> \| `ChatCompletion`\> = `wrappedChatCompletionsCreate` +> +> ###### Parameters +> +> ▪ **this**: `any` +> +> ▪ **body**: `any` +> +> ▪ **callOptions?**: `RequestOptions` & `OpenAICallOptions` +> +> ###### Returns +> +> `Promise`\<`Stream`\<`ChatCompletionChunk`\> \| `ChatCompletion`\> +> +> ###### completions +> +> > **completions**: `object` +> +> ###### completions.create +> +> > **completions.create**: (`this`, `body`, `callOptions`?) => `Promise`\<`Completion` \| `Stream`\<`Completion`\>\> = `wrappedCompletionsCreate` +> +> ###### Parameters +> +> ▪ **this**: `any` +> +> ▪ **body**: `any` +> +> ▪ **callOptions?**: `RequestOptions` & `OpenAICallOptions` +> +> ###### Returns +> +> `Promise`\<`Completion` \| `Stream`\<`Completion`\>\> +> +> ###### images +> +> > **images**: `object` +> +> ###### images.generate +> +> > **images.generate**: (`this`, `body`, `callOptions`?) => `Promise`\<`ImagesResponse`\> = `wrappedImagesGenerate` +> +> ###### Parameters +> +> ▪ **this**: `any` +> +> ▪ **body**: `any` +> +> ▪ **callOptions?**: `RequestOptions` & `OpenAICallOptions` +> +> ###### Returns +> +> `Promise`\<`ImagesResponse`\> +> + +###### langchain + +> **langchain**: `object` + +###### langchain.literalCallback + +> **langchain.literalCallback**: (`threadId`?) => `LiteralCallbackHandler` + +###### Parameters -The unique identifier of the step to delete. +▪ **threadId?**: `string` ###### Returns -`Promise`\<`string`\> +`LiteralCallbackHandler` -A `Promise` that resolves to the ID of the deleted step. +###### vercel -##### uploadFile() +> **vercel**: `object` -> **uploadFile**(`params`): `Promise`\<`object`\> +###### vercel.instrument -Uploads a file to a specified thread. This method supports uploading either through direct content or via a file path. -It first signs the upload through a pre-configured endpoint and then proceeds to upload the file using the signed URL. +> **vercel.instrument**: `InstrumentationVercelMethod` -###### Parameters +Instrument a Vercel SDK function to log all invocations to the Literal AI API. +It will be augmented with a `literalAiParent` option that allows you to specify a parent step or thread. + +###### Param -▪ **params**: `object` +The function to instrument. This can be Vercel SDK's `generateText` or `streamText` functions. -The parameters for uploading a file, including: +###### llamaIndex -▪ **params.content?**: `any` +> **llamaIndex**: `object` -The content of the file to upload. Optional if `path` is provided. +###### llamaIndex.instrument -▪ **params.path?**: `Maybe`\<`string`\> +> **llamaIndex.instrument**: () => `void` -The path to the file to upload. Optional if `content` is provided. +Instrument the LlamaIndex client to log all generations to the Literal AI API. -▪ **params.id?**: `Maybe`\<`string`\> +###### Returns -The unique identifier for the file. If not provided, a UUID will be generated. +`void` -▪ **params.threadId**: `string` +###### llamaIndex.withThread -The unique identifier of the thread to which the file is being uploaded. +> **llamaIndex.withThread**: \<`R`\>(`thread`, `callback`) => `R` -▪ **params.mime?**: `Maybe`\<`string`\> +###### Type parameters -The MIME type of the file. Defaults to 'application/octet-stream' if not provided. +▪ **R** + +###### Parameters + +▪ **thread**: `Thread` + +▪ **callback**: () => `R` ###### Returns -`Promise`\<`object`\> +`R` -An object containing the `objectKey` of the uploaded file and the signed `url`, or `null` values if the upload fails. +##### store -###### Throws +> **store**: `AsyncLocalStorage`\<`StoredContext`\> = `storage` -Throws an error if neither `content` nor `path` is provided, or if the server response is invalid. +#### Methods -##### getGenerations() +##### thread() -> **getGenerations**(`variables`): `Promise`\<`PaginatedResponse`\<`PersistedGeneration`\>\> +> **thread**(`data`?): `Thread` -Retrieves a paginated list of Generations based on the provided filters and sorting order. +Creates a new thread without sending it to the Literal AI API. ###### Parameters -▪ **variables**: `object` +▪ **data?**: `ThreadConstructor` -The variables to filter and sort the Generations. It includes: -- `first`: The number of items to return. -- `after`: The cursor to fetch items after. -- `before`: The cursor to fetch items before. -- `filters`: The filters applied to the Generations. -- `orderBy`: The order in which the Generations are sorted. +Optional initial data for the thread. -▪ **variables.first?**: `Maybe`\<`number`\> +###### Returns + +`Thread` -▪ **variables.after?**: `Maybe`\<`string`\> +A new thread instance. -▪ **variables.before?**: `Maybe`\<`string`\> +##### step() -▪ **variables.filters?**: `GenerationsFilter`[] +> **step**(`data`): `Step` -▪ **variables.orderBy?**: `GenerationsOrderBy` +Creates a new step without sending it to the Literal AI API. + +###### Parameters + +▪ **data**: `StepConstructor` + +Optional initial data for the step. ###### Returns -`Promise`\<`PaginatedResponse`\<`PersistedGeneration`\>\> +`Step` -A `Promise` that resolves to a `PaginatedResponse` object containing the filtered and sorted Generations. +A new step instance. -##### createGeneration() +##### run() -> **createGeneration**(`generation`): `Promise`\<`PersistedGeneration`\> +> **run**(`data`): `Step` -Creates a new generation entity and sends it to the platform. +Creates a new step with the type set to 'run'. ###### Parameters -▪ **generation**: `Generation` +▪ **data**: `Omit`\<`StepConstructor`, `"type"`\> -The `Generation` object to be created and sent to the platform. +Optional initial data for the step. ###### Returns -`Promise`\<`PersistedGeneration`\> +`Step` -A Promise resolving to the newly created `Generation` object. +A new step instance. -##### upsertThread() +##### experimentRun() -###### upsertThread(options) +> **experimentRun**(`data`?): `ExperimentRun` -> **upsertThread**(`options`): `Promise`\<`CleanThreadFields`\> - -Upserts a Thread with new information. +Creates a new Experiment Run. ###### Parameters -▪ **options**: `object` +▪ **data?**: `Omit`\<`StepConstructor`, `"name"` \| `"type"`\> -The parameters to upsert a thread. +Optional initial data for the step. -▪ **options.threadId**: `string` +###### Returns -The unique identifier of the thread. (Required) +`ExperimentRun` -▪ **options.name?**: `Maybe`\<`string`\> +A new step instance. -The name of the thread. (Optional) +##### \_currentThread() -▪ **options.metadata?**: `Maybe`\<`Record`\<`string`, `any`\>\> +> **\_currentThread**(): `null` \| `Thread` -Additional metadata for the thread as a key-value pair object. (Optional) +Returns the current thread from the context or null if none. -▪ **options.participantId?**: `Maybe`\<`string`\> +###### Returns -The unique identifier of the participant. (Optional) +`null` \| `Thread` -▪ **options.environment?**: `Maybe`\<`string`\> +The current thread, if any. -The environment where the thread is being upserted. (Optional) +##### \_currentStep() -▪ **options.tags?**: `Maybe`\<`string`[]\> +> **\_currentStep**(): `null` \| `Step` -An array of tags associated with the thread. (Optional) +Returns the current step from the context or null if none. ###### Returns -`Promise`\<`CleanThreadFields`\> +`null` \| `Step` -The upserted thread object. +The current step, if any. -###### upsertThread(threadId, name, metadata, participantId, environment, tags) +##### \_currentExperimentRunId() -> **upsertThread**(`threadId`, `name`?, `metadata`?, `participantId`?, `environment`?, `tags`?): `Promise`\<`CleanThreadFields`\> +> **\_currentExperimentRunId**(): `null` \| `string` -Upserts a Thread with new information. +Returns the current experiment from the context or null if none. -###### Parameters +###### Returns -▪ **threadId**: `string` +`null` \| `string` -The unique identifier of the thread. (Required) +The current experiment, if any. -▪ **name?**: `Maybe`\<`string`\> +##### getCurrentThread() -The name of the thread. (Optional) +> **getCurrentThread**(): `Thread` -▪ **metadata?**: `Maybe`\<`Record`\<`string`, `any`\>\> +Gets the current thread from the context. +WARNING : this will throw if run outside of a thread context. -Additional metadata for the thread as a key-value pair object. (Optional) +###### Returns + +`Thread` -▪ **participantId?**: `Maybe`\<`string`\> +The current thread, if any. -The unique identifier of the participant. (Optional) +##### getCurrentStep() -▪ **environment?**: `Maybe`\<`string`\> +> **getCurrentStep**(): `Step` -The environment where the thread is being upserted. (Optional) +Gets the current step from the context. +WARNING : this will throw if run outside of a step context. -▪ **tags?**: `Maybe`\<`string`[]\> +###### Returns -An array of tags associated with the thread. (Optional) +`Step` + +The current step, if any. + +##### getCurrentExperimentRunId() + +> **getCurrentExperimentRunId**(): `string` + +Gets the current experiment run ID from the context. +WARNING : this will throw if run outside of an experiment context. ###### Returns -`Promise`\<`CleanThreadFields`\> +`string` -The upserted thread object. +The current experiment, if any. -###### Deprecated +*** -Use one single object attribute instead of multiple parameters. +### BaseGeneration -##### getThreads() +Represents a utility class with serialization capabilities. -> **getThreads**(`variables`): `Promise`\<`PaginatedResponse`\<`CleanThreadFields`\>\> +#### Extends -Retrieves a paginated list of threads (conversations) based on the provided criteria. +- [`Utils`](api.md#utils) -###### Parameters +#### Constructors -▪ **variables**: `object` +##### new BaseGeneration() -The parameters to filter and paginate the threads. +> **new BaseGeneration**(): [`BaseGeneration`](api.md#basegeneration) -▪ **variables.first?**: `Maybe`\<`number`\> +###### Returns -The number of threads to retrieve after the cursor. (Optional) +[`BaseGeneration`](api.md#basegeneration) -▪ **variables.after?**: `Maybe`\<`string`\> +###### Inherited from -The cursor to start retrieving threads after. (Optional) +[`Utils`](api.md#utils).[`constructor`](api.md#constructors-4) -▪ **variables.before?**: `Maybe`\<`string`\> +#### Properties -The cursor to start retrieving threads before. (Optional) +##### promptId -▪ **variables.filters?**: `ThreadsFilter`[] +> **promptId**?: `string` -The filters to apply on the threads retrieval. (Optional) +##### provider -▪ **variables.orderBy?**: `ThreadsOrderBy` +> **provider**?: [`Maybe`](api.md#maybet)\<`string`\> -The order in which to retrieve the threads. (Optional) +##### model -▪ **variables.stepTypesToKeep?**: `StepType`[] +> **model**?: [`Maybe`](api.md#maybet)\<`string`\> -###### Returns +##### id -`Promise`\<`PaginatedResponse`\<`CleanThreadFields`\>\> +> **id**?: [`Maybe`](api.md#maybet)\<`string`\> -A promise that resolves to a paginated response of threads. +##### tags -##### getThread() +> **tags**?: [`Maybe`](api.md#maybet)\<`string`[]\> -> **getThread**(`id`): `Promise`\<`any`\> +##### error -Retrieves information from a single Thread. +> **error**?: [`Maybe`](api.md#maybet)\<`string`\> -###### Parameters +##### variables -▪ **id**: `string` +> **variables**?: [`Maybe`](api.md#maybet)\<`Record`\<`string`, `any`\>\> -The unique identifier of the thread. This parameter is required. +##### settings -###### Returns +> **settings**?: [`Maybe`](api.md#maybet)\<[`ILLMSettings`](api.md#illmsettings)\> -`Promise`\<`any`\> +##### tools -The detailed information of the specified thread. +> **tools**?: [`Maybe`](api.md#maybet)\<[`ITool`](api.md#itool)[]\> -##### deleteThread() +##### tokenCount -> **deleteThread**(`id`): `Promise`\<`any`\> +> **tokenCount**?: [`Maybe`](api.md#maybet)\<`number`\> -Deletes a single Thread by its unique identifier. +##### inputTokenCount -###### Parameters +> **inputTokenCount**?: [`Maybe`](api.md#maybet)\<`number`\> + +##### outputTokenCount + +> **outputTokenCount**?: [`Maybe`](api.md#maybet)\<`number`\> + +##### ttFirstToken + +> **ttFirstToken**?: [`Maybe`](api.md#maybet)\<`number`\> + +##### tokenThroughputInSeconds -▪ **id**: `string` +> **tokenThroughputInSeconds**?: [`Maybe`](api.md#maybet)\<`number`\> -The unique identifier of the thread to be deleted. This parameter is required. +##### duration + +> **duration**?: [`Maybe`](api.md#maybet)\<`number`\> + +#### Methods + +##### serialize() + +> **serialize**(): `any` + +Serializes the properties of the current instance into a dictionary, excluding the 'api' property. +It handles nested objects that also implement a serialize method. ###### Returns -`Promise`\<`any`\> +`any` + +A dictionary representing the serialized properties of the object. + +###### Inherited from + +[`Utils`](api.md#utils).[`serialize`](api.md#serialize-3) + +*** + +### CompletionGeneration + +Represents a utility class with serialization capabilities. + +#### Extends -The ID of the deleted thread. +- [`BaseGeneration`](api.md#basegeneration) -##### getUsers() +#### Constructors -> **getUsers**(`variables`): `Promise`\<`PaginatedResponse`\<`OmitUtils`\<`User`\>\>\> +##### new CompletionGeneration(data) -Retrieves a list of users with optional filters. +> **new CompletionGeneration**(`data`): [`CompletionGeneration`](api.md#completiongeneration) ###### Parameters -▪ **variables**: `object` +▪ **data**: [`OmitUtils`](api.md#omitutilst)\<[`CompletionGeneration`](api.md#completiongeneration)\> -The parameters used to filter and paginate the user list. +###### Returns -▪ **variables.first?**: `Maybe`\<`number`\> +[`CompletionGeneration`](api.md#completiongeneration) -Optional. The number of items to return. +###### Overrides -▪ **variables.after?**: `Maybe`\<`string`\> +[`BaseGeneration`](api.md#basegeneration).[`constructor`](api.md#constructors-1) -Optional. The cursor after which to start fetching data. +#### Properties -▪ **variables.before?**: `Maybe`\<`string`\> +##### promptId -Optional. The cursor before which to start fetching data. +> **promptId**?: `string` -▪ **variables.filters?**: `ParticipantsFilter`[] +###### Inherited from -Optional. Array of filters to apply to the user query. +[`BaseGeneration`](api.md#basegeneration).[`promptId`](api.md#promptid) -###### Returns +##### provider -`Promise`\<`PaginatedResponse`\<`OmitUtils`\<`User`\>\>\> +> **provider**?: [`Maybe`](api.md#maybet)\<`string`\> -A `PaginatedResponse` containing a list of users without utility types. +###### Inherited from -##### createUser() +[`BaseGeneration`](api.md#basegeneration).[`provider`](api.md#provider) -> **createUser**(`identifier`, `metadata`?): `Promise`\<`User`\> +##### model -Creates a new user and sends it to the platform. +> **model**?: [`Maybe`](api.md#maybet)\<`string`\> -###### Parameters +###### Inherited from -▪ **identifier**: `string` +[`BaseGeneration`](api.md#basegeneration).[`model`](api.md#model) -The unique identifier for the user. This parameter is required. +##### id -▪ **metadata?**: `Maybe`\<`Record`\<`string`, `any`\>\> +> **id**?: [`Maybe`](api.md#maybet)\<`string`\> -Optional metadata for the user. This parameter is optional. +###### Inherited from -###### Returns +[`BaseGeneration`](api.md#basegeneration).[`id`](api.md#id) -`Promise`\<`User`\> +##### tags -A promise that resolves with the newly created User object. +> **tags**?: [`Maybe`](api.md#maybet)\<`string`[]\> -##### updateUser() +###### Inherited from -> **updateUser**(`id`, `identifier`?, `metadata`?): `Promise`\<`User`\> +[`BaseGeneration`](api.md#basegeneration).[`tags`](api.md#tags) -Updates an existing user's details in the platform. +##### error -###### Parameters +> **error**?: [`Maybe`](api.md#maybet)\<`string`\> -▪ **id**: `string` +###### Inherited from -The unique identifier of the user to update. This parameter is required. +[`BaseGeneration`](api.md#basegeneration).[`error`](api.md#error) -▪ **identifier?**: `string` +##### variables -A new identifier for the user. This parameter is optional. +> **variables**?: [`Maybe`](api.md#maybet)\<`Record`\<`string`, `any`\>\> -▪ **metadata?**: `Maybe`\<`Record`\<`string`, `any`\>\> +###### Inherited from -Additional metadata for the user. This parameter is optional. +[`BaseGeneration`](api.md#basegeneration).[`variables`](api.md#variables) -###### Returns +##### settings -`Promise`\<`User`\> +> **settings**?: [`Maybe`](api.md#maybet)\<[`ILLMSettings`](api.md#illmsettings)\> -A promise that resolves with the updated User object. +###### Inherited from -##### getOrCreateUser() +[`BaseGeneration`](api.md#basegeneration).[`settings`](api.md#settings) -> **getOrCreateUser**(`identifier`, `metadata`?): `Promise`\<`string`\> +##### tools -Retrieves an existing user by their identifier or creates a new one if they do not exist. +> **tools**?: [`Maybe`](api.md#maybet)\<[`ITool`](api.md#itool)[]\> -###### Parameters +###### Inherited from -▪ **identifier**: `string` +[`BaseGeneration`](api.md#basegeneration).[`tools`](api.md#tools) -The unique identifier for the user. This parameter is required. +##### tokenCount -▪ **metadata?**: `Maybe`\<`Record`\<`string`, `any`\>\> +> **tokenCount**?: [`Maybe`](api.md#maybet)\<`number`\> -Additional metadata for the user. This parameter is optional. +###### Inherited from -###### Returns +[`BaseGeneration`](api.md#basegeneration).[`tokenCount`](api.md#tokencount) -`Promise`\<`string`\> +##### inputTokenCount -The ID of the existing or newly created user. +> **inputTokenCount**?: [`Maybe`](api.md#maybet)\<`number`\> -##### getUser() +###### Inherited from -> **getUser**(`identifier`): `Promise`\<`Maybe`\<`User`\>\> +[`BaseGeneration`](api.md#basegeneration).[`inputTokenCount`](api.md#inputtokencount) -Retrieves a user by their unique identifier. +##### outputTokenCount -###### Parameters +> **outputTokenCount**?: [`Maybe`](api.md#maybet)\<`number`\> -▪ **identifier**: `string` +###### Inherited from -The unique identifier for the user. This parameter is required. +[`BaseGeneration`](api.md#basegeneration).[`outputTokenCount`](api.md#outputtokencount) -###### Returns +##### ttFirstToken -`Promise`\<`Maybe`\<`User`\>\> +> **ttFirstToken**?: [`Maybe`](api.md#maybet)\<`number`\> -A `Promise` that resolves to a `User` object if found, otherwise `undefined`. +###### Inherited from -##### deleteUser() +[`BaseGeneration`](api.md#basegeneration).[`ttFirstToken`](api.md#ttfirsttoken) -> **deleteUser**(`id`): `Promise`\<`string`\> +##### tokenThroughputInSeconds -Deletes a single user by their unique identifier. +> **tokenThroughputInSeconds**?: [`Maybe`](api.md#maybet)\<`number`\> -###### Parameters +###### Inherited from + +[`BaseGeneration`](api.md#basegeneration).[`tokenThroughputInSeconds`](api.md#tokenthroughputinseconds) + +##### duration + +> **duration**?: [`Maybe`](api.md#maybet)\<`number`\> + +###### Inherited from + +[`BaseGeneration`](api.md#basegeneration).[`duration`](api.md#duration) + +##### type + +> **type**?: [`GenerationType`](api.md#generationtype) = `'COMPLETION'` + +##### prompt -▪ **id**: `string` +> **prompt**?: [`Maybe`](api.md#maybet)\<`string`\> -The unique identifier of the user to be deleted. This parameter is required. +##### completion + +> **completion**?: [`Maybe`](api.md#maybet)\<`string`\> + +#### Methods + +##### serialize() + +> **serialize**(): `any` + +Serializes the properties of the current instance into a dictionary, excluding the 'api' property. +It handles nested objects that also implement a serialize method. ###### Returns -`Promise`\<`string`\> +`any` + +A dictionary representing the serialized properties of the object. + +###### Inherited from + +[`BaseGeneration`](api.md#basegeneration).[`serialize`](api.md#serialize) + +*** + +### ChatGeneration -A `Promise` that resolves to the ID of the deleted user. +Represents a utility class with serialization capabilities. -##### getScores() +#### Extends -> **getScores**(`variables`): `Promise`\<`PaginatedResponse`\<`OmitUtils`\<`Score`\>\>\> +- [`BaseGeneration`](api.md#basegeneration) -Get all scores connected to the platform. +#### Constructors + +##### new ChatGeneration(data) + +> **new ChatGeneration**(`data`): [`ChatGeneration`](api.md#chatgeneration) ###### Parameters -▪ **variables**: `object` +▪ **data**: [`OmitUtils`](api.md#omitutilst)\<[`ChatGeneration`](api.md#chatgeneration)\> -The parameters for querying scores. +###### Returns -▪ **variables.first?**: `Maybe`\<`number`\> +[`ChatGeneration`](api.md#chatgeneration) -Optional. The number of scores to retrieve. +###### Overrides -▪ **variables.after?**: `Maybe`\<`string`\> +[`BaseGeneration`](api.md#basegeneration).[`constructor`](api.md#constructors-1) -Optional. The cursor after which to start fetching scores. +#### Properties -▪ **variables.before?**: `Maybe`\<`string`\> +##### promptId -Optional. The cursor before which to start fetching scores. +> **promptId**?: `string` -▪ **variables.filters?**: `ScoresFilter`[] +###### Inherited from -Optional. Filters to apply to the score query. +[`BaseGeneration`](api.md#basegeneration).[`promptId`](api.md#promptid) -▪ **variables.orderBy?**: `ScoresOrderBy` +##### provider -Optional. The order in which to sort the scores. +> **provider**?: [`Maybe`](api.md#maybet)\<`string`\> -###### Returns +###### Inherited from -`Promise`\<`PaginatedResponse`\<`OmitUtils`\<`Score`\>\>\> +[`BaseGeneration`](api.md#basegeneration).[`provider`](api.md#provider) -A `Promise` that resolves to a paginated response of scores, excluding certain utility fields. +##### model -##### createScores() +> **model**?: [`Maybe`](api.md#maybet)\<`string`\> -> **createScores**(`scores`): `Promise`\<`Score`[]\> +###### Inherited from -Creates multiple scores in the database using the provided array of scores. -Each score in the array is transformed into a GraphQL mutation call. +[`BaseGeneration`](api.md#basegeneration).[`model`](api.md#model) -###### Parameters +##### id -▪ **scores**: `Score`[] +> **id**?: [`Maybe`](api.md#maybet)\<`string`\> -An array of `Score` objects to be created. +###### Inherited from -###### Returns +[`BaseGeneration`](api.md#basegeneration).[`id`](api.md#id) -`Promise`\<`Score`[]\> +##### tags -A promise that resolves to an array of `Score` instances populated with the created scores' data. +> **tags**?: [`Maybe`](api.md#maybet)\<`string`[]\> -##### createScore() +###### Inherited from -> **createScore**(`variables`): `Promise`\<`Score`\> +[`BaseGeneration`](api.md#basegeneration).[`tags`](api.md#tags) -Creates a new score in the database using the provided parameters. +##### error -###### Parameters +> **error**?: [`Maybe`](api.md#maybet)\<`string`\> -▪ **variables**: `OmitUtils`\<`Score`\> +###### Inherited from -The score details to be used in the creation process. This includes: +[`BaseGeneration`](api.md#basegeneration).[`error`](api.md#error) -###### Returns +##### variables -`Promise`\<`Score`\> +> **variables**?: [`Maybe`](api.md#maybet)\<`Record`\<`string`, `any`\>\> -A new `Score` instance populated with the created score's data. +###### Inherited from -##### updateScore() +[`BaseGeneration`](api.md#basegeneration).[`variables`](api.md#variables) -> **updateScore**(`id`, `updateParams`): `Promise`\<`Score`\> +##### settings -Updates an existing score in the database. +> **settings**?: [`Maybe`](api.md#maybet)\<[`ILLMSettings`](api.md#illmsettings)\> -###### Parameters +###### Inherited from -▪ **id**: `string` +[`BaseGeneration`](api.md#basegeneration).[`settings`](api.md#settings) -The unique identifier of the score to update. (required) +##### tools -▪ **updateParams**: `object` +> **tools**?: [`Maybe`](api.md#maybet)\<[`ITool`](api.md#itool)[]\> -The parameters to update in the score. (required) +###### Inherited from -▪ **updateParams.comment?**: `Maybe`\<`string`\> +[`BaseGeneration`](api.md#basegeneration).[`tools`](api.md#tools) -A new or updated comment for the score. (optional) +##### tokenCount -▪ **updateParams.value**: `number` +> **tokenCount**?: [`Maybe`](api.md#maybet)\<`number`\> -The new value to set for the score. (required) +###### Inherited from -###### Returns +[`BaseGeneration`](api.md#basegeneration).[`tokenCount`](api.md#tokencount) -`Promise`\<`Score`\> +##### inputTokenCount -A `Score` instance representing the updated score. +> **inputTokenCount**?: [`Maybe`](api.md#maybet)\<`number`\> -##### deleteScore() +###### Inherited from -> **deleteScore**(`id`): `Promise`\<`any`\> +[`BaseGeneration`](api.md#basegeneration).[`inputTokenCount`](api.md#inputtokencount) -Deletes a single score from the database. +##### outputTokenCount -###### Parameters +> **outputTokenCount**?: [`Maybe`](api.md#maybet)\<`number`\> -▪ **id**: `string` +###### Inherited from -The unique identifier of the score to delete. (required) +[`BaseGeneration`](api.md#basegeneration).[`outputTokenCount`](api.md#outputtokencount) -###### Returns +##### ttFirstToken -`Promise`\<`any`\> +> **ttFirstToken**?: [`Maybe`](api.md#maybet)\<`number`\> -The ID of the deleted score. +###### Inherited from -##### createDataset() +[`BaseGeneration`](api.md#basegeneration).[`ttFirstToken`](api.md#ttfirsttoken) -> **createDataset**(`dataset`): `Promise`\<`Dataset`\> +##### tokenThroughputInSeconds -Creates a new dataset in the database. +> **tokenThroughputInSeconds**?: [`Maybe`](api.md#maybet)\<`number`\> -###### Parameters +###### Inherited from + +[`BaseGeneration`](api.md#basegeneration).[`tokenThroughputInSeconds`](api.md#tokenthroughputinseconds) -▪ **dataset**: `object` +##### duration -The dataset details to be created. +> **duration**?: [`Maybe`](api.md#maybet)\<`number`\> -▪ **dataset.name**: `string` +###### Inherited from -The name of the dataset. (required) +[`BaseGeneration`](api.md#basegeneration).[`duration`](api.md#duration) -▪ **dataset.description?**: `Maybe`\<`string`\> +##### type -The description of the dataset. (optional) +> **type**?: [`GenerationType`](api.md#generationtype) = `'CHAT'` -▪ **dataset.metadata?**: `Maybe`\<`Record`\<`string`, `any`\>\> +##### messages -Additional metadata for the dataset as a key-value pair object. (optional) +> **messages**?: [`Maybe`](api.md#maybet)\<[`IGenerationMessage`](api.md#igenerationmessage)[]\> = `[]` -▪ **dataset.type?**: `DatasetType` +##### messageCompletion + +> **messageCompletion**?: [`Maybe`](api.md#maybet)\<[`IGenerationMessage`](api.md#igenerationmessage)\> + +#### Methods -The type of the dataset, defined by the DatasetType enum. (optional) +##### serialize() + +> **serialize**(): `any` + +Serializes the properties of the current instance into a dictionary, excluding the 'api' property. +It handles nested objects that also implement a serialize method. ###### Returns -`Promise`\<`Dataset`\> +`any` -A new Dataset instance populated with the created dataset's data. +A dictionary representing the serialized properties of the object. -##### getDataset() +###### Inherited from -> **getDataset**(`variables`): `Promise`\<`null` \| `Dataset`\> +[`BaseGeneration`](api.md#basegeneration).[`serialize`](api.md#serialize) -Retrieves a dataset based on provided ID or name. +*** -###### Parameters +### Utils + +Represents a utility class with serialization capabilities. -▪ **variables**: `object` +#### Extended By -An object containing optional `id` and `name` properties to specify which dataset to retrieve. +- [`User`](api.md#user) +- [`BaseGeneration`](api.md#basegeneration) -▪ **variables.id?**: `string` +#### Constructors -▪ **variables.name?**: `string` +##### new Utils() + +> **new Utils**(): [`Utils`](api.md#utils) ###### Returns -`Promise`\<`null` \| `Dataset`\> +[`Utils`](api.md#utils) -A `Dataset` instance populated with the retrieved dataset's data, or `null` if no data is found. +#### Methods -##### updateDataset() +##### serialize() -> **updateDataset**(`id`, `dataset`): `Promise`\<`Dataset`\> +> **serialize**(): `any` -Updates a dataset with new information. +Serializes the properties of the current instance into a dictionary, excluding the 'api' property. +It handles nested objects that also implement a serialize method. -###### Parameters +###### Returns -▪ **id**: `string` +`any` -The unique identifier of the dataset to update. This parameter is required. +A dictionary representing the serialized properties of the object. -▪ **dataset**: `object` +*** -An object containing the new dataset information. +### User -▪ **dataset.name?**: `Maybe`\<`string`\> +Represents a user with optional metadata and identifier. -The new name of the dataset. (optional) +#### Extends -▪ **dataset.description?**: `Maybe`\<`string`\> +- [`Utils`](api.md#utils) -The new description of the dataset. (optional) +#### Constructors -▪ **dataset.metadata?**: `Maybe`\<`Record`\<`string`, `any`\>\> +##### new User(data) -Additional metadata for the dataset as a key-value pair object. (optional) +> **new User**(`data`): [`User`](api.md#user) + +###### Parameters + +▪ **data**: [`OmitUtils`](api.md#omitutilst)\<[`User`](api.md#user)\> ###### Returns -`Promise`\<`Dataset`\> +[`User`](api.md#user) -A new `Dataset` instance populated with the updated dataset's data. +###### Overrides -##### deleteDataset() +[`Utils`](api.md#utils).[`constructor`](api.md#constructors-4) -> **deleteDataset**(`id`): `Promise`\<`Dataset`\> +#### Properties -Deletes a single dataset by its unique identifier. +##### id -###### Parameters +> **id**?: [`Maybe`](api.md#maybet)\<`string`\> + +##### identifier + +> **identifier**: `string` -▪ **id**: `string` +##### metadata -The unique identifier of the dataset to delete. This parameter is required. +> **metadata**?: [`Maybe`](api.md#maybet)\<`Record`\<`string`, `any`\>\> + +#### Methods + +##### serialize() + +> **serialize**(): `any` + +Serializes the properties of the current instance into a dictionary, excluding the 'api' property. +It handles nested objects that also implement a serialize method. ###### Returns -`Promise`\<`Dataset`\> +`any` -A new `Dataset` instance populated with the deleted dataset's data. +A dictionary representing the serialized properties of the object. -##### createDatasetItem() +###### Inherited from -> **createDatasetItem**(`datasetId`, `datasetItem`): `Promise`\<`DatasetItem`\> +[`Utils`](api.md#utils).[`serialize`](api.md#serialize-3) -Creates a new item in a dataset. +## Interfaces -###### Parameters +### ITextContent -▪ **datasetId**: `string` +#### Properties -The unique identifier of the dataset. This parameter is required. +##### type -▪ **datasetItem**: `object` +> **type**: `"text"` -The data for the new dataset item. This parameter is required. +##### text -▪ **datasetItem.input**: `Record`\<`string`, `any`\> +> **text**: `string` -The input data for the dataset item. This field is required. +*** -▪ **datasetItem.expectedOutput?**: `Maybe`\<`Record`\<`string`, `any`\>\> +### IImageUrlContent -The expected output data for the dataset item. This field is optional. +#### Properties -▪ **datasetItem.metadata?**: `Maybe`\<`Record`\<`string`, `any`\>\> +##### type -Additional metadata for the dataset item. This field is optional. +> **type**: `"image_url"` -###### Returns +##### image\_url -`Promise`\<`DatasetItem`\> +> **image\_url**: `string` -A new `DatasetItem` instance populated with the created dataset item's data. +*** -##### getDatasetItem() +### IGenerationMessage -> **getDatasetItem**(`id`): `Promise`\<`DatasetItem`\> +#### Properties -Retrieves a single item from a dataset by its unique identifier. +##### uuid -###### Parameters +> **uuid**?: `string` -▪ **id**: `string` +##### templated -The unique identifier of the dataset item. This parameter is required. +> **templated**?: `boolean` -###### Returns +##### content -`Promise`\<`DatasetItem`\> +> **content**: `null` \| `string` \| ([`ITextContent`](api.md#itextcontent) \| [`IImageUrlContent`](api.md#iimageurlcontent))[] -A `DatasetItem` instance populated with the retrieved dataset item's data. +##### role -##### deleteDatasetItem() +> **role**: [`GenerationMessageRole`](api.md#generationmessagerole) -> **deleteDatasetItem**(`id`): `Promise`\<`DatasetItem`\> +##### name -Deletes a single item from a dataset by its unique identifier. +> **name**?: `string` -###### Parameters +##### function\_call -▪ **id**: `string` +> **function\_call**?: `Record`\<`string`, `any`\> -The unique identifier of the dataset item to be deleted. This parameter is required. +##### tool\_calls -###### Returns +> **tool\_calls**?: `Record`\<`string`, `any`\>[] -`Promise`\<`DatasetItem`\> +##### tool\_call\_id -A `DatasetItem` instance populated with the data of the deleted dataset item. +> **tool\_call\_id**?: `string` -##### addStepToDataset() +*** -> **addStepToDataset**(`datasetId`, `stepId`, `metadata`?): `Promise`\<`DatasetItem`\> +### IFunction -Adds a single step item to a dataset. +#### Properties -###### Parameters +##### name -▪ **datasetId**: `string` +> **name**: `string` -The unique identifier of the dataset. This parameter is required. +##### description -▪ **stepId**: `string` +> **description**: `string` -The unique identifier of the step to be added. This parameter is required. +##### parameters -▪ **metadata?**: `Maybe`\<`Record`\<`string`, `unknown`\>\> +> **parameters**: `object` -Additional metadata for the step as a JSON object. This parameter is optional. +###### Type declaration -###### Returns +###### required -`Promise`\<`DatasetItem`\> +> **required**: `string`[] -A `DatasetItem` instance populated with the data of the newly added step. +###### properties -##### addGenerationToDataset() +> **properties**: `Record`\<`string`, `object`\> -> **addGenerationToDataset**(`datasetId`, `generationId`, `metadata`?): `Promise`\<`DatasetItem`\> +*** -Adds a generation item to a dataset. +### ITool -###### Parameters +#### Properties -▪ **datasetId**: `string` +##### type -The unique identifier of the dataset. This parameter is required. +> **type**: `string` -▪ **generationId**: `string` +##### function -The unique identifier of the generation to be added. This parameter is required. +> **function**: [`IFunction`](api.md#ifunction) -▪ **metadata?**: `Maybe`\<`Record`\<`string`, `unknown`\>\> +## Type Aliases -Additional metadata for the generation as a JSON object. This parameter is optional. +### OpenAIGlobalOptions -###### Returns +> **OpenAIGlobalOptions**: `object` -`Promise`\<`DatasetItem`\> +#### Type declaration -A `DatasetItem` instance populated with the data of the newly added generation. +##### tags -##### addGenerationsToDataset() +> **tags**?: [`Maybe`](api.md#maybet)\<`string`[]\> -> **addGenerationsToDataset**(`datasetId`, `generationIds`): `Promise`\<`DatasetItem`[]\> +##### metadata -###### Parameters +> **metadata**?: [`Maybe`](api.md#maybet)\<`Record`\<`string`, `any`\>\> -▪ **datasetId**: `string` +*** -▪ **generationIds**: `string`[] +### GenerationMessageRole -###### Returns +> **GenerationMessageRole**: `"system"` \| `"assistant"` \| `"user"` \| `"function"` \| `"tool"` -`Promise`\<`DatasetItem`[]\> +*** -##### createExperiment() +### ILLMSettings -> **createExperiment**(`datasetExperiment`): `Promise`\<`DatasetExperiment`\> +> **ILLMSettings**: `Record`\<`string`, `string` \| `string`[] \| `number` \| `boolean`\> -###### Parameters +*** -▪ **datasetExperiment**: `object` +### GenerationType -▪ **datasetExperiment.name**: `string` +> **GenerationType**: `"COMPLETION"` \| `"CHAT"` -▪ **datasetExperiment.datasetId**: `string` +*** -▪ **datasetExperiment.promptId?**: `string` +### Generation -▪ **datasetExperiment.params?**: `Record`\<`string`, `any`\> \| `Record`\<`string`, `any`\>[] +> **Generation**: [`OmitUtils`](api.md#omitutilst)\<[`CompletionGeneration`](api.md#completiongeneration)\> \| [`OmitUtils`](api.md#omitutilst)\<[`ChatGeneration`](api.md#chatgeneration)\> -###### Returns +*** -`Promise`\<`DatasetExperiment`\> +### PersistedGeneration -##### createExperimentItem() +> **PersistedGeneration**: [`Generation`](api.md#generation) & `object` -> **createExperimentItem**(`__namedParameters`): `Promise`\<`DatasetExperimentItem`\> +#### Type declaration -###### Parameters +##### id -▪ **\_\_namedParameters**: `DatasetExperimentItem` +> **id**: `string` -###### Returns +*** -`Promise`\<`DatasetExperimentItem`\> +### Maybe`` -##### createPromptLineage() +> **Maybe**\<`T`\>: `T` \| `null` \| `undefined` -> **createPromptLineage**(`name`, `description`?): `Promise`\<`any`\> +#### Type parameters -Create a new prompt lineage. +| Parameter | +| :------ | +| `T` | -###### Parameters +*** -▪ **name**: `string` +### OmitUtils`` -The name of the prompt lineage. This parameter is required. +> **OmitUtils**\<`T`\>: `Omit`\<`T`, keyof [`Utils`](api.md#utils)\> -▪ **description?**: `string` +#### Type parameters -A description for the prompt lineage. This parameter is optional. +| Parameter | +| :------ | +| `T` | -###### Returns +*** -`Promise`\<`any`\> +### Environment -The newly created prompt lineage object, or null if creation failed. +> **Environment**: `"dev"` \| `"staging"` \| `"prod"` \| `"experiment"` -##### createPrompt() +*** -> **createPrompt**(`name`, `templateMessages`, `settings`?): `Promise`\<`Prompt`\> +### PageInfo -Create a new prompt. +> **PageInfo**: `object` -###### Parameters +#### Type declaration -▪ **name**: `string` +##### hasNextPage -The name of the prompt lineage. This parameter is required. +> **hasNextPage**: `boolean` -▪ **templateMessages**: `IGenerationMessage`[] +##### startCursor -An array of template messages defining the prompt. This parameter is required. +> **startCursor**: `string` -▪ **settings?**: `Maybe`\<`Record`\<`string`, `any`\>\> +##### endCursor -Optional settings for the prompt creation, provided as a record of string keys to any type of values. +> **endCursor**: `string` -###### Returns +*** -`Promise`\<`Prompt`\> +### PaginatedResponse`` -A new Prompt instance containing the created prompt data. +> **PaginatedResponse**\<`T`\>: `object` -##### getPrompt() +#### Type parameters -> **getPrompt**(`name`, `version`?): `Promise`\<`null` \| `Prompt`\> +| Parameter | +| :------ | +| `T` | -Retrieves a prompt by its name and optionally by its version. +#### Type declaration -###### Parameters +##### data -▪ **name**: `string` +> **data**: `T`[] -The name of the prompt to retrieve. +##### pageInfo -▪ **version?**: `number` +> **pageInfo**: [`PageInfo`](api.md#pageinfo) -The version number of the prompt (optional). +## Functions -###### Returns +### isPlainObject() + +> **isPlainObject**(`value`): `value is Record` + +#### Parameters + +▪ **value**: `unknown` -`Promise`\<`null` \| `Prompt`\> +#### Returns -An instance of `Prompt` containing the prompt data, or `null` if not found. +`value is Record` *** diff --git a/src/api.ts b/src/api.ts index f2f2084..5ca43de 100644 --- a/src/api.ts +++ b/src/api.ts @@ -5,6 +5,14 @@ import { ReadStream } from 'fs'; import { v4 as uuidv4 } from 'uuid'; import { LiteralClient } from '.'; +import { + Dataset, + DatasetExperiment, + DatasetExperimentItem, + DatasetItem, + DatasetType +} from './evaluation/dataset'; +import { Score, ScoreConstructor } from './evaluation/score'; import { GenerationsFilter, GenerationsOrderBy, @@ -16,32 +24,23 @@ import { ThreadsFilter, ThreadsOrderBy } from './filter'; +import { Attachment } from './observability/attachment'; import { Generation, IGenerationMessage, PersistedGeneration -} from './generation'; +} from './observability/generation'; +import { Step, StepType } from './observability/step'; +import { CleanThreadFields, Thread } from './observability/thread'; +import { Prompt } from './prompt-engineering/prompt'; import { - Attachment, - CleanThreadFields, - Dataset, - DatasetExperiment, - DatasetExperimentItem, - DatasetItem, - DatasetType, Environment, Maybe, OmitUtils, PaginatedResponse, - Prompt, - Score, - ScoreConstructor, - Step, - StepType, - Thread, User, Utils -} from './types'; +} from './utils'; // eslint-disable-next-line @typescript-eslint/no-var-requires const packageJson = require('../package.json'); @@ -118,13 +117,6 @@ steps { ${stepFields} }`; -/** - * Serializes the step object with a suffix ID to each key. - * - * @param object - The step object to serialize. - * @param id - The numeric identifier to append to each key in the serialized object. - * @returns A new object with serialized key-value pairs where each key is suffixed with the provided id. - */ function serialize(object: Utils, id: number) { const result: any = {}; @@ -135,12 +127,6 @@ function serialize(object: Utils, id: number) { return result; } -/** - * Constructs a variables object for GraphQL queries by serializing each step with a unique suffix. - * - * @param objects - An array of `Step` objects to be serialized and added to the variables object. - * @returns An object containing serialized steps with keys suffixed by their index in the input array. - */ function variablesBuilder(objects: Utils[]) { let variables: any = {}; for (let i = 0; i < objects.length; i++) { @@ -161,13 +147,6 @@ function generationsVariablesBuilder( return variables; } -/** - * Builds a string for GraphQL field definitions for ingesting multiple steps. - * Each step's fields are suffixed with its index to create unique variable names. - * - * @param steps - An array of `Step` objects. Each `Step` object represents a step to be ingested. - * @returns A string containing GraphQL field definitions for all provided steps. - */ function ingestStepsFieldsBuilder(steps: Step[]) { let generated = ''; for (let id = 0; id < steps.length; id++) { @@ -191,14 +170,6 @@ function ingestStepsFieldsBuilder(steps: Step[]) { return generated; } -/** - * Constructs the arguments for a GraphQL mutation to ingest multiple steps. - * Each step is transformed into a call to the `ingestStep` mutation with parameters - * suffixed by the step's index to ensure uniqueness. - * - * @param steps - An array of `Step` objects. Each `Step` object represents a step to be ingested. - * @returns A string containing the GraphQL mutation arguments for all provided steps. - */ function ingestStepsArgsBuilder(steps: Step[]) { let generated = ''; for (let id = 0; id < steps.length; id++) { @@ -228,12 +199,6 @@ function ingestStepsArgsBuilder(steps: Step[]) { return generated; } -/** - * Constructs a complete GraphQL mutation query for adding multiple steps. - * - * @param steps - An array of `Step` objects to be ingested. This parameter is required. - * @returns A string representing the complete GraphQL mutation for adding steps. - */ function ingestStepsQueryBuilder(steps: Step[]) { return ` mutation AddStep(${ingestStepsFieldsBuilder(steps)}) { @@ -616,7 +581,6 @@ export class API { return result.data.deleteStep.id; } - // Upload /** * Uploads a file to a specified thread. This method supports uploading either through direct content or via a file path. * It first signs the upload through a pre-configured endpoint and then proceeds to upload the file using the signed URL. @@ -630,7 +594,6 @@ export class API { * @returns An object containing the `objectKey` of the uploaded file and the signed `url`, or `null` values if the upload fails. * @throws {Error} Throws an error if neither `content` nor `path` is provided, or if the server response is invalid. */ - async uploadFile(params: UploadFileParamsWithContent): Promise<{ objectKey: Maybe; url: Maybe; @@ -718,6 +681,21 @@ export class API { } } + /** + * Uploads a file to a specified thread and creates an attachment object. + * If called inside a context, the attachment will be added to the current step and thread. + * + * @param params - The parameters for uploading a file, including: + * @param params.name - The name of the file. + * @param params.metadata - Additional metadata for the file as a key-value pair object. + * @param params.content - The content of the file to upload. Optional if `path` is provided. + * @param params.path - The path to the file to upload. Optional if `content` is provided. + * @param params.id - The unique identifier for the file. If not provided, a UUID will be generated. + * @param params.threadId - The unique identifier of the thread to which the file is being uploaded. + * @param params.mime - The MIME type of the file. Defaults to 'application/octet-stream' if not provided. + * @returns An object containing the `objectKey` of the uploaded file and the signed `url`, or `null` values if the upload fails. + * @throws {Error} Throws an error if neither `content` nor `path` is provided, or if the server response is invalid. + */ async createAttachment( params: UploadFileParamsWithContent & CreateAttachmentParams ): Promise; @@ -764,7 +742,6 @@ export class API { return attachment; } - // Generation /** * Retrieves a paginated list of Generations based on the provided filters and sorting order. * @@ -874,7 +851,6 @@ export class API { return response.data.createGeneration as PersistedGeneration; } - // Thread /** * Upserts a Thread with new information. * @@ -958,6 +934,7 @@ export class API { }; const response = await this.makeGqlCall(query, variables); + return new Thread(this.client, response.data.upsertThread); } @@ -1074,10 +1051,10 @@ export class API { const variables = { threadId: id }; const response = await this.makeGqlCall(query, variables); + return response.data.deleteThread.id; } - // User /** * Retrieves a list of users with optional filters. * @@ -1282,7 +1259,6 @@ export class API { return result.data.deleteParticipant.id; } - // Score /** * Get all scores connected to the platform. * @@ -1502,8 +1478,6 @@ export class API { return result.data.deleteScore; } - // Dataset - /** * List all datasets in the platform. * @@ -1805,6 +1779,13 @@ export class API { return new DatasetItem(result.data.addGenerationToDataset); } + /** + * Adds multiple generation items to a dataset. + * + * @param datasetId - The unique identifier of the dataset. This parameter is required. + * @param generationIds - An array of unique identifiers for the generations to be added. This parameter is required. + * @returns An array of `DatasetItem` instances populated with the data of the newly added generations + */ public async addGenerationsToDataset( datasetId: string, generationIds: string[] @@ -1816,6 +1797,15 @@ export class API { return Object.values(result.data).map((x: any) => new DatasetItem(x)); } + /** + * Creates a new dataset experiment. + * @param datasetExperiment + * @param datasetExperiment.name The name of the dataset experiment. + * @param datasetExperiment.datasetId The dataset ID to associate with the experiment. + * @param datasetExperiment.promptId The prompt ID to associate with the experiment. + * @param datasetExperiment.params The parameters for the experiment as a key-value pair object or an array of the same. + * @returns The newly created dataset experiment object. + */ public async createExperiment(datasetExperiment: { name: string; datasetId?: string; @@ -1840,6 +1830,18 @@ export class API { return new DatasetExperiment(this, result.data.createDatasetExperiment); } + /** + * Creates a new dataset experiment item. + * + * @param parameters + * @param parameters.datasetExperimentId The dataset experiment ID to associate with the item (required) + * @param parameters.scores An array of scores to associate with the item (required) + * @param parameters.datasetItemId The ID of the dataset item (optional) + * @param parameters.experimentRunId The ID of the experiment run (optional) + * @param parameters.input The input data for the item (optional) + * @param parameters.output The output data for the item (optional) + * @returns The dataset experiment object. + */ public async createExperimentItem({ datasetExperimentId, datasetItemId, @@ -1881,7 +1883,6 @@ export class API { }); } - // Prompt /** * Create a new prompt lineage. * @@ -1935,6 +1936,7 @@ export class API { * @param name The name of the prompt to retrieve or create. * @param templateMessages A list of template messages for the prompt. * @param settings Optional settings for the prompt. + * @param tools Optional tools for the prompt. * @returns The prompt that was retrieved or created. */ public async getOrCreatePrompt( @@ -2050,7 +2052,7 @@ export class API { return await this.getPromptWithQuery(query, { name, version }); } - public async getPromptWithQuery( + private async getPromptWithQuery( query: string, variables: Record ) { diff --git a/src/evaluation/dataset.ts b/src/evaluation/dataset.ts new file mode 100644 index 0000000..3dfcf96 --- /dev/null +++ b/src/evaluation/dataset.ts @@ -0,0 +1,245 @@ +import { API } from '../api'; +import { Maybe, OmitUtils, Utils } from '../utils'; +import { ScoreConstructor } from './score'; + +export type DatasetType = 'key_value' | 'generation'; + +class DatasetFields extends Utils { + id!: string; + createdAt!: string; + name?: Maybe; + description?: Maybe; + metadata!: Record; + items!: Array>; + type?: DatasetType; +} + +export type DatasetConstructor = OmitUtils; + +export class Dataset extends DatasetFields { + api: API; + + /** + * Constructs a new Dataset instance. + * @param api - The API instance to interact with backend services. + * @param data - The initial data for the dataset. + */ + constructor(api: API, data: DatasetConstructor) { + super(); + this.api = api; + Object.assign(this, data); + if (!this.items) { + this.items = []; + } + if (!this.type) { + this.type = 'key_value'; + } + } + + /** + * Updates the dataset with new data. + * @param dataset - The dataset data to update. + * @returns The updated dataset instance. + */ + async update(dataset: { + name?: Maybe; + description?: Maybe; + metadata?: Maybe>; + }) { + const update_res = await this.api.updateDataset(this.id, dataset); + this.name = update_res.name; + this.description = update_res.description; + this.metadata = update_res.metadata; + } + + /** + * Deletes the dataset. + * @returns A promise that resolves when the dataset is deleted. + */ + async delete() { + return this.api.deleteDataset(this.id); + } + + /** + * Creates a new item in the dataset. + * @param datasetItem - The new item to be added to the dataset. + * @returns The newly created dataset item. + */ + async createItem(datasetItem: { + input: Record; + expectedOutput?: Maybe>; + metadata?: Maybe>; + }) { + const item = await this.api.createDatasetItem(this.id, datasetItem); + + this.items.push(item); + return item; + } + + /** + * Deletes an item from the dataset. + * @param id - The ID of the item to delete. + * @returns The deleted dataset item. + */ + async deleteItem(id: string) { + const deletedItem = await this.api.deleteDatasetItem(id); + if (this.items) { + this.items = this.items.filter((item) => item.id !== id); + } + return deletedItem; + } + + /** + * Creates a new experiment associated with the dataset. + * @param experiment - The experiment details including name, optional prompt ID, and parameters. + * @returns A new instance of DatasetExperiment containing the created experiment. + */ + async createExperiment(experiment: { + name: string; + promptId?: string; + params?: Record | Array>; + }) { + const datasetExperiment = await this.api.createExperiment({ + name: experiment.name, + datasetId: this.id, + promptId: experiment.promptId, + params: experiment.params + }); + return new DatasetExperiment(this.api, datasetExperiment); + } + + /** + * Adds a step to the dataset. + * @param stepId - The ID of the step to add. + * @param metadata - Optional metadata for the step. + * @returns The added dataset item. + * @throws Error if the dataset type is 'generation'. + */ + public async addStep( + stepId: string, + metadata?: Maybe> + ) { + if (this.type === 'generation') { + throw new Error('Cannot add steps to a generation dataset'); + } + const item = await this.api.addStepToDataset(this.id, stepId, metadata); + this.items.push(item); + return item; + } + + /** + * Adds a generation to the dataset. + * @param generationId - The ID of the generation to add. + * @param metadata - Optional metadata for the generation. + * @returns The added dataset item. + */ + public async addGeneration( + generationId: string, + metadata?: Maybe> + ) { + const item = await this.api.addGenerationToDataset( + this.id, + generationId, + metadata + ); + this.items.push(item); + return item; + } + + /** + * Adds multiple generations to the dataset. + * @param generationIds - The IDs of the steps to add. + * @returns The added dataset items. + */ + public async addGenerations(generationIds?: string[]) { + if (generationIds == undefined || generationIds?.length === 0) { + return []; + } + + const items = await this.api.addGenerationsToDataset( + this.id, + generationIds + ); + this.items = this.items.concat(items); + return items; + } +} + +export class DatasetItem extends Utils { + id!: string; + createdAt!: string; + datasetId!: string; + metadata!: Record; + input!: Record; + expectedOutput?: Maybe>; + intermediarySteps!: Array>; + + constructor(data: OmitUtils) { + super(); + Object.assign(this, data); + } +} + +class DatasetExperimentItemFields extends Utils { + id?: string; + datasetExperimentId!: string; + datasetItemId?: string; + experimentRunId?: string; + scores!: ScoreConstructor[]; + input?: Record; + output?: Record; +} + +export class DatasetExperiment extends Utils { + id!: string; + createdAt!: string; + name!: string; + datasetId?: string; + promptId?: string; + api: API; + params!: Record | Array>; + items!: DatasetExperimentItem[]; + constructor(api: API, data: OmitUtils) { + super(); + this.api = api; + Object.assign(this, data); + if (!this.items) { + this.items = []; + } + } + + /** + * Logs an item in the dataset experiment. + * @param itemFields the data for this item + * @returns the created item + */ + async log( + itemFields: Omit< + OmitUtils, + 'id' | 'datasetExperimentId' + > + ) { + const experimentRunId = this.api.client._currentExperimentItemRunId(); + + const datasetExperimentItem = new DatasetExperimentItem({ + ...itemFields, + datasetExperimentId: this.id, + ...(experimentRunId && { experimentRunId }) + }); + + const item = await this.api.createExperimentItem(datasetExperimentItem); + + this.items.push(item); + return item; + } +} + +export type DatasetExperimentItemConstructor = + OmitUtils; + +export class DatasetExperimentItem extends DatasetExperimentItemFields { + constructor(data: DatasetExperimentItemConstructor) { + super(); + Object.assign(this, data); + } +} diff --git a/src/evaluation/experiment-item-run.ts b/src/evaluation/experiment-item-run.ts new file mode 100644 index 0000000..e7fbbfd --- /dev/null +++ b/src/evaluation/experiment-item-run.ts @@ -0,0 +1,61 @@ +import { LiteralClient } from '..'; +import { API } from '../api'; +import { Step, StepConstructor } from '../observability/step'; + +/** + * Represents an item in an experiment. + */ +export class ExperimentItemRun extends Step { + api: API; + client: LiteralClient; + + /** + * Constructs a new ExperimentItemRun instance. + * @param api The API instance to be used for sending and managing steps. + * @param data The initial data for the step, excluding utility properties. + */ + constructor( + client: LiteralClient, + data: StepConstructor, + ignoreContext?: true + ) { + super(client, data, ignoreContext); + this.client = client; + this.api = client.api; + } + + async wrap( + cb: (step: Step) => Output | Promise, + updateStep?: + | Partial + | ((output: Output) => Partial) + | ((output: Output) => Promise>) + ) { + const originalEnvironment = this.api.environment; + this.api.environment = 'experiment'; + + const currentStore = this.client.store.getStore(); + const output: Output = await this.client.store.run( + { + currentThread: currentStore?.currentThread ?? null, + currentStep: this, + currentExperimentItemRunId: this.id ?? null + }, + async () => { + try { + const output = await super.wrap(cb, updateStep); + return output; + } finally { + // Clear the currentExperimentRunId after execution + const updatedStore = this.client.store.getStore(); + if (updatedStore) { + updatedStore.currentExperimentItemRunId = null; + } + } + } + ); + + this.api.environment = originalEnvironment; + return output; + } +} diff --git a/src/evaluation/score.ts b/src/evaluation/score.ts new file mode 100644 index 0000000..6915aa1 --- /dev/null +++ b/src/evaluation/score.ts @@ -0,0 +1,29 @@ +import { Maybe, OmitUtils, Utils } from '../utils'; + +export type ScoreType = 'HUMAN' | 'AI'; + +class ScoreFields extends Utils { + id?: Maybe; + stepId?: Maybe; + generationId?: Maybe; + datasetExperimentItemId?: Maybe; + name: string = 'user-feedback'; + value: number = 1; + type: ScoreType = 'HUMAN'; + scorer?: Maybe; + comment?: Maybe; + tags?: Maybe; +} + +export type ScoreConstructor = OmitUtils; + +/** + * Represents a score entity with properties to track various aspects of scoring. + * It extends the `Utils` class for serialization capabilities. + */ +export class Score extends ScoreFields { + constructor(data: ScoreConstructor) { + super(); + Object.assign(this, data); + } +} diff --git a/src/index.ts b/src/index.ts index 12384e8..b4ae30a 100644 --- a/src/index.ts +++ b/src/index.ts @@ -1,19 +1,15 @@ import { AsyncLocalStorage } from 'node:async_hooks'; import { API } from './api'; +import { ExperimentItemRun } from './evaluation/experiment-item-run'; import instrumentation from './instrumentation'; -import openai from './openai'; -import { - Environment, - ExperimentItemRun, - Step, - StepConstructor, - Thread, - ThreadConstructor -} from './types'; - -export * from './types'; -export * from './generation'; +import openai from './instrumentation/openai-syncer'; +import { Step, StepConstructor } from './observability/step'; +import { Thread, ThreadConstructor } from './observability/thread'; +import { Environment } from './utils'; + +export * from './utils'; +export * from './observability/generation'; export type * from './instrumentation'; @@ -31,6 +27,15 @@ export class LiteralClient { instrumentation: ReturnType; store: AsyncLocalStorage = storage; + /** + * Initialize a new Literal AI Client. + * @param options + * @param options.apiKey The API key to use for the Literal AI API. Defaults to the LITERAL_API_KEY environment variable. + * @param options.apiUrl The URL of the Literal AI API. Defaults to the LITERAL_API_URL env var, or https://cloud.getliteral.ai. + * @param options.environment The environment to use for the Literal AI API. + * @param options.disabled If set to true, no call will be made to the Literal AI API. + * @returns A new LiteralClient instance. + */ constructor({ apiKey, apiUrl, @@ -55,14 +60,29 @@ export class LiteralClient { this.instrumentation = instrumentation(this); } + /** + * Creates a new thread without sending it to the Literal AI API. + * @param data Optional initial data for the thread. + * @returns A new thread instance. + */ thread(data?: ThreadConstructor) { return new Thread(this, data); } + /** + * Creates a new step without sending it to the Literal AI API. + * @param data Optional initial data for the step. + * @returns A new step instance. + */ step(data: StepConstructor) { return new Step(this, data); } + /** + * Creates a new step with the type set to 'run'. + * @param data Optional initial data for the step. + * @returns A new step instance. + */ run(data: Omit) { return this.step({ ...data, type: 'run' }); } @@ -75,18 +95,36 @@ export class LiteralClient { }); } + /** + * Returns the current thread from the context or null if none. + * @returns The current thread, if any. + */ _currentThread(): Thread | null { const store = storage.getStore(); return store?.currentThread || null; } + /** + * Returns the current step from the context or null if none. + * @returns The current step, if any. + */ _currentStep(): Step | null { const store = storage.getStore(); return store?.currentStep || null; } + /** + * Returns the current experiment from the context or null if none. + * @returns The current experiment, if any. + */ + _currentExperimentItemRunId(): string | null { + const store = storage.getStore(); + + return store?.currentExperimentItemRunId || null; + } + /** * Gets the current thread from the context. * WARNING : this will throw if run outside of a thread context. @@ -120,4 +158,21 @@ export class LiteralClient { return store.currentStep; } + + /** + * Gets the current experiment run ID from the context. + * WARNING : this will throw if run outside of an experiment context. + * @returns The current experiment, if any. + */ + getCurrentExperimentItemRunId(): string { + const store = storage.getStore(); + + if (!store?.currentExperimentItemRunId) { + throw new Error( + 'Literal AI SDK : tried to access current experiment outside of a context.' + ); + } + + return store?.currentExperimentItemRunId; + } } diff --git a/src/instrumentation/index.ts b/src/instrumentation/index.ts index d382c0b..c616b0d 100644 --- a/src/instrumentation/index.ts +++ b/src/instrumentation/index.ts @@ -10,25 +10,48 @@ export type OpenAIGlobalOptions = { }; export default (client: LiteralClient) => ({ + /** + * Instrument the OpenAI client to log all generations to the Literal AI API. + * Compatible with OpenAI's `chat.completions.create`, `completions.create` and `images.generate` functions. + * If you want to add tags or metadata at the call level, you should use the augmented client returned by this function. + * Its functions will be augmented with `literalaiTags` and `literalaiMetadata` options. + * @param options + * @param options.tags Tags to attach to all generations. + * @param options.metadata Metadata to attach to all generations. + * @returns + */ openai: (options?: OpenAIGlobalOptions) => instrumentOpenAI(client, options), + langchain: { literalCallback: (threadId?: string) => { try { return new LiteralCallbackHandler(client, threadId); - - // Proceed with using `handler` as intended } catch (error) { throw new Error( - 'Failed to load the langchain. Please ensure langchain is installed.' + 'Failed to load langchain. Please ensure langchain is installed.' ); } } }, + vercel: { + /** + * Instrument a Vercel SDK function to log all invocations to the Literal AI API. + * It will be augmented with a `literalAiParent` option that allows you to specify a parent step or thread. + * @param fn The function to instrument. This can be Vercel SDK's `generateText` or `streamText` functions. + * @returns A new function that logs all invocations to the Literal AI API. + */ instrument: makeInstrumentVercelSDK(client) }, + llamaIndex: { + /** + * Instrument the LlamaIndex client to log all generations to the Literal AI API. + */ instrument: () => instrumentLlamaIndex(client), + /** + * Run a callback with a thread context. + */ withThread } }); diff --git a/src/instrumentation/langchain.ts b/src/instrumentation/langchain.ts index b5e95d2..0e3f398 100644 --- a/src/instrumentation/langchain.ts +++ b/src/instrumentation/langchain.ts @@ -30,7 +30,7 @@ import { ITool, LiteralClient } from '..'; -import { Step } from '../types'; +import { Step } from '../observability/step'; // @ts-expect-error Generics export class CustomChatPromptTemplate extends ChatPromptTemplate { diff --git a/src/instrumentation/llamaindex.ts b/src/instrumentation/llamaindex.ts index d2beb03..5c6e0d6 100644 --- a/src/instrumentation/llamaindex.ts +++ b/src/instrumentation/llamaindex.ts @@ -7,9 +7,9 @@ import { type IGenerationMessage, IImageUrlContent, ITextContent, - LiteralClient, - Thread + LiteralClient } from '..'; +import { Thread } from '../observability/thread'; const convertMessage = (message: ChatMessage): IGenerationMessage => { return { diff --git a/src/openai.ts b/src/instrumentation/openai-syncer.ts similarity index 93% rename from src/openai.ts rename to src/instrumentation/openai-syncer.ts index 48c5bec..fa0032c 100644 --- a/src/openai.ts +++ b/src/instrumentation/openai-syncer.ts @@ -8,9 +8,10 @@ import type { } from 'openai/resources/beta/threads/runs/steps'; import { v5 as uuidv5 } from 'uuid'; -import { LiteralClient } from '.'; -import { ChatGeneration } from './generation'; -import { Attachment, User } from './types'; +import { LiteralClient } from '..'; +import { Attachment } from '../observability/attachment'; +import { ChatGeneration } from '../observability/generation'; +import { User } from '../utils'; class OpenAIAssistantSyncer { private NAMESPACE_UUID = '1b671a64-40d5-491e-99b0-da01ff1f3341'; @@ -191,6 +192,12 @@ class OpenAIAssistantSyncer { await Promise.all(toolPromises); } + /** + * Logs a thread and all its messages and runs to the Literal AI API. + * @param threadId The ID of the thread to log to + * @param user A User instance containing the information of the user that participated in the thread + * @param threadMetadata Additional metadata to attach to the thread, in key-value pairs + */ async syncThread( threadId: string, user?: User, diff --git a/src/instrumentation/openai.ts b/src/instrumentation/openai.ts index f5edba6..0541fae 100644 --- a/src/instrumentation/openai.ts +++ b/src/instrumentation/openai.ts @@ -14,9 +14,9 @@ import { IGenerationMessage, LiteralClient, Maybe, - OpenAIGlobalOptions, - StepConstructor + OpenAIGlobalOptions } from '..'; +import { StepConstructor } from '../observability/step'; type OriginalFunction = ( body: any, diff --git a/src/instrumentation/vercel-sdk.ts b/src/instrumentation/vercel-sdk.ts index 58d53bb..c122abb 100644 --- a/src/instrumentation/vercel-sdk.ts +++ b/src/instrumentation/vercel-sdk.ts @@ -16,10 +16,10 @@ import { IGenerationMessage, ILLMSettings, ITool, - LiteralClient, - Step, - Thread + LiteralClient } from '..'; +import { Step } from '../observability/step'; +import { Thread } from '../observability/thread'; export type VercelLanguageModel = LanguageModel; diff --git a/src/observability/attachment.ts b/src/observability/attachment.ts new file mode 100644 index 0000000..9f6492e --- /dev/null +++ b/src/observability/attachment.ts @@ -0,0 +1,24 @@ +import { v4 as uuidv4 } from 'uuid'; + +import { Maybe, OmitUtils, Utils } from '../utils'; + +/** + * Represents an attachment with optional metadata, MIME type, and other properties. + * It extends the `Utils` class for serialization capabilities. + */ +export class Attachment extends Utils { + id?: Maybe; + metadata?: Maybe>; + mime?: Maybe; + name: Maybe; + objectKey?: Maybe; + url?: Maybe; + + constructor(data: OmitUtils) { + super(); + Object.assign(this, data); + if (!this.id) { + this.id = uuidv4(); + } + } +} diff --git a/src/generation.ts b/src/observability/generation.ts similarity index 97% rename from src/generation.ts rename to src/observability/generation.ts index 967527c..07f6a09 100644 --- a/src/generation.ts +++ b/src/observability/generation.ts @@ -1,4 +1,4 @@ -import { Maybe, OmitUtils, Utils } from './types'; +import { Maybe, OmitUtils, Utils } from '../utils'; export type GenerationMessageRole = | 'system' diff --git a/src/observability/step.ts b/src/observability/step.ts new file mode 100644 index 0000000..4dc2d95 --- /dev/null +++ b/src/observability/step.ts @@ -0,0 +1,206 @@ +import { v4 as uuidv4 } from 'uuid'; + +import { LiteralClient } from '..'; +import { API } from '../api'; +import { Score } from '../evaluation/score'; +import { Environment, Maybe, OmitUtils, Utils, isPlainObject } from '../utils'; +import { Attachment } from './attachment'; +import { Generation } from './generation'; + +export type StepType = + | 'assistant_message' + | 'embedding' + | 'llm' + | 'rerank' + | 'retrieval' + | 'run' + | 'system_message' + | 'tool' + | 'undefined' + | 'user_message'; + +class StepFields extends Utils { + name!: string; + type!: StepType; + threadId?: string; + createdAt?: Maybe; + startTime?: Maybe; + id?: Maybe; + environment?: Maybe; + error?: Maybe>; + input?: Maybe>; + output?: Maybe>; + metadata?: Maybe>; + tags?: Maybe; + parentId?: Maybe; + endTime?: Maybe; + generation?: Maybe; + scores?: Maybe; + attachments?: Maybe; +} + +export type StepConstructor = OmitUtils; + +/** + * Represents a step in a process or workflow, extending the fields and methods from StepFields. + */ +export class Step extends StepFields { + api: API; + client: LiteralClient; + + /** + * Constructs a new Step instance. + * @param api The API instance to be used for sending and managing steps. + * @param data The initial data for the step, excluding utility properties. + */ + constructor( + client: LiteralClient, + data: StepConstructor, + ignoreContext?: true + ) { + super(); + this.api = client.api; + this.client = client; + + Object.assign(this, data); + + // Automatically generate an ID if not provided. + if (!this.id) { + this.id = uuidv4(); + } + + if (ignoreContext) { + return; + } + + // Automatically assign parent thread & step if there are any in the store. + this.threadId = this.threadId ?? this.client._currentThread()?.id; + this.parentId = this.parentId ?? this.client._currentStep()?.id; + + // Set the creation and start time to the current time if not provided. + if (!this.createdAt) { + this.createdAt = new Date().toISOString(); + } + if (!this.startTime) { + this.startTime = new Date().toISOString(); + } + + // If the step is a message, set the end time to the start time. + if (this.isMessage()) { + this.endTime = this.startTime; + } + } + + /** + * Serializes the step instance, converting complex objects to strings as necessary. + * @returns A serialized representation of the step. + */ + serialize() { + const serialized = super.serialize(); + + // Convert error objects to JSON string if present. + if (typeof serialized.error === 'object' && serialized.error !== null) { + serialized.error = JSON.stringify(serialized.error); + } + return serialized; + } + + /** + * Determines if the step is a type of message. + * @returns True if the step is a user, assistant, or system message. + */ + isMessage() { + return ( + this.type === 'user_message' || + this.type === 'assistant_message' || + this.type === 'system_message' + ); + } + + /** + * Creates a new step instance linked to the current step as a parent. + * @param data The data for the new step, excluding the threadId which is inherited. + * @returns A new Step instance. + */ + step(data: Omit) { + return new Step(this.client, { + ...data, + threadId: this.threadId, + parentId: this.id + }); + } + + /** + * Sends the step to the API, handling disabled state and setting the end time if not already set. + * @returns The current Step instance after potentially sending to the API. + */ + async send() { + if (this.api.disabled) { + return this; + } + if (!this.endTime) { + this.endTime = new Date().toISOString(); + } + await this.api.sendSteps([this]); + return this; + } + + /** + * Sends the step to the API, handling disabled state and setting the end time if not already set. + * @param cb The callback function to run within the context of the step. + * @param updateStep Optional update function to modify the step after the callback. + * @returns The output of the wrapped callback function. + */ + async wrap( + cb: (step: Step) => Output | Promise, + updateStep?: + | Partial + | ((output: Output) => Partial) + | ((output: Output) => Promise>) + ) { + const startTime = new Date(); + this.startTime = startTime.toISOString(); + const currentStore = this.client.store.getStore(); + + const output = await this.client.store.run( + { + currentThread: currentStore?.currentThread ?? null, + currentExperimentItemRunId: + currentStore?.currentExperimentItemRunId ?? null, + currentStep: this + }, + () => cb(this) + ); + + this.output = isPlainObject(output) ? output : { output }; + this.endTime = new Date().toISOString(); + + if (updateStep) { + const updatedStep = + typeof updateStep === 'function' + ? await updateStep(output) + : updateStep; + + this.name = updatedStep.name ?? this.name; + this.type = updatedStep.type ?? this.type; + this.threadId = updatedStep.threadId ?? this.threadId; + this.createdAt = updatedStep.createdAt ?? this.createdAt; + this.startTime = updatedStep.startTime ?? this.startTime; + this.error = updatedStep.error ?? this.error; + this.input = updatedStep.input ?? this.input; + this.output = updatedStep.output ?? this.output; + this.metadata = updatedStep.metadata ?? this.metadata; + this.tags = updatedStep.tags ?? this.tags; + this.parentId = updatedStep.parentId ?? this.parentId; + this.endTime = updatedStep.endTime ?? this.endTime; + this.generation = updatedStep.generation ?? this.generation; + this.scores = updatedStep.scores ?? this.scores; + this.attachments = updatedStep.attachments ?? this.attachments; + this.environment = updatedStep.environment ?? this.environment; + } + + this.send().catch(console.error); + + return output; + } +} diff --git a/src/observability/thread.ts b/src/observability/thread.ts new file mode 100644 index 0000000..51b37bd --- /dev/null +++ b/src/observability/thread.ts @@ -0,0 +1,134 @@ +import { v4 as uuidv4 } from 'uuid'; + +import { LiteralClient } from '..'; +import { API } from '../api'; +import { Environment, Maybe, OmitUtils, Utils } from '../utils'; +import { Step, StepConstructor } from './step'; + +/** + * Defines the structure for thread fields, inheriting utilities for serialization. + * This class encapsulates common properties used to describe a thread in various environments. + */ +class ThreadFields extends Utils { + id!: string; + participantId?: Maybe; + environment?: Maybe; + name?: Maybe; + metadata?: Maybe>; + steps?: Maybe; + tags?: Maybe; +} + +export type CleanThreadFields = OmitUtils; +export type ThreadConstructor = Omit & + Partial>; + +/** + * Represents a thread in the system, extending the properties and methods from `ThreadFields`. + * This class manages thread-specific operations such as creation and updates via the API. + */ +export class Thread extends ThreadFields { + api: API; + client: LiteralClient; + + /** + * Constructs a new Thread instance, with a generated ID if none is provided. + * @param api - The API instance to interact with backend services. + * @param data - Optional initial data for the thread. + */ + constructor(client: LiteralClient, data?: ThreadConstructor) { + super(); + this.api = client.api; + this.client = client; + + if (!data) { + data = { id: uuidv4() }; + } else if (!data.id) { + data.id = uuidv4(); + } + + Object.assign(this, data); + } + + /** + * Creates a new step associated with this thread. + * @param data - The data for the new step, excluding the thread ID. + * @returns A new Step instance linked to this thread. + */ + step(data: Omit) { + return new Step(this.client, { + ...data, + threadId: this.id + }); + } + + /** + * Creates a new Run step associated with this thread. + * @param data - The data for the new step, excluding the thread ID and the type + * @returns A new Step instance linked to this thread. + */ + run(data: Omit) { + return this.step({ ...data, type: 'run' }); + } + + /** + * Upserts the thread data to the backend, creating or updating as necessary. + * @returns The updated Thread instance. + */ + async upsert() { + if (this.api.disabled) { + return this; + } + await this.api.upsertThread({ + threadId: this.id, + name: this.name, + metadata: this.metadata, + participantId: this.participantId, + tags: this.tags + }); + return this; + } + + /** + * Sends the thread to the API, handling disabled state and setting the end time if not already set. + * @param cb The callback function to run within the context of the thread. + * @param updateThread Optional update function to modify the thread after the callback. + * @returns The output of the wrapped callback function. + */ + async wrap( + cb: (thread: Thread) => Output | Promise, + updateThread?: + | ThreadConstructor + | ((output: Output) => ThreadConstructor) + | ((output: Output) => Promise) + ) { + const currentStore = this.client.store.getStore(); + + const output = await this.client.store.run( + { + currentThread: this, + currentExperimentItemRunId: + currentStore?.currentExperimentItemRunId ?? null, + currentStep: null + }, + () => cb(this) + ); + + if (updateThread) { + const updatedThread = + typeof updateThread === 'function' + ? await updateThread(output) + : updateThread; + + this.participantId = updatedThread.participantId ?? this.participantId; + this.environment = updatedThread.environment ?? this.environment; + this.name = updatedThread.name ?? this.name; + this.metadata = updatedThread.metadata ?? this.metadata; + this.tags = updatedThread.tags ?? this.tags; + } + + this.upsert().catch(console.error); + + return output; + } +} diff --git a/src/prompt-engineering/prompt.ts b/src/prompt-engineering/prompt.ts new file mode 100644 index 0000000..e786e41 --- /dev/null +++ b/src/prompt-engineering/prompt.ts @@ -0,0 +1,141 @@ +import mustache from 'mustache'; +import { + ChatCompletionMessageParam, + ChatCompletionTool +} from 'openai/resources'; + +import { API } from '../api'; +import { DatasetItem } from '../evaluation/dataset'; +import { CustomChatPromptTemplate } from '../instrumentation/langchain'; +import { + GenerationType, + IGenerationMessage +} from '../observability/generation'; +import { Maybe, OmitUtils, Utils } from '../utils'; + +export interface IPromptVariableDefinition { + name: string; + language: 'json' | 'plaintext'; +} + +export interface IProviderSettings { + provider: string; + model: string; + frequency_penalty: number; + max_tokens: number; + presence_penalty: number; + stop?: string[]; + temperature: number; + top_p: number; +} + +class PromptFields extends Utils { + id!: string; + type!: GenerationType; + createdAt!: string; + name!: string; + version!: number; + versionDesc?: Maybe; + metadata!: Record; + items!: Array>; + variablesDefaultValues?: Maybe>; + templateMessages!: IGenerationMessage[]; + tools?: ChatCompletionTool[]; + provider!: string; + settings!: IProviderSettings; + variables!: IPromptVariableDefinition[]; +} + +export type PromptConstructor = OmitUtils; + +export class Prompt extends PromptFields { + api: API; + + /** + * Constructs a new Prompt instance. + * @param api - The API instance to interact with backend services. + * @param data - The initial data for the prompt. + */ + constructor(api: API, data: PromptConstructor) { + super(); + this.api = api; + Object.assign(this, data); + if (this.tools?.length === 0) { + this.tools = undefined; + } + } + + /** + * Formats the prompt's template messages with the given variables. + * @param variables - Optional variables to resolve in the template messages. + * @returns An array of formatted chat completion messages. + */ + formatMessages( + variables?: Record + ): ChatCompletionMessageParam[] { + const variablesWithDefault = { + ...(this.variablesDefaultValues || {}), + ...variables + }; + + const promptId = this.id; + + return this.templateMessages.map( + ({ uuid, templated, ...templateMessage }) => { + const formattedMessage = { + ...templateMessage + } as ChatCompletionMessageParam; + // @ts-expect-error Hacky way to add metadata to the formatted message + formattedMessage.literalMetadata = () => { + return { + uuid: uuid, + promptId, + variables: variablesWithDefault + }; + }; + if (Array.isArray(formattedMessage.content)) { + formattedMessage.content = formattedMessage.content.map((content) => { + if (content.type === 'text') { + return { + ...content, + text: mustache.render(content.text, variablesWithDefault) + }; + } + return content; + }); + } else if (typeof formattedMessage.content === 'string') { + formattedMessage.content = mustache.render( + formattedMessage.content, + variablesWithDefault + ); + } + + return formattedMessage; + } + ); + } + + /** + * @deprecated Please use `formatMessages` instead. + */ + format(variables?: Record): ChatCompletionMessageParam[] { + return this.formatMessages(variables); + } + + /** + * Converts the prompt's template messages into a Langchain chat prompt template. + * @returns A custom chat prompt template configured with the prompt's data. + */ + toLangchainChatPromptTemplate() { + const lcMessages: [string, string][] = this.templateMessages.map((m) => [ + m.role, + m.content as string + ]); + const chatTemplate = CustomChatPromptTemplate.fromMessages(lcMessages); + chatTemplate.variablesDefaultValues = this.variablesDefaultValues; + chatTemplate.literalTemplateMessages = this.templateMessages; + chatTemplate.promptId = this.id; + + return chatTemplate; + } +} diff --git a/src/types.ts b/src/types.ts deleted file mode 100644 index 4dfcb51..0000000 --- a/src/types.ts +++ /dev/null @@ -1,891 +0,0 @@ -import mustache from 'mustache'; -import { - ChatCompletionMessageParam, - ChatCompletionTool -} from 'openai/resources'; -import { v4 as uuidv4 } from 'uuid'; - -import { LiteralClient } from '.'; -import { API } from './api'; -import { Generation, GenerationType, IGenerationMessage } from './generation'; -import { CustomChatPromptTemplate } from './instrumentation/langchain'; - -export type Maybe = T | null | undefined; - -export type OmitUtils = Omit; - -export type Environment = 'dev' | 'staging' | 'prod' | 'experiment'; - -export type PageInfo = { - hasNextPage: boolean; - startCursor: string; - endCursor: string; -}; - -export type PaginatedResponse = { - data: T[]; - pageInfo: PageInfo; -}; - -function isPlainObject(value: unknown): value is Record { - if (typeof value !== 'object' || value === null) { - return false; - } - - const prototype = Object.getPrototypeOf(value); - return prototype === null || prototype === Object.prototype; -} - -/** - * Represents a utility class with serialization capabilities. - */ -export class Utils { - /** - * Serializes the properties of the current instance into a dictionary, excluding the 'api' property. - * It handles nested objects that also implement a serialize method. - * - * @returns A dictionary representing the serialized properties of the object. - */ - serialize(): any { - const dict: any = {}; - Object.keys(this as any).forEach((key) => { - if (['api', 'client'].includes(key)) { - return; - } - if ((this as any)[key] !== undefined) { - if (Array.isArray((this as any)[key])) { - dict[key] = (this as any)[key].map((item: any) => { - if ( - item instanceof Object && - typeof item.serialize === 'function' - ) { - return item.serialize(); - } else { - return item; - } - }); - } else if ( - (this as any)[key] instanceof Object && - typeof (this as any)[key].serialize === 'function' - ) { - dict[key] = (this as any)[key].serialize(); - } else { - dict[key] = (this as any)[key]; - } - } - }); - return dict; - } -} - -export type ScoreType = 'HUMAN' | 'AI'; - -class ScoreFields extends Utils { - id?: Maybe; - stepId?: Maybe; - generationId?: Maybe; - datasetExperimentItemId?: Maybe; - name: string = 'user-feedback'; - value: number = 0; - type: ScoreType = 'AI'; - scorer?: Maybe; - comment?: Maybe; - tags?: Maybe; -} - -export type ScoreConstructor = OmitUtils; - -/** - * Represents a score entity with properties to track various aspects of scoring. - * It extends the `Utils` class for serialization capabilities. - */ -export class Score extends ScoreFields { - constructor(data: ScoreConstructor) { - super(); - Object.assign(this, data); - } -} - -/** - * Represents an attachment with optional metadata, MIME type, and other properties. - * It extends the `Utils` class for serialization capabilities. - */ -export class Attachment extends Utils { - id?: Maybe; - metadata?: Maybe>; - mime?: Maybe; - name: Maybe; - objectKey?: Maybe; - url?: Maybe; - - constructor(data: OmitUtils) { - super(); - Object.assign(this, data); - if (!this.id) { - this.id = uuidv4(); - } - } -} - -/** - * Defines the structure for thread fields, inheriting utilities for serialization. - * This class encapsulates common properties used to describe a thread in various environments. - */ -class ThreadFields extends Utils { - id!: string; - participantId?: Maybe; - environment?: Maybe; - name?: Maybe; - metadata?: Maybe>; - steps?: Maybe; - tags?: Maybe; -} - -export type CleanThreadFields = OmitUtils; -export type ThreadConstructor = Omit & - Partial>; - -/** - * Represents a thread in the system, extending the properties and methods from `ThreadFields`. - * This class manages thread-specific operations such as creation and updates via the API. - */ -export class Thread extends ThreadFields { - api: API; - client: LiteralClient; - - /** - * Constructs a new Thread instance. - * @param api - The API instance to interact with backend services. - * @param data - Optional initial data for the thread, with an auto-generated ID if not provided. - */ - constructor(client: LiteralClient, data?: ThreadConstructor) { - super(); - this.api = client.api; - this.client = client; - - if (!data) { - data = { id: uuidv4() }; - } else if (!data.id) { - data.id = uuidv4(); - } - - Object.assign(this, data); - } - - /** - * Creates a new step associated with this thread. - * @param data - The data for the new step, excluding the thread ID. - * @returns A new Step instance linked to this thread. - */ - step(data: Omit) { - return new Step(this.client, { - ...data, - threadId: this.id - }); - } - - /** - * Creates a new Run step associated with this thread. - * @param data - The data for the new step, excluding the thread ID and the type - * @returns A new Step instance linked to this thread. - */ - run(data: Omit) { - return this.step({ ...data, type: 'run' }); - } - - /** - * Upserts the thread data to the backend, creating or updating as necessary. - * @returns The updated Thread instance. - */ - async upsert() { - if (this.api.disabled) { - return this; - } - await this.api.upsertThread({ - threadId: this.id, - name: this.name, - metadata: this.metadata, - participantId: this.participantId, - tags: this.tags - }); - return this; - } - - /** - * Sends the thread to the API, handling disabled state and setting the end time if not already set. - * @param cb The callback function to run within the context of the thread. - * @param updateThread Optional update function to modify the thread after the callback. - * @returns The output of the wrapped callback function. - */ - async wrap( - cb: (thread: Thread) => Output | Promise, - updateThread?: - | ThreadConstructor - | ((output: Output) => ThreadConstructor) - | ((output: Output) => Promise) - ) { - const currentStore = this.client.store.getStore(); - - const output = await this.client.store.run( - { - currentThread: this, - currentExperimentItemRunId: - currentStore?.currentExperimentItemRunId ?? null, - currentStep: null - }, - () => cb(this) - ); - - if (updateThread) { - const updatedThread = - typeof updateThread === 'function' - ? await updateThread(output) - : updateThread; - - this.participantId = updatedThread.participantId ?? this.participantId; - this.environment = updatedThread.environment ?? this.environment; - this.name = updatedThread.name ?? this.name; - this.metadata = updatedThread.metadata ?? this.metadata; - this.tags = updatedThread.tags ?? this.tags; - } - - this.upsert().catch(console.error); - - return output; - } -} - -export type StepType = - | 'assistant_message' - | 'embedding' - | 'llm' - | 'rerank' - | 'retrieval' - | 'run' - | 'system_message' - | 'tool' - | 'undefined' - | 'user_message'; - -class StepFields extends Utils { - name!: string; - type!: StepType; - threadId?: string; - createdAt?: Maybe; - startTime?: Maybe; - id?: Maybe; - environment?: Maybe; - error?: Maybe>; - input?: Maybe>; - output?: Maybe>; - metadata?: Maybe>; - tags?: Maybe; - parentId?: Maybe; - endTime?: Maybe; - generation?: Maybe; - scores?: Maybe; - attachments?: Maybe; -} - -export type StepConstructor = OmitUtils; - -/** - * Represents a step in a process or workflow, extending the fields and methods from StepFields. - */ -export class Step extends StepFields { - api: API; - client: LiteralClient; - - /** - * Constructs a new Step instance. - * @param api The API instance to be used for sending and managing steps. - * @param data The initial data for the step, excluding utility properties. - */ - constructor( - client: LiteralClient, - data: StepConstructor, - ignoreContext?: true - ) { - super(); - this.api = client.api; - this.client = client; - - Object.assign(this, data); - - // Automatically generate an ID if not provided. - if (!this.id) { - this.id = uuidv4(); - } - - if (ignoreContext) { - return; - } - - // Automatically assign parent thread & step if there are any in the store. - const store = this.client.store.getStore(); - - if (store?.currentThread) { - this.threadId = store.currentThread.id; - } - if (store?.currentStep) { - this.parentId = store.currentStep.id; - } - - // Set the creation and start time to the current time if not provided. - if (!this.createdAt) { - this.createdAt = new Date().toISOString(); - } - if (!this.startTime) { - this.startTime = new Date().toISOString(); - } - - // If the step is a message, set the end time to the start time. - if (this.isMessage()) { - this.endTime = this.startTime; - } - } - - /** - * Serializes the step instance, converting complex objects to strings as necessary. - * @returns A serialized representation of the step. - */ - serialize() { - const serialized = super.serialize(); - - // Convert error objects to JSON string if present. - if (typeof serialized.error === 'object' && serialized.error !== null) { - serialized.error = JSON.stringify(serialized.error); - } - return serialized; - } - - /** - * Determines if the step is a type of message. - * @returns True if the step is a user, assistant, or system message. - */ - isMessage() { - return ( - this.type === 'user_message' || - this.type === 'assistant_message' || - this.type === 'system_message' - ); - } - - /** - * Creates a new step instance linked to the current step as a parent. - * @param data The data for the new step, excluding the threadId which is inherited. - * @returns A new Step instance. - */ - step(data: Omit) { - return new Step(this.client, { - ...data, - threadId: this.threadId, - parentId: this.id - }); - } - - /** - * Sends the step to the API, handling disabled state and setting the end time if not already set. - * @returns The current Step instance after potentially sending to the API. - */ - async send() { - if (this.api.disabled) { - return this; - } - if (!this.endTime) { - this.endTime = new Date().toISOString(); - } - await this.api.sendSteps([this]); - return this; - } - - /** - * Sends the step to the API, handling disabled state and setting the end time if not already set. - * @param cb The callback function to run within the context of the step. - * @param updateStep Optional update function to modify the step after the callback. - * @returns The output of the wrapped callback function. - */ - async wrap( - cb: (step: Step) => Output | Promise, - updateStep?: - | Partial - | ((output: Output) => Partial) - | ((output: Output) => Promise>) - ) { - const startTime = new Date(); - this.startTime = startTime.toISOString(); - const currentStore = this.client.store.getStore(); - - const output = await this.client.store.run( - { - currentThread: currentStore?.currentThread ?? null, - currentExperimentItemRunId: - currentStore?.currentExperimentItemRunId ?? null, - currentStep: this - }, - () => cb(this) - ); - - this.output = isPlainObject(output) ? output : { output }; - this.endTime = new Date().toISOString(); - - if (updateStep) { - const updatedStep = - typeof updateStep === 'function' - ? await updateStep(output) - : updateStep; - - this.name = updatedStep.name ?? this.name; - this.type = updatedStep.type ?? this.type; - this.threadId = updatedStep.threadId ?? this.threadId; - this.createdAt = updatedStep.createdAt ?? this.createdAt; - this.startTime = updatedStep.startTime ?? this.startTime; - this.error = updatedStep.error ?? this.error; - this.input = updatedStep.input ?? this.input; - this.output = updatedStep.output ?? this.output; - this.metadata = updatedStep.metadata ?? this.metadata; - this.tags = updatedStep.tags ?? this.tags; - this.parentId = updatedStep.parentId ?? this.parentId; - this.endTime = updatedStep.endTime ?? this.endTime; - this.generation = updatedStep.generation ?? this.generation; - this.scores = updatedStep.scores ?? this.scores; - this.attachments = updatedStep.attachments ?? this.attachments; - } - - this.send().catch(console.error); - - return output; - } -} - -/** - * Represents an item in an experiment. - */ -export class ExperimentItemRun extends Step { - api: API; - client: LiteralClient; - - /** - * Constructs a new ExperimentItemRun instance. - * @param api The API instance to be used for sending and managing steps. - * @param data The initial data for the step, excluding utility properties. - */ - constructor( - client: LiteralClient, - data: StepConstructor, - ignoreContext?: true - ) { - super(client, data, ignoreContext); - this.client = client; - this.api = client.api; - } - - async wrap( - cb: (step: Step) => Output | Promise, - updateStep?: - | Partial - | ((output: Output) => Partial) - | ((output: Output) => Promise>) - ) { - const originalEnvironment = this.api.environment; - this.api.environment = 'experiment'; - - const currentStore = this.client.store.getStore(); - const output: Output = await this.client.store.run( - { - currentThread: currentStore?.currentThread ?? null, - currentStep: this, - currentExperimentItemRunId: this.id ?? null - }, - async () => { - try { - const output = await super.wrap(cb, updateStep); - return output; - } finally { - // Clear the currentExperimentRunId after execution - const updatedStore = this.client.store.getStore(); - if (updatedStore) { - updatedStore.currentExperimentItemRunId = null; - } - } - } - ); - - this.api.environment = originalEnvironment; - return output; - } -} - -/** - * Represents a user with optional metadata and identifier. - */ -export class User extends Utils { - id?: Maybe; - identifier!: string; - metadata?: Maybe>; - - constructor(data: OmitUtils) { - super(); - Object.assign(this, data); - } -} - -export type DatasetType = 'key_value' | 'generation'; - -class DatasetFields extends Utils { - id!: string; - createdAt!: string; - name?: Maybe; - description?: Maybe; - metadata!: Record; - items!: Array>; - type?: DatasetType; -} - -export type DatasetConstructor = OmitUtils; - -export class Dataset extends DatasetFields { - api: API; - - /** - * Constructs a new Dataset instance. - * @param api - The API instance to interact with backend services. - * @param data - The initial data for the dataset. - */ - constructor(api: API, data: DatasetConstructor) { - super(); - this.api = api; - Object.assign(this, data); - if (!this.items) { - this.items = []; - } - if (!this.type) { - this.type = 'key_value'; - } - } - - /** - * Updates the dataset with new data. - * @param dataset - The dataset data to update. - * @returns The updated dataset instance. - */ - async update(dataset: { - name?: Maybe; - description?: Maybe; - metadata?: Maybe>; - }) { - const update_res = await this.api.updateDataset(this.id, dataset); - this.name = update_res.name; - this.description = update_res.description; - this.metadata = update_res.metadata; - } - - /** - * Deletes the dataset. - * @returns A promise that resolves when the dataset is deleted. - */ - async delete() { - return this.api.deleteDataset(this.id); - } - - /** - * Creates a new item in the dataset. - * @param datasetItem - The new item to be added to the dataset. - * @returns The newly created dataset item. - */ - async createItem(datasetItem: { - input: Record; - expectedOutput?: Maybe>; - metadata?: Maybe>; - }) { - const item = await this.api.createDatasetItem(this.id, datasetItem); - - this.items.push(item); - return item; - } - - /** - * Deletes an item from the dataset. - * @param id - The ID of the item to delete. - * @returns The deleted dataset item. - */ - async deleteItem(id: string) { - const deletedItem = await this.api.deleteDatasetItem(id); - if (this.items) { - this.items = this.items.filter((item) => item.id !== id); - } - return deletedItem; - } - - /** - * Creates a new experiment associated with the dataset. - * @param experiment - The experiment details including name, optional prompt ID, and parameters. - * @returns A new instance of DatasetExperiment containing the created experiment. - */ - async createExperiment(experiment: { - name: string; - promptId?: string; - params?: Record | Array>; - }) { - const datasetExperiment = await this.api.createExperiment({ - name: experiment.name, - datasetId: this.id, - promptId: experiment.promptId, - params: experiment.params - }); - return new DatasetExperiment(this.api, datasetExperiment); - } - - /** - * Adds a step to the dataset. - * @param stepId - The ID of the step to add. - * @param metadata - Optional metadata for the step. - * @returns The added dataset item. - * @throws Error if the dataset type is 'generation'. - */ - public async addStep( - stepId: string, - metadata?: Maybe> - ) { - if (this.type === 'generation') { - throw new Error('Cannot add steps to a generation dataset'); - } - const item = await this.api.addStepToDataset(this.id, stepId, metadata); - this.items.push(item); - return item; - } - - /** - * Adds a generation to the dataset. - * @param generationId - The ID of the generation to add. - * @param metadata - Optional metadata for the generation. - * @returns The added dataset item. - */ - public async addGeneration( - generationId: string, - metadata?: Maybe> - ) { - const item = await this.api.addGenerationToDataset( - this.id, - generationId, - metadata - ); - this.items.push(item); - return item; - } - - public async addGenerations(generationIds?: string[]) { - if (generationIds == undefined || generationIds?.length === 0) { - return []; - } - - const items = await this.api.addGenerationsToDataset( - this.id, - generationIds - ); - this.items = this.items.concat(items); - return items; - } -} - -export class DatasetItem extends Utils { - id!: string; - createdAt!: string; - datasetId!: string; - metadata!: Record; - input!: Record; - expectedOutput?: Maybe>; - intermediarySteps!: Array>; - - constructor(data: OmitUtils) { - super(); - Object.assign(this, data); - } -} - -class DatasetExperimentItemFields extends Utils { - id?: string; - datasetExperimentId!: string; - datasetItemId?: string; - experimentRunId?: string; - scores!: ScoreConstructor[]; - input?: Record; - output?: Record; -} - -export class DatasetExperiment extends Utils { - id!: string; - createdAt!: string; - name!: string; - datasetId?: string; - promptId?: string; - api: API; - params!: Record | Array>; - items!: DatasetExperimentItem[]; - - constructor(api: API, data: OmitUtils) { - super(); - this.api = api; - Object.assign(this, data); - if (!this.items) { - this.items = []; - } - } - - async log( - itemFields: Omit< - OmitUtils, - 'id' | 'datasetExperimentId' - > - ) { - const currentStore = this.api.client.store.getStore(); - const experimentRunId = currentStore?.currentExperimentItemRunId; - - const datasetExperimentItem = new DatasetExperimentItem({ - ...itemFields, - datasetExperimentId: this.id, - ...(experimentRunId && { experimentRunId }) - }); - - const item = await this.api.createExperimentItem(datasetExperimentItem); - - this.items.push(item); - return item; - } -} - -export type DatasetExperimentItemConstructor = - OmitUtils; - -export class DatasetExperimentItem extends DatasetExperimentItemFields { - constructor(data: DatasetExperimentItemConstructor) { - super(); - Object.assign(this, data); - } -} -export interface IPromptVariableDefinition { - name: string; - language: 'json' | 'plaintext'; -} - -export interface IProviderSettings { - provider: string; - model: string; - frequency_penalty: number; - max_tokens: number; - presence_penalty: number; - stop?: string[]; - temperature: number; - top_p: number; -} - -class PromptFields extends Utils { - id!: string; - type!: GenerationType; - createdAt!: string; - name!: string; - version!: number; - versionDesc?: Maybe; - metadata!: Record; - items!: Array>; - variablesDefaultValues?: Maybe>; - templateMessages!: IGenerationMessage[]; - tools?: ChatCompletionTool[]; - provider!: string; - settings!: IProviderSettings; - variables!: IPromptVariableDefinition[]; -} - -export type PromptConstructor = OmitUtils; - -export class Prompt extends PromptFields { - api: API; - - /** - * Constructs a new Prompt instance. - * @param api - The API instance to interact with backend services. - * @param data - The initial data for the prompt. - */ - constructor(api: API, data: PromptConstructor) { - super(); - this.api = api; - Object.assign(this, data); - if (this.tools?.length === 0) { - this.tools = undefined; - } - } - - /** - * Formats the prompt's template messages with the given variables. - * @param variables - Optional variables to resolve in the template messages. - * @returns An array of formatted chat completion messages. - */ - formatMessages( - variables?: Record - ): ChatCompletionMessageParam[] { - const variablesWithDefault = { - ...(this.variablesDefaultValues || {}), - ...variables - }; - - const promptId = this.id; - - return this.templateMessages.map( - ({ uuid, templated, ...templateMessage }) => { - const formattedMessage = { - ...templateMessage - } as ChatCompletionMessageParam; - // @ts-expect-error Hacky way to add metadata to the formatted message - formattedMessage.literalMetadata = () => { - return { - uuid: uuid, - promptId, - variables: variablesWithDefault - }; - }; - if (Array.isArray(formattedMessage.content)) { - formattedMessage.content = formattedMessage.content.map((content) => { - if (content.type === 'text') { - return { - ...content, - text: mustache.render(content.text, variablesWithDefault) - }; - } - return content; - }); - } else if (typeof formattedMessage.content === 'string') { - formattedMessage.content = mustache.render( - formattedMessage.content, - variablesWithDefault - ); - } - - return formattedMessage; - } - ); - } - - /** - * @deprecated Please use `formatMessages` instead. - */ - format(variables?: Record): ChatCompletionMessageParam[] { - return this.formatMessages(variables); - } - - /** - * Converts the prompt's template messages into a Langchain chat prompt template. - * @returns A custom chat prompt template configured with the prompt's data. - */ - toLangchainChatPromptTemplate() { - const lcMessages: [string, string][] = this.templateMessages.map((m) => [ - m.role, - m.content as string - ]); - const chatTemplate = CustomChatPromptTemplate.fromMessages(lcMessages); - chatTemplate.variablesDefaultValues = this.variablesDefaultValues; - chatTemplate.literalTemplateMessages = this.templateMessages; - chatTemplate.promptId = this.id; - - return chatTemplate; - } -} diff --git a/src/utils.ts b/src/utils.ts new file mode 100644 index 0000000..d04e5ff --- /dev/null +++ b/src/utils.ts @@ -0,0 +1,81 @@ +export type Maybe = T | null | undefined; + +export type OmitUtils = Omit; + +export type Environment = 'dev' | 'staging' | 'prod' | 'experiment'; + +export type PageInfo = { + hasNextPage: boolean; + startCursor: string; + endCursor: string; +}; + +export type PaginatedResponse = { + data: T[]; + pageInfo: PageInfo; +}; + +export function isPlainObject(value: unknown): value is Record { + if (typeof value !== 'object' || value === null) { + return false; + } + + const prototype = Object.getPrototypeOf(value); + return prototype === null || prototype === Object.prototype; +} + +/** + * Represents a utility class with serialization capabilities. + */ +export class Utils { + /** + * Serializes the properties of the current instance into a dictionary, excluding the 'api' property. + * It handles nested objects that also implement a serialize method. + * + * @returns A dictionary representing the serialized properties of the object. + */ + serialize(): any { + const dict: any = {}; + Object.keys(this as any).forEach((key) => { + if (['api', 'client'].includes(key)) { + return; + } + if ((this as any)[key] !== undefined) { + if (Array.isArray((this as any)[key])) { + dict[key] = (this as any)[key].map((item: any) => { + if ( + item instanceof Object && + typeof item.serialize === 'function' + ) { + return item.serialize(); + } else { + return item; + } + }); + } else if ( + (this as any)[key] instanceof Object && + typeof (this as any)[key].serialize === 'function' + ) { + dict[key] = (this as any)[key].serialize(); + } else { + dict[key] = (this as any)[key]; + } + } + }); + return dict; + } +} + +/** + * Represents a user with optional metadata and identifier. + */ +export class User extends Utils { + id?: Maybe; + identifier!: string; + metadata?: Maybe>; + + constructor(data: OmitUtils) { + super(); + Object.assign(this, data); + } +} diff --git a/tests/api.test.ts b/tests/api.test.ts index 34a09cb..ff78430 100644 --- a/tests/api.test.ts +++ b/tests/api.test.ts @@ -1,7 +1,9 @@ import 'dotenv/config'; import { v4 as uuidv4 } from 'uuid'; -import { ChatGeneration, Dataset, LiteralClient, Score } from '../src'; +import { ChatGeneration, LiteralClient } from '../src'; +import { Dataset } from '../src/evaluation/dataset'; +import { Score } from '../src/evaluation/score'; describe('End to end tests for the SDK', function () { let client: LiteralClient; @@ -224,6 +226,7 @@ describe('End to end tests for the SDK', function () { it('should test steps', async function () { const thread = await client.thread({ id: uuidv4() }); + const step = await thread .step({ name: 'test', @@ -232,9 +235,7 @@ describe('End to end tests for the SDK', function () { }) .send(); - if (!step.id) { - throw new Error('Step id is null'); - } + expect(step.id).not.toBeNull(); await new Promise((resolve) => setTimeout(resolve, 1000)); @@ -243,7 +244,7 @@ describe('End to end tests for the SDK', function () { { field: 'id', operator: 'eq', - value: step.id + value: step.id! }, { field: 'tags', @@ -507,7 +508,7 @@ describe('End to end tests for the SDK', function () { }); it('should add a step to a dataset', async () => { - const thread = await client.thread({ id: uuidv4() }).upsert(); + const thread = await client.thread({ id: uuidv4() }); const step = await thread .step({ name: 'Run', @@ -526,6 +527,8 @@ describe('End to end tests for the SDK', function () { }) .send(); + await new Promise((resolve) => setTimeout(resolve, 1000)); + const datasetItem = await dataset.addStep(step.id!); expect(datasetItem.id).not.toBeNull(); diff --git a/tests/attachments.test.ts b/tests/attachments.test.ts index 514f4e4..ccb277d 100644 --- a/tests/attachments.test.ts +++ b/tests/attachments.test.ts @@ -1,7 +1,8 @@ import 'dotenv/config'; import { createReadStream, readFileSync } from 'fs'; -import { Attachment, LiteralClient, Maybe } from '../src'; +import { LiteralClient, Maybe } from '../src'; +import { Attachment } from '../src/observability/attachment'; const apiUrl = process.env.LITERAL_API_URL; const apiKey = process.env.LITERAL_API_KEY; diff --git a/tests/integration/openai.test.ts b/tests/integration/openai.test.ts index 26dcff1..a8954c9 100644 --- a/tests/integration/openai.test.ts +++ b/tests/integration/openai.test.ts @@ -3,13 +3,8 @@ import OpenAI from 'openai'; import { PassThrough } from 'stream'; import { v4 as uuidv4 } from 'uuid'; -import { - ChatGeneration, - LiteralClient, - Maybe, - OmitUtils, - Step -} from '../../src'; +import { ChatGeneration, LiteralClient, Maybe, OmitUtils } from '../../src'; +import { Step } from '../../src/observability/step'; const apiUrl = process.env.LITERAL_API_URL; const apiKey = process.env.LITERAL_API_KEY; diff --git a/tests/wrappers.test.ts b/tests/wrappers.test.ts index b52fae7..7a961df 100644 --- a/tests/wrappers.test.ts +++ b/tests/wrappers.test.ts @@ -1,6 +1,8 @@ import 'dotenv/config'; -import { DatasetExperimentItem, LiteralClient, Maybe, Step } from '../src'; +import { LiteralClient, Maybe } from '../src'; +import { DatasetExperimentItem } from '../src/evaluation/dataset'; +import { Step } from '../src/observability/step'; const url = process.env.LITERAL_API_URL; const apiKey = process.env.LITERAL_API_KEY;