Use langgraph (agent) to call tools

[brichet]

This PR adds langgraph, which allows to call tools that can interact with Jupyterlab API.

As an example, the PR includes a default tools, that only shows a Jupyterlab modal with the response from the model.

UPDATED RECORDING

output.webm

record-2025-06-26_17.23.44.webm

Current state

• add a setting to enable the use of tool (only a test tool available as of now)
• create an agent (LLM + tool) using langgraph, and call it instead of only the LLM
• the LLM decides by itself if it should call a tool or not. The PR updates the system prompt to require calling it, for now.
• add a message for each message received from the LLM (user is Jupyternaut) and from the tool (user is the tool name)

To do, To discuss

• add a registry (with a token) to enable additional tools from extensions
• @jtpio suggested to switch from LLM to agent from the chat panel itself (instead of the settings). If we do that, we should probably be able to select the tool(s) to use from the chat panel too.
• currently there is an additional instruction in the system prompt to request the use of the tool. We should explore if there is an option when binding the tool to the model, if we want to force the use of it.
• we should probably pass only a subset of the agent for security reason (as we did for chat model and completer model in Do not expose providers api #84)

[brichet]

I open this PR to move forward, and to allow working on useful tools in a separate PR.
When we have some tools that are useful, we'll probably want to remove the current test tool.

I also wonder if should allow disabling some tools from the settings (in future PR maybe), to avoid displaying them in the list, if a lot are provided by extensions.

[jtpio]

Thanks! I'll have a look locally.

I also wonder if should allow disabling some tools from the settings (in future PR maybe), to avoid displaying them in the list, if a lot are provided by extensions.

After looking at the screencast, could the list of tools allow for multiple selection? That way users could more easily choose which tools to enable and disable (all could be enabled by default for convenience).

[brichet]

After looking at the screencast, could the list of tools allow for multiple selection?

Currently no, I take a look at it.

[jtpio]

Also wondering what the granularity of tools should be. Maybe it would result in too many entries if each command can become a tool. Instead higher level tools like editNotebook could make use of multiple JupyterLab commands to perform actions (add a new cell, replace content, show diff).

[jtpio]

When we have some tools that are useful, we'll probably want to remove the current test tool.

As an alternative to the test tool, maybe we could add a first tool to generate a new notebook? (similar to what Jupyter AI currently provides). Although this tool may be replaced later with a more generic one (for example editFiles or editNotebooks).

The tool would make use of existing commands to create the notebook with cells. Although not sure how much work this would require, I think Jupyter AI uses a special prompt to make sure the notebook is well-formed. Otherwise maybe we can have another simpler default tool, that would still perform some useful action.

[brichet]

Also wondering what the granularity of tools should be. Maybe it would result in too many entries if each command can become a tool. Instead higher level tools like editNotebook could make use of multiple JupyterLab commands to perform actions (add a new cell, replace content, show diff).

That's what I thought too, tools handling several scopes of actions. Indeed it would not be usable if you have to allow each command.

[brichet]

The tool would make use of existing commands to create the notebook with cells.

I updated the default tool.

output.webm

Currently it can only use one command: notebook:create-new.
I tried to allow it to use any command with args to create a Notebook, rename it, and add some cells with content.
Although the commands and args seems to be correct, it doesn't work well because of popup requiring async action from user:

• creating a notebook require a kernel selection (or we should provide the existing kernel to the LLM for it to provide it in the args)
• Rename command doesn't expect args, it only open a popup to rename the file

To make it work properly, we should probably have a subset of commands that do not open popups and select default values instead.

[jtpio]

Nice, this is already looking really good!

Although the commands and args seems to be correct, it doesn't work well because of popup requiring async action from user:

Normally there is a set of args we should be able to pass to avoid the dialogs (for example using python as the kernel preference).

Rename command doesn't expect args, it only open a popup to rename the file

Maybe this could also be a good opportunity to improve some of the core commands (the rename command could take an optional argument).

[jtpio]

Maybe we could pass the kernelName with something like the following:

Suggested change

	export const createNotebook = (
	commands: CommandRegistry
	): StructuredToolInterface => {
	return tool(
	async ({ command, args }) => {
	let result: any = 'No command called';
	if (command === 'notebook:create-new') {
	result = await commands.execute(
	command,
	args as ReadonlyPartialJSONObject
	);
	}
	const output = `
	The test tool has been called, with the following query: "${command}"
	The args for the commands where ${JSON.stringify(args)}
	The result of the command (if called) is "${result}"
	`;
	return output;
	},
	{
	name: 'createNotebook',
	description: 'Run jupyterlab command to create a notebook',
	schema: z.object({
	command: z.string().describe('The Jupyterlab command id to execute'),
	args: z
	.object({})
	.passthrough()
	.describe('The argument for the command')
	})
	}
	);
	};
	export const createNotebook = (
	commands: CommandRegistry
	): StructuredToolInterface => {
	return tool(
	async ({ command, args, kernelName }) => {
	let result: any = 'No command called';
	if (command === 'notebook:create-new') {
	// Create args object with kernelName if provided
	const commandArgs: ReadonlyPartialJSONObject = {
	...args,
	...(kernelName && { kernelName })
	};
	result = await commands.execute(command, commandArgs);
	}
	const output = `
	The test tool has been called, with the following query: "${command}"
	The args for the commands where ${JSON.stringify(args)}
	${kernelName ? `The kernel name specified: "${kernelName}"` : ''}
	The result of the command (if called) is "${result}"
	`;
	return output;
	},
	{
	name: 'createNotebook',
	description: 'Run jupyterlab command to create a notebook',
	schema: z.object({
	command: z.string().describe('The Jupyterlab command id to execute'),
	args: z
	.object({})
	.passthrough()
	.describe('The argument for the command'),
	kernelName: z
	.string()
	.optional()
	.default('python3')
	.describe(
	'The name of the kernel to use for the notebook. If not specified, will use the default kernel.'
	)
	})
	}
	);
	};

[jtpio]

Although instead of hardcoding python3 we could also get the default kernel name from app.serviceManager.kernelspecs

[jtpio]

The extension seems to be building fine locally, although it leaves some kind of error / warnings in the logs:

WARNING in ./node_modules/@langchain/langgraph/dist/pregel/validate.js
Invalid dependencies have been reported by plugins or loaders for this module. All reported dependencies need to be absolute paths.
Invalid dependencies may lead to broken watching and caching.
As best effort we try to convert all invalid values to absolute paths and converting globs into context dependencies, but this is deprecated behavior.
Loaders: Pass absolute paths to this.addDependency (existing files), this.addMissingDependency (not existing files), and this.addContextDependency (directories).
Plugins: Pass absolute paths to fileDependencies (existing files), missingDependencies (not existing files), and contextDependencies (directories).
Globs: They are not supported. Pass absolute path to the directory as context dependencies.
The following invalid values have been reported:
 * "../../src/pregel/validate.ts"
 @ ./node_modules/@langchain/langgraph/dist/pregel/index.js 20:0-60 439:8-21 1149:12-24 1156:12-24
 @ ./node_modules/@langchain/langgraph/dist/graph/graph.js 7:0-53 329:35-41 368:32-51
 @ ./node_modules/@langchain/langgraph/dist/graph/state.js 5:0-59 82:32-37 470:40-53 777:15-21
 @ ./node_modules/@langchain/langgraph/dist/prebuilt/agent_executor.js 2:0-47 39:25-35
 @ ./node_modules/@langchain/langgraph/dist/prebuilt/index.js 1:0-59 1:0-59
 @ ./node_modules/@langchain/langgraph/prebuilt.js 1:0-40 1:0-40
 @ ./lib/provider.js 1:0-65 329:29-45
 @ ./lib/index.js 14:0-48 138:37-55
 @ container entry ./extension[0] ./index[0]

[jtpio]

Maybe we'll need to show a message somewhere that the selected provider does not support tool calling, or disable the functionality automatically if it can be detected.

For example when trying to use tools with ChromeAI:

[brichet]

Thanks, I'm looking into this.

[brichet]

I updated the PR.

• the allowTools setting is now in the provider registry settings instead of chat settings, for consistency (the agent is built there).
• the set tools button is now disabled if the provider cannot handle tools, or if the provider is not set (e.g. no APIkey).

Unfortunately, it seems that it does not work at the model level. The tools are still available for example with Claude 2.1. When sending a message, we receive an error message ('claude-2.1' does not support tool use via API fields.)

[brichet]

The extension seems to be building fine locally, although it leaves some kind of error / warnings in the logs:

Yes, I saw it, it seems to be a issue with the packaging of @langchain/langgraph.

Maybe we can handle these warnings with a webpack setting...

[jtpio]

Maybe we can handle these warnings with a webpack setting...

Yeah providing a custom webpack config to ignore these warnings seem to do the trick for now: brichet#2

[jtpio]

Thanks @brichet!

Just tried quickly again with the latest state and it looks like a good start 👍

Spotted a few small things. The tool greyed out icon looks like the feature is not enabled and would not allow selecting tools, although it does. Maybe it should have the same color as the send button?

jupyterlite-ai-tool-icon.mp4

Ideally we should be able to see the list of arguments instead of [object Object]:

And there is likely something to do with the message history too, as the model seems to be taking into account previous requests:

But I guess it would be ok to fix that separately too.

[brichet]

Maybe it should have the same color as the send button?

The idea was to easily catch if a tool is set or not (blue if tools are selected, grey otherwise).
We could maybe use another color when the tool are not used (light blue for example).

And there is likely something to do with the message history too

I wonder if it would help to update the prompt with something like:
"Only the last message should be taken into account, the others are only history for context."

[brichet]

Trying to rebase to resolve the conflicts, I can't build anymore:

node_modules/@langchain/langgraph/dist/graph/messages_annotation.d.ts:82:15 - error TS2589: Type instantiation is excessively deep and possibly infinite.

82     messages: import("./zod/meta.js").ReducedZodChannel<z.ZodType<BaseMessage[], z.ZodTypeDef, BaseMessage[]>, import("@langchain/core/utils/types").InteropZodType<Messages>>;
                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

src/tools/create-notebook.ts:9:3 - error TS2589: Type instantiation is excessively deep and possibly infinite.

9   return tool(
    ~~~~~~

src/tools/create-notebook.ts:9:10 - error TS2589: Type instantiation is excessively deep and possibly infinite.

  9   return tool(
             ~~~~~
 10     async ({ command, args }) => {
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
... 
 35     }
    ~~~~~
 36   );
    ~~~


Found 3 errors in 2 files.

Errors  Files
     1  node_modules/@langchain/langgraph/dist/graph/messages_annotation.d.ts:82
     2  src/tools/create-notebook.ts:9

Restoring zod: 3.25.67 (version before #117) fixes it locally.