Conversational puppets
See example at https://makerlab.github.io/puppet/
gh repo clone makerlab/puppet
cd puppet
npm install
npm run build
npm install tiny-server
tiny-server
Voice input
Note you will need a powerful laptop/rig to run this demo with any reasonable performance (circa 2024).
In the first version of the conversational puppet framework we want to:
- be able to handle user input using a text chat window with limited history
- pass user prompts to an llm and get back a response
- break down the response into "breath" segments of about a single breath each
- have responses be visible to the user
- allow interruptions and stopping of the llm at any time
- display state of when the llm is thinking or speaking versus when it is ready for input
- disambiguate bot 'actions' (smiling, gesturing, pointing) from speaking
We also have to establish a few conventions:
-
Bundling ... ESBuild will do for now although it doesn't have introspection inside dynamically fetched imports so we have to go and specify imports (see 'npm run build and esbuild-make.mjs). The main reason to use esbuild instead of Vite is that Vite has opaque configuration options such as for github pages. See https://esbuild.github.io/ .
-
Bring up web-llm ... Our aspirations are simply to bring up one of their demos and see how it well it works. Older hardware may be challenged. See https://github.com/mlc-ai/web-llm/
-
Bring in Aider ... Aider has become a very helpful tool for scaffolding apps quickly and conversationally. All of the HTML for this app was built by Aider. See https://aider.chat/
-
Stage on github and have a static github page. For now we will have the /index.html point to the dist folder generated by esbuild.
I do notice a few issues:
- it is reporting ready too soon
- use service workers
- input box was not foregrounded?