Feat: real-time Voice Input + Transcript on Rebrowse Recorder

[zk1tty]

Problem

Analyzing the intent behind the uesr's input is very hard. since it's hard, the time to text and monitor the initial workflows take more and more time. Currently I made a recorder to capture user's behavior(mouse input, keyboard input, user's click). But still no way to capture the intent of user's behavior.

Solution

I want to add voice recorder or voice agents to extract user's intention very well. So users can tell what they're looking for very comfortably, and it reduces the development/testing time for making the deterministic workflows.

Options

1. Voice memo during recording → transcribe on stop
2. Live caption while recording
3. Realtime conversation with an Voice AI

Current Recording Pipeline

1. Sidepanel UI

• User clicks “Start Recording” in InitialView → startRecording() sends START_RECORDING.
• User clicks “Stop Recording” in RecordingView → stopRecording() sends STOP_RECORDING.

2. Background (service worker)

• On START_RECORDING: clears sessionLogs/tabInfo, sets isRecordingEnabled=true, broadcasts SET_RECORDING_STATUS to tabs and recording_status_updated to sidepanel.
• On STOP_RECORDING: sets isRecordingEnabled=false, broadcasts status (and notifies server with RECORDING_STOPPED).
• On events from content: receives RRWEB_EVENT and CUSTOM_* events, attaches screenshots, stores in sessionLogs[tabId], rebuilds Workflow via convertStoredEventsToSteps, and exposes it via GET_RECORDING_DATA.

3. Content script (per tab)

• On SET_RECORDING_STATUS=true: starts rrweb recorder + custom listeners; emits RRWEB_EVENT, CUSTOM_CLICK_EVENT, CUSTOM_INPUT_EVENT, CUSTOM_KEY_EVENT, etc.
• On SET_RECORDING_STATUS=false: stops recorder and removes listeners.

4. Sidepanel data flow

• Polls GET_RECORDING_DATA during recording; listens to recording_status_updated. Renders EventViewer from workflow.steps. Transitions to StoppedView after stop for upload.

Must-have UI

• We can safely add a Mic toggle in InitialView (beside “Start Recording”)

• show a transcript panel in RecordingView's bottom;

• transcript data can live in sidepanel state and be uploaded alongside, or after workflow.

• where to store transcript upding recording: in-memory state mirrored to localStorage keyed by runId
  -where to store transcript after recording: backend writes JSON as transcript file (keep it seperate from rrweb recording data)

• AssemblyAI credential management: okay for dev(which I need by tomorrow), think apter about production

[zk1tty]

Current Implementation

Sample transcript file:

Can you hear me?
Perfect.
Perfect.
Okay, let's get. Start recording.
So I'm going to product hunt.
And I will look for today's launch.
And then first one is this one.
And when I come to the main page of the first product launching today and I will take a look, team.
And I will find CEO.
And I will copy and paste this seagull as name to upload.
And I will. Copen and mcgon name here.
And find the same name.
Of today's launch product.
I think.
What's the name? What's the name of product?
No. Okay.

Next step

• Mix permission checker:
  How it works:
  1. First tries the Permissions API (navigator.permissions.query())
  2. Falls back to getUserMedia() if Permissions API is unavailable
  3. Returns 'unknown' if both methods fail