⚡Perrformance test: this Context-enrichment is right or not?

[zk1tty]

Goal

Compare the key factors of automation of browser experiences.

indicator	our assumption	reality
Speed	20x faster	?
Accuracy	10%+ success rate of workflow	?
Workflow optimization	??	??
Visibility	??	??
development time	??	??

Preparation

• Step 1: Browser action history file with Intent annotation

1. Add time stamp to transcript
2. merge transcript to a user-input history file based on timestamp

transcript

{
  entries: [
    { t: 15000, text: "Click login", final: true },
    { t: 23000, text: "Enter my email", final: true }
  ],
  startedAtMs: 1000000,  // Absolute starting timestamp
  partial: ""
}

rrweb recorder data

{
  steps: [
    { type: "click", timestamp: 1016000, ... }, // Timestamps are absolute in transcript
    { type: "input", timestamp: 1023500, ... }
  ]
}

• Step 2: Compiler to convert history file to optimized workflows

Compiler:

• input: user-input history + intent(transcript) for each steps

• Process:

  • 1. Correlate Voice with Actions

• output: optimized workflows run by browser-use agent[actions + tools] + direct playwright(patchwright)

  • is an optimized workflow a subset of [actions + tools]?
  • how to design data structure:
    • the list of steps with user-intent
    • what is the list of user-intent?

• Step 3: Measure the indicator's performance

Components