Skip to content

feat: allow training multiple capture actions in one recording session #562

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 58 commits into from
Apr 30, 2025
Merged
Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
58 commits
Select commit Hold shift + click to select a range
7fe982a
feat: allow multiple capture action execution
RohitR311 Apr 27, 2025
c6266fd
feat: emit action type
RohitR311 Apr 27, 2025
cd4820f
feat: serialize data by action type
RohitR311 Apr 27, 2025
bd5087e
feat: run and abort categorize data by action type
RohitR311 Apr 27, 2025
35e7778
feat: schedule categorize data by action type
RohitR311 Apr 27, 2025
00ef3ba
feat: record api categorize data by action type
RohitR311 Apr 27, 2025
6243563
feat: track browser actions state
RohitR311 Apr 27, 2025
6376fd6
feat: reset interpret log list screenshot
RohitR311 Apr 28, 2025
b5c5ed7
feat: check action exists in workflow
RohitR311 Apr 28, 2025
42b56d7
feat: emit recording editor actions by type
RohitR311 Apr 28, 2025
82d6f70
feat: revamp output preview log ui
RohitR311 Apr 28, 2025
a00e69e
feat: change scrape schema merge logic
RohitR311 Apr 28, 2025
a7771cf
feat: memoize handle url change
RohitR311 Apr 28, 2025
f975862
feat: revamp run content ui
RohitR311 Apr 28, 2025
8d06146
feat: add translations for run content
RohitR311 Apr 28, 2025
4ed3160
feat: emit socket events for stopping
RohitR311 Apr 28, 2025
2ffbdc7
feat: revamp gsheet integration multiple actions
RohitR311 Apr 29, 2025
109afff
feat: revamp airtable integration multiple actions
RohitR311 Apr 29, 2025
882b25c
feat: correct key used when checking unconfirmed list fields
RohitR311 Apr 29, 2025
01ab958
feat: replace banned Function type with an explicit signature
RohitR311 Apr 29, 2025
c7e3a66
feat: maxlen 0 if field not exist
RohitR311 Apr 29, 2025
f1d0cbd
feat: rm other actions logic
RohitR311 Apr 30, 2025
b4e3ccd
Merge branch 'develop' into all-record
amhsirak Apr 30, 2025
f1c1488
fix: lint
amhsirak Apr 30, 2025
0c5e98c
fix: lint
amhsirak Apr 30, 2025
d0f284c
chore: -rm unused imports
amhsirak Apr 30, 2025
f65dda0
feat: -rm download all json
amhsirak Apr 30, 2025
302ec00
feat: -rm horizontal view
amhsirak Apr 30, 2025
e5f63be
feat: -rm vertical view
amhsirak Apr 30, 2025
9eeb367
feat: -rm box
amhsirak Apr 30, 2025
458392d
feat: -rm horizontal view from screenshots
amhsirak Apr 30, 2025
db890e0
feat: -rm vertical view from screenshots
amhsirak Apr 30, 2025
e6a7fdf
feat: -rm box
amhsirak Apr 30, 2025
2b04634
feat: -rm unused icons
amhsirak Apr 30, 2025
c202a50
feat: -rm icons for capture text and list
amhsirak Apr 30, 2025
cf5be61
feat: -rm icons
amhsirak Apr 30, 2025
47ed5ce
feat: -rm icons
amhsirak Apr 30, 2025
7c7116a
feat: -rm captured data
amhsirak Apr 30, 2025
1f06bcd
feat: -rm captured screenshots
amhsirak Apr 30, 2025
392a5fd
feat: rm workflow in progress logic
RohitR311 Apr 30, 2025
624d7fc
Merge branch 'all-record' of https://github.com/getmaxun/maxun into a…
RohitR311 Apr 30, 2025
9bc9815
feat: -rm chips
amhsirak Apr 30, 2025
94fecc1
feat: -rm download all json
amhsirak Apr 30, 2025
9c57824
feat: -rm download all json
amhsirak Apr 30, 2025
decf14a
fix: cleanup
amhsirak Apr 30, 2025
b72d0dc
chore: remove unused import
amhsirak Apr 30, 2025
43b7a7d
feat: rm view mode logic
RohitR311 Apr 30, 2025
02a150d
feat: paginate capture screenshots
RohitR311 Apr 30, 2025
b83fcb2
feat: rm left space
RohitR311 Apr 30, 2025
db25627
feat: change translations
RohitR311 Apr 30, 2025
5cd756c
feat: rm card componenent
RohitR311 Apr 30, 2025
3b618d8
feat: rm screenshot items chip
RohitR311 Apr 30, 2025
6419d31
feat: buttons ui change, rm render expand
RohitR311 Apr 30, 2025
daa9779
feat: rm unnecessray imports
RohitR311 Apr 30, 2025
eed8ff3
Merge pull request #574 from getmaxun/all-record-ui
RohitR311 Apr 30, 2025
7b6de7d
feat: fix download ss logic
RohitR311 Apr 30, 2025
f5df4c9
feat: rm other page and data
RohitR311 Apr 30, 2025
e09e794
feat: show tabs only if multiple actions
RohitR311 Apr 30, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 41 additions & 0 deletions maxun-core/src/interpret.ts
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@ interface InterpreterOptions {
debugChannel: Partial<{
activeId: Function,
debugMessage: Function,
setActionType: Function,
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Verification agent

🧩 Analysis chain

Verify that the UI correctly handles action state throughout the capture flow

These changes support the action state management in the UI layer. Let's ensure that the setActionType callback integrates correctly with the UI's action lifecycle management.

Also applies to: 381-383, 391-393, 421-423, 432-434, 478-480, 493-495, 507-509, 520-522, 532-534, 566-568


🏁 Script executed:

#!/bin/bash
# Check how the UI components use the new setActionType callback

# Find the implementation of the action state management in the UI
rg -A 3 -B 3 "startAction|finishAction" --type=tsx

# Check for imports or usage of the setActionType in React components
rg "setActionType" --type=tsx

Length of output: 138


🏁 Script executed:

#!/bin/bash
# Correct search for UI usage of action lifecycle callbacks in TSX/TS files

# Search for startAction in TSX files
rg -n 'startAction' -g '*.tsx'

# Search for finishAction in TSX files
rg -n 'finishAction' -g '*.tsx'

# Search for setActionType in TSX files
rg -n 'setActionType' -g '*.tsx'

# Also check for setActionType usage in plain TS files, in case UI logic is separated
rg -n 'setActionType' -g '*.ts'

Length of output: 2760


Ensure the UI wires up the new setActionType callback

We found calls to debugChannel.setActionType in maxun-core/src/interpret.ts but no corresponding handling in the UI. The recorder panel only subscribes to startAction/finishAction, so your new action-type updates won’t propagate unless you explicitly pass and handle setActionType in your React components.

Locations to update:

  • src/components/recorder/RightSidePanel.tsx
    • Include setActionType in the debug channel props alongside startAction/finishAction.
    • Add logic (e.g., a state setter or effect) to respond to setActionType(type) calls and update the UI accordingly.

Without these changes, the UI won’t reflect the action‐type changes emitted by interpret.ts.

🧰 Tools
🪛 Biome (1.9.4)

[error] 48-48: Don't use 'Function' as a type.

Prefer explicitly define the function shape. This type accepts any function-like value, which can be a common source of bugs.

(lint/complexity/noBannedTypes)

}>
}

Expand Down Expand Up @@ -377,12 +378,20 @@ export default class Interpreter extends EventEmitter {
*/
const wawActions: Record<CustomFunctions, (...args: any[]) => void> = {
screenshot: async (params: PageScreenshotOptions) => {
if (this.options.debugChannel?.setActionType) {
this.options.debugChannel.setActionType('screenshot');
}

const screenshotBuffer = await page.screenshot({
...params, path: undefined,
});
await this.options.binaryCallback(screenshotBuffer, 'image/png');
},
enqueueLinks: async (selector: string) => {
if (this.options.debugChannel?.setActionType) {
this.options.debugChannel.setActionType('enqueueLinks');
}

const links: string[] = await page.locator(selector)
.evaluateAll(
// @ts-ignore
Expand All @@ -409,13 +418,21 @@ export default class Interpreter extends EventEmitter {
await page.close();
},
scrape: async (selector?: string) => {
if (this.options.debugChannel?.setActionType) {
this.options.debugChannel.setActionType('scrape');
}

await this.ensureScriptsLoaded(page);

const scrapeResults: Record<string, string>[] = await page.evaluate((s) => window.scrape(s ?? null), selector);
await this.options.serializableCallback(scrapeResults);
},

scrapeSchema: async (schema: Record<string, { selector: string; tag: string, attribute: string; shadow: string}>) => {
if (this.options.debugChannel?.setActionType) {
this.options.debugChannel.setActionType('scrapeSchema');
}

await this.ensureScriptsLoaded(page);

const scrapeResult = await page.evaluate((schemaObj) => window.scrapeSchema(schemaObj), schema);
Expand Down Expand Up @@ -458,6 +475,10 @@ export default class Interpreter extends EventEmitter {
},

scrapeList: async (config: { listSelector: string, fields: any, limit?: number, pagination: any }) => {
if (this.options.debugChannel?.setActionType) {
this.options.debugChannel.setActionType('scrapeList');
}

await this.ensureScriptsLoaded(page);
if (!config.pagination) {
const scrapeResults: Record<string, any>[] = await page.evaluate((cfg) => window.scrapeList(cfg), config);
Expand All @@ -469,6 +490,10 @@ export default class Interpreter extends EventEmitter {
},

scrapeListAuto: async (config: { listSelector: string }) => {
if (this.options.debugChannel?.setActionType) {
this.options.debugChannel.setActionType('scrapeListAuto');
}

await this.ensureScriptsLoaded(page);

const scrapeResults: { selector: string, innerText: string }[] = await page.evaluate((listSelector) => {
Expand All @@ -479,6 +504,10 @@ export default class Interpreter extends EventEmitter {
},

scroll: async (pages?: number) => {
if (this.options.debugChannel?.setActionType) {
this.options.debugChannel.setActionType('scroll');
}

await page.evaluate(async (pagesInternal) => {
for (let i = 1; i <= (pagesInternal ?? 1); i += 1) {
// @ts-ignore
Expand All @@ -488,6 +517,10 @@ export default class Interpreter extends EventEmitter {
},

script: async (code: string) => {
if (this.options.debugChannel?.setActionType) {
this.options.debugChannel.setActionType('script');
}

const AsyncFunction: FunctionConstructor = Object.getPrototypeOf(
async () => { },
).constructor;
Expand All @@ -496,6 +529,10 @@ export default class Interpreter extends EventEmitter {
},

flag: async () => new Promise((res) => {
if (this.options.debugChannel?.setActionType) {
this.options.debugChannel.setActionType('flag');
}

this.emit('flag', page, res);
}),
};
Expand Down Expand Up @@ -526,6 +563,10 @@ export default class Interpreter extends EventEmitter {
const params = !step.args || Array.isArray(step.args) ? step.args : [step.args];
await wawActions[step.action as CustomFunctions](...(params ?? []));
} else {
if (this.options.debugChannel?.setActionType) {
this.options.debugChannel.setActionType(String(step.action));
}

// Implements the dot notation for the "method name" in the workflow
const levels = String(step.action).split('.');
const methodName = levels[levels.length - 1];
Expand Down
12 changes: 11 additions & 1 deletion server/src/api/record.ts
Original file line number Diff line number Diff line change
Expand Up @@ -586,6 +586,12 @@ async function executeRun(id: string, userId: string) {
const binaryOutputService = new BinaryOutputService('maxun-run-screenshots');
const uploadedBinaryOutput = await binaryOutputService.uploadAndStoreBinaryOutput(run, interpretationInfo.binaryOutput);

const categorizedOutput = {
scrapeSchema: interpretationInfo.scrapeSchemaOutput || {},
scrapeList: interpretationInfo.scrapeListOutput || {},
other: interpretationInfo.otherOutput || {}
};

await destroyRemoteBrowser(plainRun.browserId, userId);

const updatedRun = await run.update({
Expand All @@ -594,7 +600,11 @@ async function executeRun(id: string, userId: string) {
finishedAt: new Date().toLocaleString(),
browserId: plainRun.browserId,
log: interpretationInfo.log.join('\n'),
serializableOutput: interpretationInfo.serializableOutput,
serializableOutput: {
scrapeSchema: Object.values(categorizedOutput.scrapeSchema),
scrapeList: Object.values(categorizedOutput.scrapeList),
other: Object.values(categorizedOutput.other),
},
binaryOutput: uploadedBinaryOutput,
});

Expand Down
111 changes: 82 additions & 29 deletions server/src/pgboss-worker.ts
Original file line number Diff line number Diff line change
Expand Up @@ -255,7 +255,6 @@ async function processRunExecution(job: Job<ExecuteRunData>) {
return { success: true };
}

// Process the results
const binaryOutputService = new BinaryOutputService('maxun-run-screenshots');
const uploadedBinaryOutput = await binaryOutputService.uploadAndStoreBinaryOutput(run, interpretationInfo.binaryOutput);

Expand All @@ -264,36 +263,57 @@ async function processRunExecution(job: Job<ExecuteRunData>) {
return { success: true };
}

// Update the run record with results
const categorizedOutput = {
scrapeSchema: interpretationInfo.scrapeSchemaOutput || {},
scrapeList: interpretationInfo.scrapeListOutput || {},
other: interpretationInfo.otherOutput || {}
};

await run.update({
...run,
status: 'success',
finishedAt: new Date().toLocaleString(),
browserId: plainRun.browserId,
log: interpretationInfo.log.join('\n'),
serializableOutput: interpretationInfo.serializableOutput,
serializableOutput: {
scrapeSchema: Object.values(categorizedOutput.scrapeSchema),
scrapeList: Object.values(categorizedOutput.scrapeList),
other: Object.values(categorizedOutput.other),
},
binaryOutput: uploadedBinaryOutput,
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Verification agent

🧩 Analysis chain

Storing Object.values(...) drops the item-index mapping

collectDataByType builds an object keyed as item-{index}, but persisting only Object.values(...) removes those keys, making it impossible to correlate back to the original action that produced each element.
If consumers do not need the mapping you can remove the intermediate object and store the raw array directly; otherwise, persist the keyed object.

-        serializableOutput: {
-          scrapeSchema: Object.values(categorizedOutput.scrapeSchema),
-          scrapeList: Object.values(categorizedOutput.scrapeList),
-          other: Object.values(categorizedOutput.other),
-        },
+        // Keep the keyed structure to preserve traceability
+        serializableOutput: categorizedOutput,

Are downstream readers (e.g. analytics, UI) prepared for this schema change? Please double-check before merging.


I’d like to inspect how categorizedOutput is produced and where serializableOutput flows downstream. Let’s gather references and context around these identifiers:


🏁 Script executed:

#!/bin/bash
set -e

echo "Searching for collectDataByType usage/definition..."
rg "collectDataByType" -n .

echo ""
echo "Searching for categorizedOutput references..."
rg "categorizedOutput" -n .

echo ""
echo "Searching for serializableOutput usages..."
rg "serializableOutput" -n .

echo ""
echo "Displaying lines 250–340 in server/src/pgboss-worker.ts for context..."
sed -n '250,340p' server/src/pgboss-worker.ts

Length of output: 9378


I want to confirm how items are keyed and how the UI expects serializableOutput to be shaped. Let’s inspect the collectDataByType definition and the UI code that reads serializableOutput:


🏁 Script executed:

#!/bin/bash
set -e

echo "1) collectDataByType definition (around line 566):"
sed -n '550,620p' server/src/pgboss-worker.ts

echo ""
echo "2) RunContent.tsx usage (first 200 lines):"
sed -n '1,200p' src/components/run/RunContent.tsx

Length of output: 10556


Review serializableOutput schema and downstream compatibility
Changing serializableOutput to a different shape will break several consumers that today expect either arrays under category keys or top-level indexed keys. You must update them in lockstep or stick with the current structure.

Affected locations:
• server/src/pgboss-worker.ts (lines 278–283): where you build and persist serializableOutput
• src/components/run/RunContent.tsx: assumes row.serializableOutput[firstKey] is an Array
• server/src/workflow-management/integrations/gsheet.ts & airtable.ts: look up plainRun.serializableOutput['item-0'] at top level
• server/src/routes/storage.ts & server/src/api/record.ts: similar top-level “item-#” access
• server/src/models/Run.ts: typed as Record<string, any[]>

Next steps:

  1. Decide on a single output schema (e.g. categorized arrays vs. indexed objects).
  2. Update each consumer to traverse that schema (UI: iterate categories or flatten arrays; integrations: drill into nested objects; API/storage: adjust key lookups).
  3. Align the Run model type to match the final shape.

Until all of these are updated, please do not swap in categorizedOutput directly.

Committable suggestion skipped: line range outside the PR's diff.

});

// Track extraction metrics
let totalRowsExtracted = 0;
let totalSchemaItemsExtracted = 0;
let totalListItemsExtracted = 0;
let extractedScreenshotsCount = 0;
let extractedItemsCount = 0;

if (run.dataValues.binaryOutput && run.dataValues.binaryOutput["item-0"]) {
extractedScreenshotsCount = 1;

if (categorizedOutput.scrapeSchema) {
Object.values(categorizedOutput.scrapeSchema).forEach((schemaResult: any) => {
if (Array.isArray(schemaResult)) {
totalSchemaItemsExtracted += schemaResult.length;
} else if (schemaResult && typeof schemaResult === 'object') {
totalSchemaItemsExtracted += 1;
}
});
}

if (run.dataValues.serializableOutput && run.dataValues.serializableOutput["item-0"]) {
const itemsArray = run.dataValues.serializableOutput["item-0"];
extractedItemsCount = itemsArray.length;

totalRowsExtracted = itemsArray.reduce((total, item) => {
return total + Object.keys(item).length;
}, 0);

if (categorizedOutput.scrapeList) {
Object.values(categorizedOutput.scrapeList).forEach((listResult: any) => {
if (Array.isArray(listResult)) {
totalListItemsExtracted += listResult.length;
}
});
}

console.log(`Extracted Items Count: ${extractedItemsCount}`);

if (uploadedBinaryOutput) {
extractedScreenshotsCount = Object.keys(uploadedBinaryOutput).length;
}

const totalRowsExtracted = totalSchemaItemsExtracted + totalListItemsExtracted;

console.log(`Extracted Schema Items Count: ${totalSchemaItemsExtracted}`);
console.log(`Extracted List Items Count: ${totalListItemsExtracted}`);
console.log(`Extracted Screenshots Count: ${extractedScreenshotsCount}`);
console.log(`Total Rows Extracted: ${totalRowsExtracted}`);

Expand All @@ -306,7 +326,8 @@ async function processRunExecution(job: Job<ExecuteRunData>) {
created_at: new Date().toISOString(),
status: 'success',
totalRowsExtracted,
extractedItemsCount,
schemaItemsExtracted: totalSchemaItemsExtracted,
listItemsExtracted: totalListItemsExtracted,
extractedScreenshotsCount,
}
);
Expand Down Expand Up @@ -339,7 +360,7 @@ async function processRunExecution(job: Job<ExecuteRunData>) {
robotName: recording.recording_meta.name,
status: 'success',
finishedAt: new Date().toLocaleString()
});;
});

// Check for and process queued runs before destroying the browser
const queuedRunProcessed = await checkAndProcessQueuedRun(data.userId, plainRun.browserId);
Expand Down Expand Up @@ -458,7 +479,11 @@ async function abortRun(runId: string, userId: string): Promise<boolean> {
}

let currentLog = 'Run aborted by user';
let serializableOutput: Record<string, any> = {};
let categorizedOutput = {
scrapeSchema: {},
scrapeList: {},
other: {}
};
let binaryOutput: Record<string, any> = {};

try {
Expand All @@ -467,16 +492,16 @@ async function abortRun(runId: string, userId: string): Promise<boolean> {
currentLog = browser.interpreter.debugMessages.join('\n') || currentLog;
}

if (browser.interpreter.serializableData) {
browser.interpreter.serializableData.forEach((item, index) => {
serializableOutput[`item-${index}`] = item;
});
if (browser.interpreter.serializableDataByType) {
categorizedOutput = {
scrapeSchema: collectDataByType(browser.interpreter.serializableDataByType.scrapeSchema || []),
scrapeList: collectDataByType(browser.interpreter.serializableDataByType.scrapeList || []),
other: collectDataByType(browser.interpreter.serializableDataByType.other || [])
};
}

if (browser.interpreter.binaryData) {
browser.interpreter.binaryData.forEach((item, index) => {
binaryOutput[`item-${index}`] = item;
});
binaryOutput = collectBinaryData(browser.interpreter.binaryData);
}
}
} catch (interpreterError) {
Expand All @@ -488,7 +513,11 @@ async function abortRun(runId: string, userId: string): Promise<boolean> {
finishedAt: new Date().toLocaleString(),
browserId: plainRun.browserId,
log: currentLog,
serializableOutput,
serializableOutput: {
scrapeSchema: Object.values(categorizedOutput.scrapeSchema),
scrapeList: Object.values(categorizedOutput.scrapeList),
other: Object.values(categorizedOutput.other),
},
binaryOutput,
});

Expand Down Expand Up @@ -529,6 +558,30 @@ async function abortRun(runId: string, userId: string): Promise<boolean> {
}
}

/**
* Helper function to collect data from arrays into indexed objects
* @param dataArray Array of data to be transformed into an object with indexed keys
* @returns Object with indexed keys
*/
function collectDataByType(dataArray: any[]): Record<string, any> {
return dataArray.reduce((result: Record<string, any>, item, index) => {
result[`item-${index}`] = item;
return result;
}, {});
}

/**
* Helper function to collect binary data (like screenshots)
* @param binaryDataArray Array of binary data objects to be transformed
* @returns Object with indexed keys
*/
function collectBinaryData(binaryDataArray: { mimetype: string, data: string, type?: string }[]): Record<string, any> {
return binaryDataArray.reduce((result: Record<string, any>, item, index) => {
result[`item-${index}`] = item;
return result;
}, {});
}

async function registerRunExecutionWorker() {
try {
const registeredUserQueues = new Map();
Expand Down
Loading