Skip to content

feat(webui): Create result metadata document in MongoDB to track search signal for Presto queries. #1168

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 20 commits into from
Aug 11, 2025
Merged

feat(webui): Create result metadata document in MongoDB to track search signal for Presto queries. #1168

Show file tree
Hide file tree
Changes from 15 commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
fbc0fb4
feat: Add presto search route.
hoophalab Jul 22, 2025
73d41d5
fix a typo in jsdoc
hoophalab Jul 25, 2025
767dcf5
Merge branch 'main' into presto-search-route
junhaoliao Jul 30, 2025
879941e
latest
Jul 31, 2025
b2c8614
latest
Jul 31, 2025
d1ce3f3
latest
Jul 31, 2025
20b4f2f
latest:
Aug 1, 2025
b132990
latest
Aug 5, 2025
51a1eab
latest:
Aug 5, 2025
0573aab
latest
Aug 5, 2025
0953cc1
metadata
Aug 5, 2025
6240164
latest
Aug 5, 2025
6de9a93
latest
Aug 6, 2025
b6c8c8b
latest
Aug 6, 2025
6fb776b
latest
Aug 6, 2025
ff4d0c4
changes
Aug 8, 2025
5a26a33
latest
Aug 8, 2025
adc0a0d
latest
Aug 8, 2025
f79b1ef
Merge branch 'main' into metadata
davemarco Aug 11, 2025
3cdca78
Merge branch 'main' into metadata
hoophalab Aug 11, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion components/webui/client/public/settings.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"ClpStorageEngine": "clp",
"ClpQueryEngine": "native",
"ClpQueryEngine": "clp",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should also configure this in server_settings_json_updates in components/clp-package-utils/clp_package_utils/scripts/start_clp.py

@coderabbitai create an issue to track this

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧩 Analysis chain

I'll examine the current configuration in the start_clp.py file and then create an issue to track this update.

@junhaoliao I've created an issue to track updating the ClpQueryEngine setting in the server_settings_json_updates configuration within start_clp.py. The issue has been assigned to you and includes the context from this PR and comment thread.

🏁 Script executed:

#!/bin/bash
# Examine the server_settings_json_updates configuration in start_clp.py
rg -n -A10 -B5 "server_settings_json_updates" components/clp-package-utils/clp_package_utils/scripts/start_clp.py

Length of output: 2708

🏁 Script executed:

#!/bin/bash
# Create a GitHub issue to track updating the ClpQueryEngine setting
gh issue create \
  --title "Update ClpQueryEngine setting in start_clp.py server_settings_json_updates" \
  --body "## Description

The ClpQueryEngine setting was updated from \"native\" to \"clp\" in PR #1168 for both client and server settings.json files, but the corresponding configuration in \`server_settings_json_updates\` within \`components/clp-package-utils/clp_package_utils/scripts/start_clp.py\` also needs to be updated for consistency.

## Task

Update the \`server_settings_json_updates\` configuration in \`components/clp-package-utils/clp_package_utils/scripts/start_clp.py\` to change \`ClpQueryEngine\` from \"native\" to \"clp\".

## Context

- **PR**: #1168
- **Comment**: https://github.com/y-scope/clp/pull/1168#discussion_r2296622518
- **Requested by**: @junhaoliao

This ensures consistency across all configuration files in the codebase." \
  --assignee junhaoliao

Length of output: 907

@junhaoliao I've successfully created issue #1259 to track updating the ClpQueryEngine setting in the server_settings_json_updates configuration.

From examining the current code, I can see that the server_settings_json_updates dictionary in start_clp.py doesn't currently include a ClpQueryEngine setting, so it will need to be added with the value "clp" to maintain consistency with the changes made in this PR.

The issue has been assigned to you and includes all the relevant context and links.

🧠 Learnings used 
Learnt from: junhaoliao
PR: y-scope/clp#1152
File: components/clp-package-utils/clp_package_utils/scripts/start_clp.py:613-613
Timestamp: 2025-08-08T06:59:42.436Z
Learning: In components/clp-package-utils/clp_package_utils/scripts/start_clp.py, generic_start_scheduler sets CLP_LOGGING_LEVEL using clp_config.query_scheduler.logging_level for both schedulers; compression scheduler should use its own logging level. Tracking via an issue created from PR #1152 discussion.

"MongoDbSearchResultsMetadataCollectionName": "results-metadata",
"SqlDbClpArchivesTableName": "clp_archives",
"SqlDbClpDatasetsTableName": "clp_datasets",
Expand Down
9 changes: 1 addition & 8 deletions components/webui/client/src/config/index.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
import {CLP_QUERY_ENGINES} from "../../../common/index.js";
import {settings} from "../settings";


Expand All @@ -9,14 +10,6 @@ enum CLP_STORAGE_ENGINES {
CLP_S = "clp-s",
}

/**
* Query engine options.
*/
enum CLP_QUERY_ENGINES {
NATIVE = "native",
PRESTO = "presto",
}

const SETTINGS_STORAGE_ENGINE = settings.ClpStorageEngine as CLP_STORAGE_ENGINES;
const SETTINGS_QUERY_ENGINE = settings.ClpQueryEngine as CLP_QUERY_ENGINES;

Expand Down
2 changes: 1 addition & 1 deletion components/webui/client/src/pages/SearchPage/SearchControls/index.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ const SearchControls = () => {
return (
<form onSubmit={handleSubmit}>
<div className={styles["searchControlsContainer"]}>
{SETTINGS_QUERY_ENGINE === CLP_QUERY_ENGINES.NATIVE ?
{SETTINGS_QUERY_ENGINE !== CLP_QUERY_ENGINES.PRESTO ?
(
<>
{CLP_STORAGE_ENGINES.CLP_S === SETTINGS_STORAGE_ENGINE && <Dataset/>}
Expand Down
32 changes: 31 additions & 1 deletion components/webui/common/index.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,33 @@ enum SEARCH_SIGNAL {
RESP_QUERYING = "resp-querying",
}

/**
* Presto search-related signals.
*
* Note: Using type instead of enum to match `presto-client-node` type definition.
*/
type PRESTO_SEARCH_SIGNAL =
| "WAITING_FOR_PREREQUISITES"
| "QUEUED"
| "WAITING_FOR_RESOURCES"
| "DISPATCHING"
| "PLANNING"
| "STARTING"
| "RUNNING"
| "FINISHING"
| "FINISHED"
| "CANCELED"
| "FAILED";

/**
* CLP query engines.
*/
enum CLP_QUERY_ENGINES {
CLP = "clp",
CLP_S = "clp-s",
PRESTO = "presto",
}

/**
* MongoDB document for search results metadata. `numTotalResults` is optional
* since it is only set when the search job is completed.
Expand All @@ -98,13 +125,16 @@ interface SearchResultsMetadataDocument {
// eslint-disable-next-line no-warning-comments
// TODO: Replace with Nullable<string> when the `@common` directory refactoring is completed.
errorMsg: string | null;
lastSignal: SEARCH_SIGNAL;
lastSignal: SEARCH_SIGNAL | PRESTO_SEARCH_SIGNAL;
numTotalResults?: number;
queryEngine: CLP_QUERY_ENGINES;
}
export {
CLP_QUERY_ENGINES,
SEARCH_SIGNAL,
};
export type {
PRESTO_SEARCH_SIGNAL,
SearchResultsMetadataDocument,
ClientToServerEvents,
Err,
Expand Down
2 changes: 1 addition & 1 deletion components/webui/server/settings.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
"StreamFilesS3PathPrefix": null,
"StreamFilesS3Profile": null,

"ClpQueryEngine": "native",
"ClpQueryEngine": "clp",
"PrestoHost": "localhost",
"PrestoPort": 8889
}
31 changes: 31 additions & 0 deletions components/webui/server/src/routes/api/presto-search/index.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,17 @@
import {FastifyPluginAsyncTypebox} from "@fastify/type-provider-typebox";
import {StatusCodes} from "http-status-codes";

import {
CLP_QUERY_ENGINES,
type SearchResultsMetadataDocument,
} from "../../../../../common/index.js";
import settings from "../../../../settings.json" with {type: "json"};
import {ErrorSchema} from "../../../schemas/error.js";
import {
PrestoQueryJobCreationSchema,
PrestoQueryJobSchema,
} from "../../../schemas/presto-search.js";
import {updateSearchResultsMeta} from "../search/utils.js";
import {insertPrestoRowsToMongo} from "./utils.js";


Expand All @@ -28,6 +34,10 @@ const plugin: FastifyPluginAsyncTypebox = async (fastify) => {
throw new Error("MongoDB database not found");
}

const searchResultsMetadataCollection = mongoDb.collection<SearchResultsMetadataDocument>(
settings.MongoDbSearchResultsMetadataCollectionName
);

/**
* Submits a search query.
*/
Expand All @@ -50,6 +60,7 @@ const plugin: FastifyPluginAsyncTypebox = async (fastify) => {
let searchJobId: string;

try {
// eslint-disable-next-line max-lines-per-function
searchJobId = await new Promise<string>((resolve, reject) => {
let isResolved = false;
Presto.client.execute({
Expand Down Expand Up @@ -100,9 +111,29 @@ const plugin: FastifyPluginAsyncTypebox = async (fastify) => {
state: stats.state,
}, "Presto search state updated");

// Insert metadata and resolve queryId on first call
if (false === isResolved) {
searchResultsMetadataCollection.insertOne({
_id: queryId,
lastSignal: stats.state,
errorMsg: null,
queryEngine: CLP_QUERY_ENGINES.PRESTO,
}).catch((err: unknown) => {
request.log.error(err, "Failed to insert Presto metadata");
});
isResolved = true;
resolve(queryId);
} else {
// Update metadata on subsequent calls
updateSearchResultsMeta({
fields: {lastSignal: stats.state},
jobId: queryId,
logger: request.log,
searchResultsMetadataCollection:
searchResultsMetadataCollection,
}).catch((err: unknown) => {
request.log.error(err, "Failed to update Presto metadata");
});
}
},
success: () => {
Expand Down
2 changes: 1 addition & 1 deletion components/webui/server/src/routes/api/presto-search/utils.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ const insertPrestoRowsToMongo = (
columns: {name: string}[],
searchJobId: string,
mongoDb: Db
): Promise<InsertManyResult<Document>> => {
): Promise<InsertManyResult> => {
const collection = mongoDb.collection(searchJobId);
const resultDocs = data.map((row) => prestoRowToObject(row, columns));
return collection.insertMany(resultDocs);
Expand Down
6 changes: 5 additions & 1 deletion components/webui/server/src/routes/api/search/index.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ import {
import {StatusCodes} from "http-status-codes";

import {
CLP_QUERY_ENGINES,
SEARCH_SIGNAL,
type SearchResultsMetadataDocument,
} from "../../../../../common/index.js";
Expand Down Expand Up @@ -44,6 +45,8 @@ const plugin: FastifyPluginAsyncTypebox = async (fastify) => {
settings.MongoDbSearchResultsMetadataCollectionName
);

const queryEngine = settings.ClpQueryEngine as CLP_QUERY_ENGINES;

/**
* Submits a search query and initiates the search process.
*/
Expand Down Expand Up @@ -113,6 +116,7 @@ const plugin: FastifyPluginAsyncTypebox = async (fastify) => {
_id: searchJobId.toString(),
lastSignal: SEARCH_SIGNAL.RESP_QUERYING,
errorMsg: null,
queryEngine: queryEngine,
});

// Defer signal update until after response is sent
Expand Down Expand Up @@ -202,7 +206,7 @@ const plugin: FastifyPluginAsyncTypebox = async (fastify) => {
lastSignal: SEARCH_SIGNAL.RESP_DONE,
errorMsg: "Query cancelled before it could be completed.",
},
jobId: searchJobId,
jobId: searchJobId.toString(),
lastSignal: SEARCH_SIGNAL.RESP_QUERYING,
logger: request.log,
searchResultsMetadataCollection: searchResultsMetadataCollection,
Expand Down
4 changes: 2 additions & 2 deletions components/webui/server/src/routes/api/search/typings.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,8 @@ const SEARCH_MAX_NUM_RESULTS = 1000;

type UpdateSearchResultsMetaProps = {
fields: Partial<SearchResultsMetadataDocument>;
jobId: number;
lastSignal: SEARCH_SIGNAL;
jobId: string;
lastSignal?: SEARCH_SIGNAL;
logger: FastifyBaseLogger;
searchResultsMetadataCollection: Collection<SearchResultsMetadataDocument>;
};
Expand Down
11 changes: 7 additions & 4 deletions components/webui/server/src/routes/api/search/utils.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,11 +38,14 @@ const updateSearchResultsMeta = async ({
logger,
searchResultsMetadataCollection,
}: UpdateSearchResultsMetaProps) => {
const filter = {
_id: jobId.toString(),
lastSignal: lastSignal,
const filter: Record<string, unknown> = {
_id: jobId,
};

if (lastSignal) {
filter.lastSignal = lastSignal;
}

const modifier = {
$set: fields,
};
Expand Down Expand Up @@ -112,7 +115,7 @@ const updateSearchSignalWhenJobsFinish = async ({
SEARCH_MAX_NUM_RESULTS
),
},
jobId: searchJobId,
jobId: searchJobId.toString(),
lastSignal: SEARCH_SIGNAL.RESP_QUERYING,
logger: logger,
searchResultsMetadataCollection: searchResultsMetadataCollection,
Expand Down
Loading