Skip to content

feat(benchmark): Support Falcon Text2SQL Datasets LLM benchmark #2918

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 56 commits into
base: main
Choose a base branch
from
Open

feat(benchmark): Support Falcon Text2SQL Datasets LLM benchmark #2918

wants to merge 56 commits into from

Conversation

chenliang15405
Copy link
Collaborator

Description

  1. DB-GPT open-source Text2SQL benchmark dataset repository: Falcon
  2. DB-GPT supports LLM evaluation based on the Falcon benchmark dataset

How Has This Been Tested?

  1. Run DB-GPT service
  2. Waiting for the download of the Falcon database to be completed
  3. Enter the model evaluation module on the product page and create benchmark task

Please see docs/docs/modules/benchmark.md -> How To Use

Snapshots:

env_load 20251022160201

20251022160220

20251022160248

20251022160347 20251022162752

20251022160302

Checklist:

  • My code follows the style guidelines of this project
  • I have already rebased the commits and make the commit message conform to the project standard.
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • Any dependent changes have been merged and published in downstream modules

yaoyifan-yyf and others added 30 commits September 25, 2025 11:26
@yaoyifan-yyf
feat: async loading benchmark data
9e50bff
@yaoyifan-yyf
opt: code format
bc14084
@yaoyifan-yyf
feat: benchmark post_dispatch service
4de41d1
@yaoyifan-yyf
opt: async load benchmark data on init
92c251d
feat(benchmark): execute benchmark task
6293629
Merge remote-tracking branch 'origin/feat_dataset_benchmark' into fea…
4875df1 
…t_dataset_benchmark

# Conflicts:
#	packages/dbgpt-serve/src/dbgpt_serve/evaluate/service/benchmark/data_compare_service.py
#	packages/dbgpt-serve/src/dbgpt_serve/evaluate/service/benchmark/file_parse_service.py
#	packages/dbgpt-serve/src/dbgpt_serve/evaluate/service/benchmark/models.py
#	packages/dbgpt-serve/src/dbgpt_serve/evaluate/service/benchmark/user_input_execute_service.py
@yaoyifan-yyf
feat: query benchmark dataset api
57e8bab
@yaoyifan-yyf
feat: add benchmark result query api
c923e64
@chenliang15405
Merge remote-tracking branch 'origin/feat_dataset_benchmark' into fea…
61e350f 
…t_dataset_benchmark

# Conflicts:
#	packages/dbgpt-serve/src/dbgpt_serve/evaluate/api/endpoints.py
#	packages/dbgpt-serve/src/dbgpt_serve/evaluate/api/schemas.py
#	packages/dbgpt-serve/src/dbgpt_serve/evaluate/service/benchmark/file_parse_service.py
#	packages/dbgpt-serve/src/dbgpt_serve/evaluate/service/benchmark/user_input_execute_service.py
@chenliang15405
chore: resolve confict
0330dd3
@chenliang15405
feat(benchmark): optimize benchmark task and write evaluate result ro…
f6351ae 
…ws offset
@yaoyifan-yyf
fix: add table mapping
81e2a1c
@chenliang15405
feat(benchmark): create benchmark task
8217408
@chenliang15405
Merge remote-tracking branch 'origin/feat_dataset_benchmark' into fea…
5b27b6e 
…t_dataset_benchmark
@chenliang15405
fix(benchmark): fix post dispatch param
33a4e04
@yaoyifan-yyf
opt: compare result write to excel not db
9b81a10
@chenliang15405
feat(benchmark): multi model post process
87f11b5
@yaoyifan-yyf
opt: multi model compare write result
8d8d455
@chenliang15405
feat(benchmark): query benchmark task list
c14b68d
@yaoyifan-yyf
opt: add standard result col to output excel
ab96ebb
@yaoyifan-yyf
Merge remote-tracking branch 'origin/feat_dataset_benchmark' into fea…
838bc35 
…t_dataset_benchmark
@chenliang15405
feat(benchmark): benchmark result file download
5df8d94
@chenliang15405
fix(benchmark): parse multi standard anwser
92243cb
@chenliang15405
fix(benchmark): update standard anwser result field
65fd87b
@yaoyifan-yyf
fix: ant_icube table mapping correct
24064d7
@yaoyifan-yyf
fix: col name sanitize modification
05b1fb6
@yaoyifan-yyf
fix: benchmark compare summary write to db
ff34064
@yaoyifan-yyf
fix: benchmark compare summary write to db
41da1b3
@yaoyifan-yyf
opt: benchmark result api output adjust
fb83e30
@yaoyifan-yyf
opt: api name adjuest
ba80df5
yaoyifan-yyf and others added 26 commits October 16, 2025 14:56
@yaoyifan-yyf
feat: support multi benchmark datasets
2a823ee
@chenliang15405
feat(benchmark): update benchmark task status & benchmark task info list
8e025c8
@chenliang15405
fix: fix page list request
8f0b2c3
@chenliang15405
fix(benchmark): execute benchmark with model param
39ac73d
@chenliang15405
fix(benchmark): process sql query timeout
cda5b74
@chenliang15405
fix(benchmark): fix sql query db timeout for blocking thread
ed59a71
@chenliang15405
fix(benchmark): remove useless code
b410e85
@iterminatorheart @kobzevvv @Aries-ckt
feat: add datasets evaluation page (#2908)
d2e92e9 
Co-authored-by: VLADIMIR KOBZEV <vladimir.kobzev@improvado.io>
Co-authored-by: Aries-ckt <916701291@qq.com>
Co-authored-by: xiandu.wl <xiandu.wl@antgroup.com>
@iterminatorheart @kobzevvv @Aries-ckt
feat: evaluation dataset info pages (#2911)
19bbc6f 
Co-authored-by: VLADIMIR KOBZEV <vladimir.kobzev@improvado.io>
Co-authored-by: Aries-ckt <916701291@qq.com>
Co-authored-by: xiandu.wl <xiandu.wl@antgroup.com>
@chenliang15405
fix(benchmark): custom model temperature and max token
9de988f
@iterminatorheart @kobzevvv @Aries-ckt
feat: multi language for models evaluation (#2912)
cfdb1db 
Co-authored-by: VLADIMIR KOBZEV <vladimir.kobzev@improvado.io>
Co-authored-by: Aries-ckt <916701291@qq.com>
Co-authored-by: xiandu.wl <xiandu.wl@antgroup.com>
@yaoyifan-yyf
fix: table error fix
e69a4e5
@yaoyifan-yyf
Merge branch 'feat_dataset_benchmark' of github.com:eosphoros-ai/DB-G…
3f0d63b 
…PT into feat_dataset_benchmark
@yaoyifan-yyf
fix: table error fix
48a3798
@chenliang15405
feat(benchmark): show task name
48aa204
@chenliang15405
chore: web build file
9ada90b
@chenliang15405
fix(benchmark): fix download result url
2caa317
@chenliang15405
chore: update ignore
2cd22a2
@chenliang15405
Merge remote-tracking branch 'origin/feat_dataset_benchmark' into fea…
8c1bfed 
…t_dataset_benchmark
@yaoyifan-yyf
fix: answer adjustment
6904d02
@yaoyifan-yyf
Merge branch 'feat_dataset_benchmark' of github.com:eosphoros-ai/DB-G…
3beb281 
…PT into feat_dataset_benchmark
@chenliang15405
chore: update falcon repo url
59e39b2
@chenliang15405
docs: add dataset benchmark docs
61bbba3
@chenliang15405
docs: update image
81cee77
@chenliang15405
docs: update benchmark doc
f3263c8
@chenliang15405
docs: update benchmark doc
f0de206
@github-actions github-actions bot added the enhancement New feature or request label Oct 22, 2025
Aries-ckt
Aries-ckt approved these changes Oct 22, 2025
View reviewed changes
Copy link
Collaborator

@Aries-ckt Aries-ckt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

fangyinc
fangyinc reviewed Oct 22, 2025
View reviewed changes
Copy link
Collaborator

@fangyinc fangyinc left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please update the changed MySQL DDL to assets/schema/dbgpt.sql @chenliang15405

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@fangyinc fangyinc fangyinc left review comments

@Aries-ckt Aries-ckt Aries-ckt approved these changes

At least 2 approving reviews are required to merge this pull request.

Assignees

No one assigned

Labels

enhancement New feature or request

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@chenliang15405 @Aries-ckt @fangyinc @yaoyifan-yyf @iterminatorheart