-
Notifications
You must be signed in to change notification settings - Fork 208
Open
Description
Self-Check List
Before submitting an issue, please ensure you have completed the following steps:
- I have carefully read the relevant user documentation
- I have reviewed the Frequently Asked Questions
- I have searched and reviewed existing issues to confirm this is not a duplicate problem
Problem Description
Please briefly describe the problem you encountered.
The third-party VLM API carries params, which can be accessed via curl. However, params cannot be configured through evalscope code, making evaluation impossible
EvalScope Version (Required)
v1.0.2
Tools Used
- Native / Native framework
- Opencompass backend
- VLMEvalKit backend
- RAGEval backend
- Perf / Model inference stress testing tool
- Arena / Arena mode
Executed Code or Instructions
Please provide the main code or instructions you executed.
from evalscope import TaskConfig, run_task
from evalscope.constants import EvalType
task_cfg = TaskConfig(
work_dir='outputs',
model='gpt-5',
api_url='https://xxx.xxx.xxx.xxx/openai/deployments/gpt-5?api-version=xxxx-preview',
api_key='apikey',
eval_type=EvalType.SERVICE,
datasets=[
'real_world_qa',
],
eval_batch_size=5,
generation_config={
'max_completion_tokens': 10000,
'n': 1,
},
)
run_task(task_cfg=task_cfg)
Error Log
Please paste the complete error log or console output.
} prediction failed: due to Error code: 404 - {'error': {'code': '404', 'message': 'Resource not found'}}
Traceback:
Traceback (most recent call last):
File "/data/zhangxinfeng3/evalscope/evalscope/evalscope/evaluator/evaluator.py", line 182, in get_answers
task_state = future.result()
^^^^^^^^^^^^^^^
File "/usr/python/lib/python3.11/concurrent/futures/_base.py", line 449, in result
Running Environment
- Operating System: Centos7.9
- Python Version: python3.11
Additional Information
If there is any other relevant information, please provide it here.
I need to add an extra params at the end of the url and try to get the same error through generation_config["extra_body"]
Access the normal curl command
curl https://xxx/openai/deployments/gpt-5/chat/completions?api-version=xxxx-preview \
-H "Content-Type: application/json" \
-H "Authorization: Bearer xxxx" \
-d $'{
"model": "gpt-5",
"messages": [
{
"content": [
{
"image_url": {
"url": "images_url"
},
"type": "image_url"
},
{
"text": "What does the picture mainly talk about?",
"type": "text"
}
],
"role": "user"
}
]
}'
Metadata
Metadata
Assignees
Labels
No labels