Skip to content

Multithreaded Archon Benchmark Quality/Time/Cost Estimation Framework #1

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 10 commits into
base: dev
Choose a base branch
from
Open

Multithreaded Archon Benchmark Quality/Time/Cost Estimation Framework #1

wants to merge 10 commits into from

Conversation

yash94404
Copy link
Collaborator

@yash94404 yash94404 commented Apr 30, 2025 •
edited
Loading

  • Generated 100 archon config operators of various depths 2-3 based on Archon rules with generator, critic, ranker, and fuser layers (generate_config.py)
  • Added temporary buffer for each thread and flushed buffer at the end of thread execution of a config to log for a certain question (utils.py, benchmark.py)
  • Added GPQA_Diamond Benchmark execution class to Archon (benchmark.py)
  • Log parsing logic to generate quality/cost/time estimates based on log and model outputs

yash94404 added 10 commits March 31, 2025 16:40
@yash94404
archon testing framework initial push
cd5168e
@yash94404
Remove large files and ignore them
c591402
@yash94404
Track large files with Git LFS
054b180
@yash94404
Re-added large files with Git LFS
d25c1a9
@yash94404
committing archon operator testing code against benchmarks
ad19b61
@yash94404
cleaning up extraneous files
f021dc7
@yash94404
Multithreaded Archon Benchmark Quality/Time/Cost Estimation Framework
8d1e13b 
- Added temporary buffer for each thread and flushed buffer at the end of thread execution of a config to log for a certain question (in utils.py and benchmark.py)
- Added GPQA_Diamond Benchmark execution class to Archon (benchmark.py)
- Log parsing logic to generate quality/cost/time estimates based on log and model outputs
@yash94404
cleaning up code
6023bfa
@yash94404
adding more log processing files
55c528f
@yash94404
adding 100 operators, tweaked config generation script
05be722
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@yash94404