[EPIC] AI Scientist 2 Issue cleanup

[erkinalp]

AI Scientist 2 is about to be released. Let's review and complete the issues for AI Scientist to avoid duplicate work.

[isCopyman]

When will it be released?

[erkinalp]

Soon, according to https://sakana.ai/ai-scientist-first-publication/

[feier-syh]

Could you send me the open source code of version 2? Thank you very much.

[conglu1997]

It is released here: https://github.com/SakanaAI/AI-Scientist-v2

[erkinalp]

The following issues from AI-Scientist v1 can be closed because they've been addressed in AI-Scientist v2 (recheck them)

Model Support Issues

• Test with Grok 3 #196: Test with Grok 3
• Test with DeepSeek-R1 #195: Test with DeepSeek-R1
• Test with Kimi k1.5 #184: Test with Kimi k1.5
• Have anyone tried to replace the chatgpt model with DEEPSEEK? Do you know how to change this? #121: Have anyone tried to replace the chatgpt model with DEEPSEEK?
• Has anyone tried with open source LLM or gemini free usage? #101: Has anyone tried with open source LLM or gemini free usage?
• edit the code to be able using the hermes-3-llama-3.1-405b free version from openrouter cause the error #100: Edit the code to be able using the hermes-3-llama-3.1-405b free version from openrouter
• support for huggingface llms #87: Support for huggingface llms
• Add support for Grok-2 #66: Add support for Grok-2
• Can we add azure? #48: Can we add azure?
• Can I directly use the API of an AI model that is not on the list? #211: Can I directly use the API of an AI model that is not on the list?

Template and Research Domain Issues

• Updating NanoGPT with a "speed run" version #181: Updating NanoGPT with a "speed run" version
• Template based on Img2Txt Diffusion finetuning? #171: Template based on Img2Txt Diffusion finetuning?
• About outputting other types of languages #159: About outputting other types of languages
• Question: Adding a new template and getting accurate content for the paper. #149: Question: Adding a new template and getting accurate content for the paper
• add Prompt for topic Research with NanoGPT #148: Add Prompt for topic Research with NanoGPT
• Is it possible to generate ideas without running baseline (experiment.py)? #142: Is it possible to generate ideas without running baseline (experiment.py)?
• How to control the research subject? #132: How to control the research subject?
• how to run my own template? #110: How to run my own template?
• Lift code restrictions and allow other subjects to be explored  #70: Lift code restrictions and allow other subjects to be explored

Architecture and Design Issues

• calling AI scientists is overstatement #198: Calling AI scientists is overstatement
• Inquiry Regarding Open-Ended Feedback Loop in AI Scientist #193: Inquiry Regarding Open-Ended Feedback Loop in AI Scientist
• Is editing ideas.json and prompt.json enough ? #172: Is editing ideas.json and prompt.json enough?
• Question: How are experiment results accessed when doing the paper writeup? #147: Question: How are experiment results accessed when doing the paper writeup?
• AI - Scientist for Non-Developers #124: AI - Scientist for Non-Developers
• Why Not Use The TeXLive.Net Server? #117: Why Not Use The TeXLive.Net Server?
• Use unsloth as a booster? #111: Use unsloth as a booster?
• run_experiment() still seems to be just executing the original experiment.py file #106: run_experiment() still seems to be just executing the original experiment.py file
• For an existing idea, what exactly do i need to configure to run this project? #85: For an existing idea, what exactly do i need to configure to run this project?
• Sharing of AI-Scientist project and further automaton of sci. res. using the S.D.A.D. DAQ #80: Sharing of AI-Scientist project and further automaton of sci. res. using the S.D.A.D. DAQ
• A potential approach to enhance idea generation. #19: A potential approach to enhance idea generation

API and External Service Issues

• citationStyles key for OpenAlex API #179: citationStyles key for OpenAlex API
• rate_limit #155: rate_limit
• Response Status Code: 429 - Too many requests #133: Response Status Code: 429 - Too many requests
• tokens has exceeded your daily rate limit #115: Tokens has exceeded your daily rate limit
• S2_API_KEY for semanticscholar.org  #104: S2_API_KEY for semanticscholar.org
• https://www.semanticscholar.org/product/api#api-key-form", "code": "429" #84: https://www.semanticscholar.org/product/api#api-key-form", "code": "429"
• Why set openai api key,show "Connection error" , and Claude3.5 show "403" #61: Why set openai api key, show "Connection error", and Claude3.5 show "403"
• How to setup the api  #41: How to setup the api
• Semantic Scholar requests limited. #18: Semantic Scholar requests limited

Error Handling and Bug Issues

• The LLM did not conform to the edit format. #154: The LLM did not conform to the edit format
• litellm.BadRequestError #151: litellm.BadRequestError
• Stuck On Paper Generation Experiments python launch_scientist.py --model "gpt-4o-2024-05-13" --experiment nanoGPT_lite --num-ideas 2 #131: Stuck On Paper Generation Experiments
• Review Error ! (file not found 'report.pdf') #127: Review Error ! (file not found 'report.pdf')
• How to comment out review paper innovative code #126: How to comment out review paper innovative code
• Failed to generate idea: 'str' object has no attribute 'choices' #125: Failed to generate idea: 'str' object has no attribute 'choices'
• The following error occurred #118: The following error occurred
• The process has been stuck at the retrieval phase for about an hour. Is this normal? #116: The process has been stuck at the retrieval phase for about an hour. Is this normal?
• Checking novelty of idea 0: adaptive_block_size Response Status Code: 200 Response Content: {"totalHits":0,"limit":5,"offset":0,"results":[]} #113: Checking novelty of idea 0: adaptive_block_size Response Status Code: 200 Response Content: {"totalHits":0,"limit":5,"offset":0,"results":[]}
• When running in a Docker environment, the following log is output and there is no result. #108: When running in a Docker environment, the following log is output and there is no result
• Execution error please!! #93: Execution error please!!
• RuntimeError: probability tensor contains either inf, nan or element < 0 #90: RuntimeError: probability tensor contains either inf, nan or element < 0
• nanoGPT_lite sanity check...RuntimeWarning: Mean of empty slice. #89: nanoGPT_lite sanity check...RuntimeWarning: Mean of empty slice
• I get the following error python experiment.py --out_dir run_0 #81: I get the following error python experiment.py --out_dir run_0
• always backing off #78: Always backing off
• process stops progressing after reaching "generating idea 2/2 #71: Process stops progressing after reaching "generating idea 2/2"
• FileNotFoundError: [Errno 2] No such file or directory: 'DatasaurusDozen.tsv' #60: FileNotFoundError: [Errno 2] No such file or directory: 'DatasaurusDozen.tsv'
• FileNotFoundError: [Errno 2] No such file or directory: '../../data/shakespeare_char/train.bin' #55: FileNotFoundError: [Errno 2] No such file or directory: '../../data/shakespeare_char/train.bin'
• extract_json_between_markers doesn't handle responses missing json markers #53: extract_json_between_markers doesn't handle responses missing json markers
• No such file or directory: 'templates/nanoGPT_lite/run_0/final_info.json' #51: No such file or directory: 'templates/nanoGPT_lite/run_0/final_info.json'
• An error "The LLM did not conform to the edit format." reported and entering in an infinite loop when editting the template.tex #45: An error "The LLM did not conform to the edit format." reported and entering in an infinite loop when editting the template.tex
• from aider.coders import Coder still can not be resolved #23: from aider.coders import Coder still can not be resolved

Environment and Setup Issues

• Feature Request: Enhancing Accessibility and Efficiency by Hosting on Google Colab #200: Feature Request: Enhancing Accessibility and Efficiency by Hosting on Google Colab
• Hardware/software requirements for repo #186: Hardware/software requirements for repo
• After trying several times installing it I get constantly reload window and WSL breaks. #174: After trying several times installing it I get constantly reload window and WSL breaks

[BradKML]

@erkinalp pack each of these issues into clusters and re-start them in AI-Scientist v2. Assume v2 is a refactored rewrite instead of being a small update. And then each PR can rely on coding agents to enhace