Logging for DefaultAgent?

[doehyunbaek]

Describe the issue

Hi! I'm having a blast running this little gem.

One question: When invoking DefaultAgent from other python project, how do you envision observing the progress of DefaultAgent running?

I just did a very simple addition of adding loggings to default.py, which is probably not the cleanest way to do it XD:

00a3d29

Is there any better way to do this?

[klieret]

Hi, thanks for the question and the kind workds :)

Default.py is designed to be absolute bare-bones and it will keep all the data in DefaultAgent.messages, so logging simply doesn't have any additional advantage IMO (we use minisweagent.run.utils.save.save_traj(agent) to save all the data).

You can use the InteractiveAgent subclass (but that one is more for printing on the screen and interaction), or you just do a small subclass

class AgentWithLogs
    def add_message(self, role, content, **kwargs):
        logger.debug("..."_
        super().add_message(...)

[klieret]

Though on the other hand, you're not the only one to ask, so maybe it's something to consider... Let me think about it! Would probably have to disable it for multiprocessing/batch use cases though

[klieret]

I guess you want the logging because you want to see the live progress but you don't need interactivity?

[doehyunbaek]

minisweagent.run.utils.save.save_traj(agent)

Thanks for the pointer! I was also using this method call for the purpose of saving the output of DefaultAgent at the end.

class AgentWithLogs

I think this would be cleaner than me just updating default.py, So will adapt this for now. Thanks!

You can use the InteractiveAgent subclass (but that one is more for printing on the screen and interaction)
I guess you want the logging because you want to see the live progress but you don't need interactivity?

I am already using InteractiveAgent! IMO both are useful for my workflow (InteractiveAgent for hands-on dev, DefaultAgent for long-running eval). I think you can live without live logging for DefaultAgent, but I would rather prefer to have both the live log and the final trajectory.

Though on the other hand, you're not the only one to ask, so maybe it's something to consider... Let me think about it! Would probably have to disable it for multiprocessing/batch use cases though

If many people has this need I think simple configuration with additional argument would do.

[doehyunbaek]

btw I do not want to hijack this issue (I think I should rather make a separate issue) but for the use case of parallel evaluation with swe-rex, how do you envision using mini-swe-agent with swerex? Maybe not through logging but I would assume you would need some kind of mechanism to hand out data in DefaultAgent.messages to the progress dashboard. I think it's a tangentially related problem to this (relaying some intermediate agent result in non-interactive situation).

https://swe-rex.com/latest/assets/swerex.gif

[klieret]

Hi, @doehyunbaek let me get back to you on the logging, but for the progress board, we're doing that for the swebench.py batch processing (in mini under run/extra). Basically you only need to put a hook into the step method (or add_messages, equivalently).

Doesn't actually have anything to do with swe-rex

[klieret]

class ProgressTrackingAgent(DefaultAgent):
    """Simple wrapper around DefaultAgent that provides progress updates."""

    def __init__(self, *args, progress_manager: RunBatchProgressManager, instance_id: str = "", **kwargs):
        super().__init__(*args, **kwargs)
        self.progress_manager: RunBatchProgressManager = progress_manager
        self.instance_id = instance_id

    def step(self) -> dict:
        """Override step to provide progress updates."""
        self.progress_manager.update_instance_status(
            self.instance_id, f"Step {self.model.n_calls + 1:3d} (${self.model.cost:.2f})"
        )
        return super().step()

see https://github.com/SWE-agent/mini-swe-agent/blob/main/src/minisweagent/run/extra/swebench.py

[klieret]

Let me get back to this thread tomorrow