Paper collaboration: Neurips workshop measuring autonomy of open source agents

[pcihon]

Hello!

Together with @gagb @merlin-stein @khxu and others, I'm working on a scalable method of measuring agent autonomy via code inspection. We have a paper accepted to a Neurips workshop, and are looking to expand the work beyond AutoGen. Curious if you or a colleague at LangChain may be interested in collaborating?

Abstract:
AI agents are systems that can achieve complex goals autonomously. Assessing an AI agent’s level of autonomy is key to understanding its benefits and risks. Current assessments of autonomy largely focus on specific risks and use run-time evaluations. Inspired by practices in other industries, we articulate levels of agent autonomy. We introduce a code-based assessment of autonomy – wherein the orchestration code used to run an AI system is scored according to a taxonomy that assesses attributes of autonomy: impact and oversight. We demonstrate this approach with the AutoGen framework and select applications.

@hwchase17 @baskaryan