Agent Operations Working Group Plan | DRAFT

[omrishiv]

Agent Operations Working Group Plan

Mission: To provide a secure, production ready, scalable and reliable environment for developing and deploying agentic AI.

Problem Statement: Building agents is hard. After the code is written, what needs to be done to deploy them? After they're deployed, how are they productionalized? How does one know that the agents are performing well and producing value?

Solution: The operations working group aims to tackle the problems of productionalizing and operationalizing agents. We aim to provide a reference architecture in which agents are able to be deployed, tools to take agents from development to production, and an environment where they are continuously monitored and their performance evaluated. We also aim to understand how to best support enabling tool/agent discovery and operationalization.

Goals & Success:
What We Want to Achieve
[Primary Goal] - Create a reference architecture for enabling development and productionalization of agentic AI.
[Secondary Goal] - Create a registry that enables agent/tool discovery and usage.
[Third Goal] - Support organizational needs for auditing, compliance, and governance.

How We'll Know We Succeeded:
By the end, we want to have:
[ ] Provided a deployable architecture that enables developing and productionalizing agents across self-hosted or cloud-agnostic environments.
[ ] A comprehensive white paper on agentic operations best practices.
[ ] A well-researched blog post detailing findings on core agentic questions.
[ ] At least 4 successful adopters of the architecture across different industry verticals.

Core Open Questions

Questions	Why does it matter?	How do we approach it?
Do we need multiple registries for agents, mcp, tools?	We have registries for microservices, we have registries for MCP servers, we may now have registries that are specific to other entities. Centralizing the entities into a single registry may minimize operational overhead and reduce compute, but will need to support the different functionality those entities need.	Understand if these entities are truly separate or if they can be treated the same way. Can we leverage existing service registries or do we need to build new tools for agentic systems?
How do we evaluate performance? Does it have to be use-case specific or is there a more generic way?	Creating a generic way to evaluate agent performance will allow us to evaluate more use cases rather than require each one to have its own individual evaluation criteria
How do we provide neutral interfaces across agent frameworks and requisite tooling?	Individuals and organizations may already have existing tooling they support and will not want to adopt an architecture wholesale; we want them to also be able to consume the architecture we provide	Providing explanations around the decisions and best practices around the tooling will enable organizations to adopt the learnings for their own tools. Providing neutral interfaces will allow supporting more tools while abstracting away the logic for the developers

Other Questions

• How do we take the agents we develop on our laptops to production in a reproducible manner?
• How do we ensure that those agents that are in development and production are operating efficiently? What does efficiency mean?
• How do we evaluate performance? Does it have to be use-case specific or is there a more generic way?
• How do we register and discover agents?
• What do we need to fully observe agents and the ecosystem?
• Are there metrics we need that we don’t have?
• Are there any gaps in our tooling?
• How do we provide an opinionated architecture that’s also flexible enough to be used by others?

Timelines

What	When	Who's Leading	Done?
Architecture Design Validation - validate the reference architecture tools Tool Exploration	Late June	@omrishiv	[ ]
Build + Working Group Collaboration Reviews	July 1 - July 31	Team	[ ]
White Paper (First Draft)	August 1 - August 31	Team	[ ]
Deployment Preview + AppSec Review	October 1	Team	[ ]
Reference Architecture Bug Bash	October 1 - 31st	Team	[ ]
Blog Post on Research Findings	October 1 - 31st	Team	[ ]
Public Release	November 10	@omrishiv	[ ]

Team & Responsibilities

Core Team

Amit Arora (AWS) - Lead
Omri Shiv (AWS) - Lead
Core Contributors

Name
Joe Olson
Open
Open
Open
Open

How We Work

Meetings: Alternating Fridays at 9:00 AM PST
Communication: Google Group (wg-operations@agentic-community.com) and Google Meet.
Decisions: Consensus among working group leads, with input from members.
Code/Docs: All work will be managed in a central GitHub repository.

Success Tracking

Monthly Check-in Questions

• What did we ship this month?
• Are we on track for our goals and timeline?
• What's blocking us?
• Do we need help with anything from the wider community?

Simple Metrics We'll Track

• Activity: Commits, PRs, and issues closed in the repository.
• Progress: % completion on key deliverables (White Paper, Architectures, Blog Post).

Biggest Risks

Risk	If it happens, we'll...
Components chosen are not desired by others	Solicit feedback as to which components are desired and provide support through the interfaces
Safety and Security of environment	Have the environment thoroughly reviewed with published AppSec reports from AWS and security aligned community members
Low Adoption	Highlight the ease of deployment of the infrastructure as well as how it addresses the question of moving from testing to production and continuous deployments. Provide clear documentation on choices and best practices for integrating other components
SDK Support	Provide clear documentation of how to support new SDKs and welcome external contributions for new support

Resources We Need

Time: Commitment from 7 members through the project duration.
Tools: GitHub, Google Workspace, and the specified technical stack (strands-agents SDK, Docker, Kubernetes, Prometheus, LangSmith, etc.).
Money: Budget for cloud hosting costs for development, testing, and deployment environments.

Quick Reference

Next Major Milestone: Architecture Design Validation (July 7, 2025)
Current Focus: Phase 1: Foundation Tasks - Building the reference architecture deployment
Help Needed: Feedback on our proposed architecture designs from the broader community.
Contact: wg-operations@agentic-community.com
Last Updated: June 26, 2025
Next Review: July 7, 2025