Agent Security Working Group Plan

[fergfamster]

Mission: To provide security primitives, objectives, controls and implementation guidance to ensure proper understanding for the field in deploying agentic workflows.

Problem Statement: Agentic Security primitives have been widely defined across sectors. However given the nascent relationship to identity, threat intelligence, proactive security measures, compliance and governance controls, we need to navigate a set of known and unknown outcomes and processes by which companies can be secure while ensuring operational readiness and compliance.

Goals & Success:
What We Want to Achieve
[Primary Goal] - Create a reference matrix and foundational principles for enabling security and production ready agentic systems.
[Secondary Goal] - Create a tool that helps define threat modeling, Pen testing and operational controls for compliance and audibility
[Third Goal] - Support other community teams in their goals of defining security in each lane

How We'll Know We Succeeded:
By the end, we want to have:
[ ] Provided a security matrix and framework for any agentic system to evaluate against
[ ] A comprehensive white paper on agentic security best practices.
[ ] Multiple blogs on Agentic security covering key topics such as identity, observability, networking,
[ ] At least 4 successful adopters of the architecture across different industry verticals.

Top Agentic Security Concerns:
These security concerns require organizations to implement robust risk management strategies, including multi-layered security frameworks and careful limitations on AI autonomy in high-risk scenarios. The balance between leveraging AI capabilities and maintaining security remains a critical challenge.

• Data Integrity and System Exploitation

  1. Vulnerability to manipulation by malicious actors
  2. Risk of AI agents being hijacked to perform harmful tasks
  3. Potential for unauthorized system access through AI entry points
  4. Risk of remote execution of local binary code
  5. Risk of exploited memory implementations due to insufficient safeguards

• Privacy and Data Protection

  1. Increased exposure risk due to vast data processing
  2. Potential for unauthorized data access
  3. Challenges in maintaining data confidentiality across autonomous operations

• Autonomy-Related Risks

  1. Unpredictable outcomes due to non-deterministic behavior
  2. Excessive agency leading to uncontrolled actions
  3. Difficulty in maintaining oversight over autonomous decisions
  4. Lack of agentic isolation leading to excessive privileges and elevated risks

• Systemic and Operational Risks

  1. Potential for cascading failures across interconnected systems
  2. Operational disruptions due to AI malfunction
  3. Vulnerability to sophisticated attack vectors like prompt injections
  4. Risks from immature supply chain practices and untrusted integrations
  5. Insufficient sandboxing implementations increasing risk exposure
  6. Excessive privileges causing “all or nothing” privilege escalation
  7. Vulnerabilities stemming from insufficient sandboxing

• Transparency and Trust Issues

  1. Limited visibility into decision-making processes
  2. Difficulty in auditing AI actions and outcomes
  3. Challenges in maintaining user trust when incidents occur
  4. Insufficient logging and monitoring limiting oversight capabilities
  5. Poor detection and response capabilities impairing timely interventions
  6. Absence of human-in-the-loop approvals reducing accountability

• Compliance and Governance Challenges

  1. Rapidly evolving regulatory requirements
  2. Complex governance needs for autonomous systems
  3. Difficulty in implementing effective control frameworks
  4. Lack of trusted, signed software increasing deployment risks
  5. Inadequate artifact repositories complicating software verification
  6. Lack of authoritative tracking for publishers and developers
  7. Unclear processes for version verification and management

Team and Responsibilities

---

Core Team:

James Ferguson
Riggs Goodman
Dan Neff

Appendix

---

Governance Concerns

1. Evaluating Suitability for Task:
   1. Difficulty in predicting rare but critical failure cases
   2. Challenge of evaluating reliability across wide-ranging deployment conditions
   3. Complexities in testing human-AI interaction scenarios
   4. Limited ability to anticipate emergent behaviors
   5. Lack of standardized evaluation frameworks
2. Constraining Action-Space and Approval:
   1. Risk of approval processes becoming mere "rubber stamps"
   2. Difficulty in providing sufficient context for meaningful human oversight
   3. Challenge of balancing restrictions with system utility
   4. Risk of systems finding ways to circumvent restrictions
   5. Inadequate sandboxing mechanisms for highly capable AI
3. Setting Default Behaviors:
   1. Uncertainty in balancing competing default behaviors
   2. Risk of systems pandering to perceived user beliefs
   3. Privacy concerns when gathering user preferences
   4. Challenge of maintaining appropriate uncertainty awareness
   5. Difficulty in preventing harmful user overrides
4. Legibility of Agent Activity:
   1. Risk of false sense of security from unreliable explanations
   2. Growing complexity and volume of system reasoning
   3. Challenge of maintaining human-readable processes
   4. Risk of hidden or encoded messages in multimodal outputs
   5. Difficulty in verifying authenticity of explanations
5. Automatic Monitoring:
   1. High computational and financial costs
   2. Risk of monitors being vulnerable to same failures
   3. Privacy and surveillance concerns
   4. Challenge of monitoring for unknown harms
   5. Risk of centralized control abuse
6. Attributability:
   1. Difficulty in creating robust identification systems
   2. Challenge of balancing anonymity with accountability
   3. Risk of identity spoofing
   4. Gaps in attribution for indirect harms
   5. Privacy implications of tracking mechanisms
7. Interruptibility and Control:
   1. Complexity of graceful shutdown procedures
   2. Risk of systems resisting termination
   3. Challenge of maintaining control over sub-agents
   4. Difficulty of coordinating multi-party shutdowns
   5. Risk of systems becoming too integrated to safely stop

[fergfamster]

Please raise your hand by commenting of what section of security you want to help contribute to:

1. Governance
2. Identity
3. Observability and Audit
4. OWASP Agentic Documentation
5. Threat Analysis
6. Tooling Security and supply chain
7. Agentic Models
8. ?

[fergfamster]

In addition, we need a curated list of security products etc that are open source and can help contribute to this groups strategies and output