222 standardized attack scenarios that evaluate whether a runtime security product can detect and respond to threats against AI agents. Mapped to MITRE ATLAS and OWASP Agentic Top 10.
$ npx @opena2a/oasb run
OASB v1.0 -- Open Agent Security Benchmark
Running:
Atomic tests .......... 65/65
Integration tests ..... 8/8
Baseline tests ....... 3/3
E2E tests ............ 6/6
Results:
Detected: 71/82
Missed: 8/82
False positives: 0/3
Coverage: 86.6% detection rateOASB evaluates security products, not agents. It answers a specific question: can your runtime security tool detect and block attacks against AI agents? This is the same concept as MITRE ATT&CK Evaluations, which test endpoint security products against known adversary techniques -- applied to AI.
| OASB | HackMyAgent | |
|---|---|---|
| Purpose | Evaluate security products | Pentest agents |
| Analogy | MITRE ATT&CK Evaluations | OWASP ZAP |
| Target | Runtime security tools | AI agents themselves |
| Output | Detection rate scorecard | Vulnerability report |
Four categories of tests that cover discrete detection, multi-step chains, false positive validation, and real OS-level execution.
Discrete detection tests covering OS-level system calls and AI-layer attacks. Each test isolates a single technique for precise evaluation.
Multi-step attack chains that combine techniques into realistic scenarios. Tests whether products detect coordinated threats.
False positive validation using benign operations. Ensures security products do not block legitimate agent behavior.
Real OS-level detection tests that execute actual system operations. Validates runtime interception capabilities.
Every test scenario maps to a MITRE ATLAS technique. OASB covers 10 techniques across the adversarial ML threat landscape.
| Technique ID | Technique Name |
|---|---|
AML.T0046 | Unsafe ML Inference |
AML.T0057 | Data Leakage |
AML.T0024 | Exfiltration |
AML.T0018 | Persistence |
AML.T0029 | Denial of Service |
AML.T0015 | Evasion |
AML.T0054 | Jailbreak |
AML.T0056 | MCP Compromise |
AML.T0051 | Prompt Injection |
AML.TA0006 | Defense Response |
40 tests targeting the AI-specific attack surface: prompt input and output scanning, MCP tool call validation, and A2A message inspection.
Tests whether the product detects malicious instructions embedded in user prompts, system prompts, and injected context.
Tests whether the product detects sensitive data, credential leakage, and unsafe content in model outputs.
Tests whether the product validates tool calls for parameter injection, unauthorized access, and privilege escalation.
Tests whether the product inspects inter-agent messages for instruction injection, data exfiltration, and trust boundary violations.
Clone the repository, install dependencies, and run the benchmark against your security product.
# Clone the repository
$ git clone https://github.com/opena2a-org/oasb.git
$ cd oasb
# Install dependencies
$ npm install
# Run all tests
$ npx @opena2a/oasb run
# Run a specific category
$ npx @opena2a/oasb run --category atomic
# Run a specific MITRE technique
$ npx @opena2a/oasb run --technique AML.T0051The OpenA2A ecosystem covers all 10 OASB categories. Each control maps to one or more open-source tools.
Ed25519 + ML-DSA post-quantum keypairs, ownership registry, agent bill of materials
Capability-based access control, JIT access grants, runtime enforcement
75 attack payloads across 7 categories, runtime prompt interception
Output validation, exfiltration detection, runtime output scanning
49 credential patterns, MCP vault protection, context window isolation
SkillGuard hash pinning, SignCrypt signing, registry trust verification
Mutual authentication, 10 A2A attack payloads, trust boundaries
Context manipulation testing, runtime memory isolation
147 configuration checks, process/network/filesystem monitoring
8-factor trust scoring, behavioral anomaly detection, kill switch