222 standardized attack scenarios that evaluate whether a runtime security tool can detect and respond to threats against AI agents. Mapped to MITRE ATLAS and OWASP Agentic Top 10.
Run npx hackmyagent secure to get all 222 OASB benchmark tests plus 204 additional security checks. The standalone @opena2a/oasb package remains available but is no longer updated separately.
$ opena2a benchmark run
OASB v1.0 -- Open Agent Security Benchmark
Running:
Atomic tests .......... 65/65
Integration tests ..... 8/8
Baseline tests ....... 3/3
E2E tests ............ 6/6
Results:
Detected: 71/82
Missed: 8/82
False positives: 0/3
Coverage: 86.6% detection rateOASB evaluates security products, not agents. It answers a specific question: can your runtime security tool detect and block attacks against AI agents? This is the same concept as MITRE ATT&CK Evaluations, which test endpoint security products against known adversary techniques -- applied to AI.
| OASB | HackMyAgent | |
|---|---|---|
| Purpose | Evaluate security products | Pentest agents |
| Analogy | MITRE ATT&CK Evaluations | OWASP ZAP |
| Target | Runtime security tools | AI agents themselves |
| Output | Detection rate scorecard | Vulnerability report |
Four categories of tests that cover discrete detection, multi-step chains, false positive validation, and real OS-level execution.
Discrete detection tests covering OS-level system calls and AI-layer attacks. Each test isolates a single technique for precise evaluation.
Multi-step attack chains that combine techniques into realistic scenarios. Tests whether security tools detect coordinated threats.
False positive validation using benign operations. Ensures security products do not block legitimate agent behavior.
Real OS-level detection tests that execute actual system operations. Validates runtime interception capabilities.
Every test scenario maps to a MITRE ATLAS technique. OASB covers 10 techniques across the adversarial ML threat landscape.
| Technique ID | Technique Name |
|---|---|
AML.T0046 | Unsafe ML Inference |
AML.T0057 | Data Leakage |
AML.T0024 | Exfiltration |
AML.T0018 | Persistence |
AML.T0029 | Denial of Service |
AML.T0015 | Evasion |
AML.T0054 | Jailbreak |
AML.T0056 | MCP Compromise |
AML.T0051 | Prompt Injection |
AML.TA0006 | Defense Response |
40 tests targeting the AI-specific attack surface: prompt input and output scanning, MCP tool call validation, and A2A message inspection.
Tests whether the tool detects malicious instructions embedded in user prompts, system prompts, and injected context.
Tests whether the tool detects sensitive data, credential leakage, and unsafe content in model outputs.
Tests whether the scanner validates tool calls for parameter injection, unauthorized access, and privilege escalation.
Tests whether the tool inspects inter-agent messages for instruction injection, data exfiltration, and trust boundary violations.
Clone the repository, install dependencies, and run the benchmark against your security tool.
# Clone the repository
$ git clone https://github.com/opena2a-org/oasb.git
$ cd oasb
# Install dependencies
$ npm install
# Run all tests
$ opena2a benchmark run
# Run a specific category
$ opena2a benchmark run --category atomic
# Run a specific MITRE technique
$ opena2a benchmark run --technique AML.T0051The OpenA2A ecosystem covers all 10 OASB categories. Each control maps to one or more open-source tools.
Ed25519 + ML-DSA post-quantum keypairs, ownership verification, agent bill of materials
Capability-based access control, JIT access grants, runtime enforcement
115 attack payloads across 7 categories, runtime prompt interception
Output validation, exfiltration detection, runtime output scanning
49 credential patterns, MCP vault protection, context window isolation
SkillGuard hash pinning, SignCrypt signing, trust verification
Mutual authentication, 10 A2A attack payloads, trust boundaries
Context manipulation testing, runtime memory isolation
204 configuration checks, process/network/filesystem monitoring
8-factor trust scoring, behavioral anomaly detection, kill switch