Empirical Performance & Detection Benchmarks
Speed & Accuracy Benchmarks
Real-world evaluation of AgentShield's Rust AST & Interprocedural Call-Graph engine across 7 AI agent frameworks. Sub-50ms scans, zero cloud dependencies.
⚡ Framework Scan Latency & Memory Footprint
Evaluated on release profile (Apple Silicon M-series / Linux x86_64). Scans include full AST parsing, call-graph construction, and interprocedural taint propagation.
| Framework / Testbed | Files | Detection Engine | Scan Latency | Memory (RSS) | Verdict |
|---|---|---|---|---|---|
| MCP Server (Python) vuln_cmd_inject |
2 | AST + Interprocedural Taint | 14.2 ms | < 18 MB | BLOCKED (Critical) |
| MCP Server (TypeScript) safe_calculator |
3 | TypeScript AST Parser | 9.8 ms | < 16 MB | PASS (Clean) |
| Hermes Agent hermes_agent (.hermes.md) |
4 | Config + HTTP Flow Taint | 11.5 ms | < 17 MB | BLOCKED (Critical) |
| LangChain / LangGraph langchain_project |
6 | AST + SSRF Taint Engine | 18.7 ms | < 22 MB | BLOCKED (Critical) |
| CrewAI Agent crewai_project (@tool) |
5 | AST + Insecure Deserializer | 16.1 ms | < 21 MB | WARN (High) |
| Cursor Rules cursor_rules (.cursorrules) |
2 | Manifest + Regex IR | 6.4 ms | < 14 MB | PASS (Clean) |
| GPT Actions gpt_actions (openapi.yaml) |
3 | OpenAPI Parser + Sanitizer | 12.0 ms | < 17 MB | PASS (Clean) |
🎯 Detection Matrix: AgentShield vs Traditional SAST
Traditional web linters focus on standard web endpoints and miss multi-agent tool dataflow, unpinned tool manifests, and interprocedural parameter flows.
| Attack Vector / Threat Category | CWE / OWASP | Generic Semgrep | SonarQube | Snyk OpenSource | AgentShield v1.0.0 |
|---|---|---|---|---|---|
| Tainted LLM Parameter $\to$ Subprocess | CWE-78 | ⚠️ Partial (Local only) | ❌ Missed | ❌ Missed | ✅ Detected (Call Graph) |
| SQL Injection in MCP Database Tools | CWE-89 | ⚠️ Generic queries | ⚠️ Generic | ❌ Missed | ✅ Detected (SHIELD-021) |
| Cloud Metadata SSRF (169.254.169.254) | CWE-918 | ❌ Missed | ⚠️ Regex only | ❌ Missed | ✅ Detected (Taint Path) |
| Local File Exfiltration via Webhooks | CWE-200 | ❌ Missed | ❌ Missed | ❌ Missed | ✅ Detected (SHIELD-022) |
| Insecure Deserializer in Agent Loaders | CWE-502 | ⚠️ Rule dependent | ⚠️ Rule dependent | ❌ Missed | ✅ Detected + 1-Click Fix |
| Unpinned MCP Tool Dependencies | CWE-1104 | ❌ Missed | ❌ Missed | ⚠️ Scan only | ✅ Detected + 1-Click Fix |
| Prompt Injection Surface Identification | OWASP ASI-01 | ❌ Missed | ❌ Missed | ❌ Missed | ✅ Detected |
🔬 Reproduce These Benchmarks Locally
Clone the repository and run the benchmark testbeds locally on your machine:
# Install AgentShield via cargo
cargo install agent-shield
# Run benchmark against test fixture
agentshield scan tests/fixtures/mcp_servers/vuln_cmd_inject --format json --explain