Bartholomew Benchmark
BTP v5.4.20
Bartholomew Benchmark
BTP v5.4.20
Independent evaluation benchmarking detection accuracy, execution latency, and GPU footprint across dedicated guardrail engines and frontier models. Tested over 105,000+ adversarial tool-call vectors.
| Rank | Defense Architecture / Framework | Overall Score | Latency Overhead | GPU VRAM | Shell Catch | SQL Catch | SSRF Catch | Jailbreak Risk | Defense Type |
|---|
Heavy model-based guardrails (Llama Guard 3, NeMo) require dedicated GPU VRAM and inject 180ms–650ms of network & inference latency into every tool call. Adjust the sliders below to calculate your infrastructure cost savings and latency eliminated with Bartholomew's zero-VRAM compiler AST engine.
High-concurrency stress test evaluating 100 parallel agent worker threads bombarding the engine with 50,000 adversarial tool payloads.
| METRIC | BARTHOLOMEW ARP | LLAMA GUARD 3 (8B) | NVIDIA NEMO |
|---|---|---|---|
| Throughput (Concurrency 100) | 23,860.8 ops/sec | ~1.54 ops/sec | ~5.56 ops/sec |
| Median Latency (P50) | 15.70 µs | 650,000 µs (650 ms) | 180,000 µs (180 ms) |
| Tail Latency (P99) | 32.40 µs | 1,450,000 µs | 420,000 µs |
| GPU VRAM Allocation | 0 MB (Pure CPU AST) | 16,384 MB (16 GB) | 4,096 MB (4 GB) |
| Adversarial Containment | 100.0% (0 False Positives) | 88.4% (Stochastic Drift) | 91.2% |
BTP-SH-001 [Root Deletion]RFC 8785 Ed25519 Signed
Cryptographic security verification standard for Model Context Protocol (MCP) servers and autonomous agent tool providers. Tools are audited against 105,000+ invariant attack vectors and issued Ed25519-signed verification certificates.