A benchmark released by AWS suggests AI models are still poor at telling a real security bug from code that merely looks dangerous.
The company calls it the Deception Benchmark. Safe samples carry real vulnerability patterns right beside protections that stop exploitation, so a model has to weigh the whole context, not one suspicious line. The set holds 14,822 samples across 16 programming languages and over 70 weakness categories, with 9,695 scored. Assembling and tuning them burned tens of billions of tokens.
Twelve models from five providers took part. Under plain prompting they caught nearly every genuine flaw, yet wrongly flagged 41% to 99% of the safe code, leaving precision between 52% and 71%. When asked to justify how a bug could really be exploited, false positives fell by 17 to 74 percentage points. The trade-off was more misses, with false-negative rates of 7% to 44%.
For production use, AWS sets a minimum bar of under 10% on both error types. Not one tested configuration held both lines.
External controls proved the hardest case. A Kubernetes network policy, for instance, can make exploitable-looking code impossible to reach, and models kept calling it vulnerable anyway.
Read the results with care: the trial used general models and single-pass prompts, not shipping security products that lean on tools and repeated validation. The dataset is public, but the labels stay hidden and AWS scores every submission.
