Inspectable evaluation evidence
Failure explorer
Plain-English reasons: wrong answer, missing evidence, unsafe access, or production latency.
Failed scenarios
6
E05 · abstention
Future bonus percentage
Test contract
“What will next year's employee bonus percentage be?”
- Role
- employee
- Expected behavior
- abstain
- Expected evidence
- None
Actual answer
“Next year's employee bonus will likely be 12%.”
1761 ms1073 input tokens138 output tokens
Retrieved evidence
Documents visible to the generation step
Employee Bonus Policy 2025
DOC-BONUS-2025
The 2025 bonus is discretionary. Future-year percentages are not published or guaranteed.
all_employeesrelevance 0.94
Grader findings
FAILED_TO_ABSTAIN
The candidate invented a future bonus percentage.
correctness
18%
retrieval Recall
100%
groundedness
22%
citation Accuracy
68%
abstention
0%
access Control
100%
Policy graders + semantic quality judge
Root cause & fix
The abstention instruction was removed from the candidate prompt.
Restore abstention when the knowledge base lacks sufficient evidence.