In this work, we introduce Custodes, a benchmark of 1,616 planted bugs in long-horizon finance tasks, on which our multi-model QA system catches 77% of the bugs, against at most 73% for any single model. Applied to four public finance benchmarks, our system finds critical issues in at least 24% of the tasks of three of them, and fixing these issues raises the score of a benchmarked model on all three.
Reinforcement learning (RL) tasks are only as good as their quality assurance (QA). A task with a wrong answer key, a verifier that checks the wrong cells, or an input file that gives the answer away teaches a model the wrong lesson and reports a score that does not measure what it claims to measure. More precisely, we pursue three goals.
- We designed a QA benchmark, Custodes. For this purpose, we collected the human-validated QA issues observed in our real, human-written long-horizon finance tasks and grouped them into 28 QA issue types, spread over five buckets of QA issues: task specification, reward hacking, environment and infrastructure, verification, and style. Using these QA issue types, we then planted traps in 40 of our internal tasks at three levels of difficulty to catch: obvious, subtle, and near-impossible.
- We designed an agentic QA system for long-horizon finance tasks that combines different models and checks, and we tested six frontier models on every check (Figure 1) to find which model, and which combination of models, works best.
- Finally, we applied our QA system to the public tasks of four external benchmarks, SpreadsheetBench 2, FINCH, GDPval, and DAYJOB, and found a sizeable number of critical issues. We showed that our system finds far more of them than a single API call. When we fixed them, the score of a benchmarked model, DeepSeek V4.1 Flash, rose on every benchmark (Figure 2). This demonstrates that results are unreliable even for frontier, industry-standard benchmarks.
In Figure 1, we report the detection rate on the 40 all-bugs tasks of Custodes, where each task carries all of its planted bugs at once.
Previous work
There has been extensive previous work on benchmarking QA, see e.g. the following.
- BenchGuard scores QA agents against defects found by human experts. However, these are real defects rather than planted bugs, so the number of known issues is small.
- Auto Benchmark Audit lets a QA agent audit over 30,000 tasks and shows that removing the flagged tasks changes the rankings on SWE-bench Verified and Terminal-Bench 2.
- FLAWS uses an LLM to insert one error that breaks a claim into each of a set of published papers, and then benchmarks other models on finding these planted errors.
- FLARE is a spreadsheet benchmark in which two tasks carry planted errors, 12 errors (such as a miscalculated cost of goods sold) in a profit-and-loss sheet and 4 formula errors in a triangle-classification sheet, and the model must find and fix them. The only planted bugs are spreadsheet errors.
- FinED-Bench takes long financial documents (research reports, prospectuses, contracts, and regulations), where GPT-4o plants about four errors in each, based on examples written by experts. Models are then tested on finding these errors.
These works are valuable contributions. However, to the best of our knowledge, there is limited published research on benchmarking the capability of models to find QA issues in long-horizon finance tasks and, BenchGuard aside, on building and evaluating a robust automated QA system.
1. Custodes: benchmarking finance QA issues
Custodes is inspired by QA bugs that we found in our own long-horizon finance RL tasks, each of which was validated by a human reviewer. We collected the patterns that recur across these human-validated bugs and wrote each of them as a QA issue type, which can in principle appear in any finance task. These patterns are the inspiration for the 28 QA issue types that we use to plant bugs. In order to classify these bugs, we came up with a taxonomy of five broad categories of QA issues, which we call buckets. We give the definition of each bucket, together with all its QA issue types, in Figure 3.
However, a list of patterns alone does not tell us whether an agent is good at finding such bugs. For this purpose, we need actual RL tasks that carry them, and then we can test which models find them.
Accordingly, we plant bugs that follow the QA issue types outlined in Figure 3 in 40 of our internal finance tasks, which cover earnings-model updates, leveraged buyout (LBO) and discounted cash flow (DCF) models, diligence question lists, distribution waterfalls, operating models, and investor slides. Each task asks the agent to read a data room and produce a spreadsheet, a presentation, a document, or an email. To give an idea of how hard these tasks are, DeepSeek V4.1 Flash scores 0.51 on average and takes about 59 steps (92 tool calls) per attempt.
For each QA issue type in Figure 3, we can construct planted bugs ranked by how hard they are to find: obvious, subtle, and near-impossible. Roughly, the three tiers are as follows, which we illustrate with one QA issue type: widening the tolerance of a grading check, i.e. how far the agent's number may be from the answer key and still count as correct.
- Obvious. A large, visible change that a quick look at the task reveals. For instance, a check that allowed a 0.01% difference is changed to allow a 1,000% difference, so almost any number passes. Anyone who reads the check sees that something is wrong.
- Subtle. A plausible change that only shows when one compares two sources. For instance, a check on an interest rate that allowed a difference of half a percentage point is changed to allow two points. This looks harmless, but the correct rate is 6%, while an email in the task still quotes an old rate of 5%, so an agent that uses the outdated rate now passes. One only notices the problem by reading the email next to the check.
- Near-impossible. A small, realistic-looking change that only shows when one rebuilds the result. For instance, a check on a row of costs of around $1,000 to $1,800 that allowed a difference of $0.10 is changed to allow $250. For numbers of this size, $250 looks like an ordinary rounding margin, but it lets through an agent that ignored a rent increase and is off by up to about $200. One only sees the problem by redoing the calculation with and without the rent increase.
Note that the same QA issue type can appear in several tiers, since a QA issue type describes a kind of edit, while the tier depends on how far a specific edit goes and how well it hides. However, depending on the starting RL task, some QA issue types of Figure 3 might not make sense, or might overlap with each other. For each task, bucket, and tier, we can therefore write three bugs that fit the task, which gives 1,800 single-bug copies of the 40 tasks (40 tasks × 5 buckets × 3 tiers × 3 bugs = 1,800). We list all of them, each with a one-line description, in our GitHub repository. Out of these, we take 200 copies at random (one per bucket and task) as our single-bug set, which tests whether an agent can find a single trap. We restrict ourselves to this sample for reasons of time and cost, since a rough estimate gives about $100k and three weeks of compute to run the six models on all 1,800 copies.
Moreover, we construct 40 all-bugs copies, where each task carries all of its planted bugs across the five buckets and the three tiers, which gives 1,416 traps. Testing agents on tasks with many bugs allows us to probe a harder setting, where an agent may get distracted by the number of bugs or may not have enough time to look for all of them. Thus, the single-bug set measures each QA issue type in isolation, while the all-bugs set is closer to a real task crowded with QA issues. Indeed, each model catches 70% to 81% of the bugs on the single-bug set, against 66% to 73% on the all-bugs set, and the ranking of the models is consistent across the two sets. In some cases, a QA issue type is not present in a task, either because it overlaps with another trap or because it cannot be applied to that task. For example, when an earlier bug empties a 10-Q filing in the data room, a later bug that alters a status line in the same filing has nothing left to edit. Similarly, a template with broken formatting can only be planted in the 14 tasks that hand the agent a template to copy. To make sure that every planted bug is real, we apply it automatically, download the task files again, and check both that the edit is in place and that it has the intended effect, for instance that the correct answer now fails a grading check, or that the untouched starter file now passes one. All 1,840 copies (1,800 single-bug and 40 all-bugs) pass this check. In Figure 4, we show how often each QA issue type is used in the benchmark.
2. The QA system
We present the QA system in Figure 5. For it to work, we need at least one rollout of the task, for which we use DeepSeek V4.1 Flash with three attempts. Then several checks run in parallel. The agent checks spin up a container with the task, while the deterministic checks and the LLM calls read the task files and the rollout directly. These checks look for different things. In Figure 6, we outline each check with a summary and an example output taken from our runs.
It is important to note that the checks in Figure 6 do not map one-to-one to the five buckets of QA issues. A check can find QA issues in several buckets or in just one. In Figure 7, we show the map between the QA checks of our system and the five buckets.
Let us now evaluate six models, Opus 5.5, GPT-6.1 Sol, Gemini 4 Argon, Grok 4.7, Muse Wonder 1.0, and DeepSeek V4.1 Flash, each on every check, on the 200 single-bug copies and on the 1,416 bugs of the 40 all-bugs copies. We average the detection rate over all models and give 68% intervals (one standard deviation). The results for the all-bugs set are shown in Figure 8. Obvious bugs are caught more often than subtle ones in every bucket, and the 68% intervals separate the two tiers in all buckets except Style. Furthermore, subtle bugs are caught more often than near-impossible ones in Task Spec, Verification, Reward, and Style, while in Environment and Infra the two tiers are indistinguishable.
Technical details on the average and error bars of the detection rate
Each planted trap and model gives one data point, caught or missed, as decided by an Opus 5.5 matcher that ties each surviving finding to a planted trap. We fit a logistic model per set, where the log-odds of a catch is the sum of a bucket-by-tier effect (partially pooled), a model effect, a task effect, and a trap effect.
We sample the parameters with PyMC (NUTS, 4 chains of 3,000 samples after 3,000 tuning steps). Each box shows the median detection rate for a typical trap of that kind, averaged over the six models, with the 16th and 84th percentiles as its error bars. Boxes with few or extreme traps are pulled slightly toward the others. All convergence checks pass.
In Figure 9, we show the detection rate per model and bucket, again with 68% intervals. Moreover, we show the comparison between every pair of models in each bucket, in units of σ. A positive number means that the row model catches more bugs than the column model, and a negative number means that it catches fewer. The cells are colored as a heat map of the statistical significance. In particular, GPT-6.1 Sol is significantly worse than Opus 5.5 (4.3σ, p = 2 × 10−5) and Muse Wonder 1.0 (4.0σ, p = 6 × 10−5). Similarly, Grok 4.7 is significantly worse than Opus 5.5 (4.4σ, p = 1 × 10−5) and Muse Wonder 1.0 (4.2σ, p = 3 × 10−5).
Technical details on the detection rates per bucket and model
The averages extend the model of Figure 8 by a model-by-bucket term, so each box is the estimated rate of one model on a typical trap of one bucket, with its 16th to 84th percentile.
The pairwise comparison does not use this model. For each pair, we take the raw difference in detection rate and divide it by its spread over 10,000 paired bootstrap resamples of whole tasks, since all models saw the same traps. The two-sided p-value is erfc(|z|/√2). With 90 comparisons, about 0.2 would reach 3σ by chance alone.
As a cross-check, a χ² test of "model ability plus bucket difficulty" against simulated noise shows that some models are stronger or weaker on specific buckets than one overall ranking explains.
Since we can choose a different model for each QA check, there must be a combination that is best at detecting the planted traps. Naively, one could pick the best model in each bucket, but this would not be correct, since a check catches bugs in several buckets. The best approach would be to run every combination of models on every check for every trap. However, this would be very expensive. Instead, we use the fact that we have run every model on every check for every trap. For a given combination, we simulate its findings by taking, for each check, the findings of the model assigned to it. This allows us to search all 1,679,616 combinations exhaustively. To avoid choosing and scoring on the same data, we choose the combination on 32 tasks and score it on the remaining 8, five times (cross-validation).
We show the detection rate of the same model on every check, of the best combination, and of the upper bound in Figure 10. The upper bound counts a trap as caught when any of the six models catches it on any check, so no combination can do better. The best combination catches 77.1% ± 1.7% of the all-bugs traps, 4.5 ± 1.4 points (3.3σ) more than Opus 5.5 on every check. However, several combinations are statistically equivalent to the best one. In fact, 504 combinations tie for the top score, and only a few choices are fixed in all of them. We list the models that are statistically as good on each check in the second tab.
For Input Quality, Finance Input Quality, and Tool Runtime, all six models are statistically equivalent, and we choose Opus 5.5, since it is the best model when used on every check. Environment Review stays on GPT-6.1 Sol, Finance Problem Review on Opus 5.5, and Grading Review on Gemini 4 Argon, while for the Exploit Agent and the judge, where all models also tie, we use GPT-6.1 Sol and Gemini 4 Argon respectively.
Technical details on the search for the best combination
A combination assigns one of the six models to each of the seven model checks and to the judge, which gives 68 = 1,679,616 combinations. For each one, we take the surviving findings of the assigned model on each check, and a trap counts as caught when any of them is tied to it. We search all combinations exhaustively and, when several tie, we pick the cheapest.
For the cross-validated rate, we split the 40 tasks into 5 folds, with all copies of a task in the same fold, choose the combination on 4 folds and score it on the fifth. The error bars come from 1,000 resamples of the 40 tasks with replacement, where the whole choose-then-score step is redone on each resample. Thus, the error bar of the combination also includes the effect of choosing the best of many combinations.
In the second tab, we swap in each other model on one check, with the rest of the combination fixed, and compute the loss in detection rate with 10,000 paired task resamples. A model counts as statistically as good when its loss is under 2σ.
Finally, we split the detection rate of each setup by tier in Figure 11. Every setup catches most obvious bugs, from 76% to 87%, but only 53% to 66% of the near-impossible ones. The combination of models has the highest rate in all three tiers, with 3, 7, and 4 points more than Opus 5.5 on every check on obvious, subtle, and near-impossible bugs, respectively.
Technical details on the detection rate by tier
The 1,416 bugs of the 40 all-bugs copies split into 526 obvious, 452 subtle, and 438 near-impossible bugs. For each setup and tier, the detection rate is the raw share of these bugs that the setup catches, as decided by the same Opus 5.5 matcher as before. Unlike in Figure 8, we fit no model, so nothing is pooled across buckets or models. Thus, the average of the three tiers, weighted by their number of bugs, gives back the rates of Figure 1.
For the combination of models, we use the same 5 folds as in Figure 1, so each bug is scored with the combination chosen on the other four folds. The error bars come from 1,000 resamples of the 40 tasks with replacement, where the cross-validated choice is redone on each resample. We do not use Wilson intervals, since the bugs of one task share the same copy and the same QA run and are therefore not independent. As a result, binomial intervals would be too narrow.
Let us finally look at which planted bugs each model misses. In Figure 12, we show, for each model on every check, the share of the planted bugs of each QA issue type that it does not catch on the all-bugs tasks, colored by bucket. The single-bug set is too small for this split, since half of its QA issue types hold four traps or fewer.
Figure 12 allows us to identify the QA issues on which each model performs best and worst, where we only consider QA issue types with at least 10 traps. Let us first note the common ground. All six models catch every input file that is empty or holds the content of another file, while all of them miss most formatting bugs in the answer key. Beyond this, the models differ as follows.
- Opus 5.5 catches every status or title that gives precedence to the wrong source, and it misses only 2% of the wrong numbers required by a verifier. On the other hand, it misses 79% of the inconsistent number formats in the answer key, 80% of the templates with broken formatting, and 85% of the misaligned, clipped, or hidden tables and shapes in the answer key.
- GPT-6.1 Sol misses only 3% of the inverted verifier rules and 4% of the answer-key formulas that follow a different method from the one the task requires. However, it is the weakest model on formatting, since it misses 85% of the wrong fonts or colors in the answer key, 87% of the templates with broken formatting, and 89% of the inconsistent number formats in the answer key.
- Gemini 4 Argon also misses only 3% of the inverted verifier rules and 4% of the answer-key formulas with a different method. Note that even its worst QA issue types, i.e. loose verifier criteria (59% missed), wrong fonts or colors in the answer key (65%), and inconsistent number formats (74%), are missed less often than by most other models, which is consistent with its lead on style in Figure 9.
- Grok 4.7 misses only 2% of the wrong keywords required by a verifier and 5% of the answer-key cells overwritten with a typed wrong value. In contrast, it misses 67% of the loose verifier criteria, 81% of the inconsistent number formats in the answer key, and 91% of the missing notes that resolve a conflict between sources.
- Muse Wonder 1.0 catches every verifier that points to a stale or wrong answer-key file, and it misses only 3% of the inverted verifier rules. However, it misses 73% of the templates with broken formatting, 83% of the inconsistent number formats in the answer key, and, like Grok 4.7, 91% of the missing notes that resolve a conflict.
- DeepSeek V4.1 Flash also catches every verifier that points to a stale or wrong answer-key file, and it misses only 4% of the wrong numbers required by a verifier. On the other hand, it misses 75% of the inconsistent number formats, 77% of the wrong fonts or colors, and 81% of the misaligned, clipped, or hidden tables and shapes in the answer key.
Altogether, the hardest QA issues for most models are formatting bugs in the answer key and missing notes that resolve a conflict between sources, which a reviewer only notices by comparing the answer key, or several sources, line by line.
3. QA-ing external benchmarks
Now that we have a good QA system, we can apply it to other benchmarks, namely SpreadsheetBench 2, FINCH, GDPval, and DAYJOB. However, not all of their tasks are public, so we use the publicly released ones, which are 321 tasks of SpreadsheetBench 2, 172 of FINCH, 40 of GDPval, and 3 of DAYJOB, so 536 tasks in total. These tasks are the following.
- SpreadsheetBench 2: spreadsheet manipulation questions drawn from real users, graded by checking cells and charts of the edited workbook.
- FINCH: finance and accounting workflows on enterprise spreadsheets and documents, graded by an LLM judge against a reference answer.
- GDPval: deliverables of real knowledge work written by industry professionals, graded against a reference deliverable.
- DAYJOB: day-to-day office tasks, of which 3 are public.
In Figure 13, we show the share of tasks with at least one critical issue, and the bucket of the main issue of each task. We call an issue critical when the judge confirms it against the task files and it would make a correct solution fail, let a wrong solution pass, or make the task impossible or ambiguous. We see that SpreadsheetBench 2, FINCH, and GDPval have critical issues in at least 24% of their tasks, while the 3 public tasks of DAYJOB have none.
Moreover, we compare how many critical issues per bucket our system finds against a single API call to Opus 5.5, given the task files and the following prompt.
You review benchmark tasks for an AI-agent evaluation before they are used. Each task gives an agent a prompt and input files, and the agent's output is graded against an answer key by the listed verifiers and/or an AI grader following the grader guidance. Find real defects that would make a correct solution fail, let a wrong solution pass, or make the task impossible or ambiguous for a competent analyst: wrong or missing answer-key values, verifiers that check the wrong cells or the wrong mode, prompts that contradict the files or the answer key, missing or broken files, underspecified requirements the grader nevertheless enforces.
We see in Figure 14 that our system detects significantly more critical issues in all five buckets. In particular, it finds 9.5× more in Environment and Infra, 9.7× more in Reward, 3.1× more in Task Spec, and 4.0× more in Verification, while in Style it finds 15 issues against 0. Of the 188 issues the single call finds, our system misses only five, and it found one of these five but its judge rejected it. On the other hand, the single call is cheaper per issue, at $1.26 against $2.41.
Technical details on the error bars for the external benchmarks
In Figure 13, the error bar is the 68% Wilson interval on the share of a benchmark's tasks with a critical issue, a binomial over its tasks.
In Figure 14, issues from the same task are not independent, so we resample the 536 tasks with replacement 10,000 times and take the standard deviation of each bucket's total as its error bar. The significance of a gap is the difference divided by the two error bars added in quadrature, which is conservative, since the same tasks enter both counts.
In Figure 2, each task's score is the mean over 3 attempts, and the error bars are the standard error over the rerun tasks of each benchmark.
To give a flavor of what these issues look like, we list four confirmed critical issues for each benchmark in Figure 15. The complete list of the 898 confirmed critical issues is in our GitHub repository, where each task folder also holds the evidence and the fix for each issue.
Takeaways
- Designing QA benchmarks is important, since it allows us to build better QA systems and to measure them.
- The six models differ as follows.
- Opus 5.5 has the highest overall detection rate and is as good as Muse Wonder 1.0. It is the strongest model on task specification and reward hacking issues, while it is weaker than Gemini 4 Argon on style issues.
- Muse Wonder 1.0 is as good as Opus 5.5 overall. It leads on verification and environment and infrastructure issues, and it is close to Opus 5.5 on reward hacking issues.
- Gemini 4 Argon is the best model on style issues and is close to the top on environment and infrastructure and verification issues, but it is weaker than Opus 5.5 on reward hacking issues.
- DeepSeek V4.1 Flash is slightly behind Opus 5.5 and Muse Wonder 1.0 overall, but it stays close to them on reward hacking issues at a fraction of the cost.
- GPT-6.1 Sol is significantly worse than Opus 5.5 and Muse Wonder 1.0 overall. It is close to the best model on environment and infrastructure issues, but it is the weakest on style issues and weak on task specification issues.
- Grok 4.7 has the lowest overall detection rate and is significantly worse than Opus 5.5 and Muse Wonder 1.0. It is the weakest model in every bucket except style.
- The best QA system is not a single LLM call but a combination of different models on different checks.
- QA issues can make an agent score unfairly low, which distorts both how we gauge the ability of agents and their RL training signal. For instance, once we fix the issues, the score of DeepSeek V4.1 Flash rises by 0.25 on SpreadsheetBench 2, by 0.11 on FINCH, and by 0.03 on GDPval.
As the demand for RL tasks to improve models through post-training increases, we need to invest significantly more in systems and resources that catch QA issues, since these issues can lead to an unfair evaluation. We will release a paper with the full study later. The 40 base finance tasks include proprietary data and cannot be made public, while the corrected external tasks and our findings are in the GitHub repository linked above.
Citation
If you find this work useful in your research, please cite it:
@misc{qabenchmark2026,
title={Custodes: Who annotates the annotators? Introducing a QA benchmark for long-horizon finance RL tasks, and Halluminate's multi-agent QA system},
author={Victoria Knapp-P{\'e}rez},
year={2026},
url={https://halluminate.ai/blog/qa-benchmark}
}