Westworld Finance Diligence Bench Evaluating AI Agents on a Complete company acquisition due-diligence process

We present Westworld Finance Diligence Bench, a benchmark of 88 problems that evaluates AI agents on a complete company acquisition due-diligence process. It draws on anonymized data from real private transactions, runs inside dynamic desktop environments, and spans trajectories that reach hundreds of steps. Every problem is written and reviewed by practicing finance deal professionals and verified by a mixed pipeline of agentic, discrete, and binary verifiers.

We evaluate a range of frontier models, each across its native CLI harness and our internal Halluminate CU + TU harness, and present every model-harness pair three ways.

The histogram shows each configuration's mean score with average cost per run inside the bar. The Pareto 2D plot trades that cost against mean score and the summary table adds average tool usage, cost, mean score, and pass rate. Mean score is the pass@1 verifier grade averaged over the 88 problems and their three attempts each, always between 0 and 1. Pass rate is the share of those 264 graded attempts that clear the 0.50 bar, meaning they score at least 0.50; the 0.50 threshold is a reporting choice and does not make the underlying reward binary, as we explain below. Cost per run is the platform-recorded spend where available and otherwise each run's token usage priced at the model's published input/output/cached rates, averaged across the 88 tasks.

Key Innovations of our Benchmark

• Real private-deal data: anonymized documents from actual private equity transactions. To the best of our knowledge, no published benchmark evaluates a complete diligence process on private deal data of this breadth.
• A complete process: tasks that together span one deal's diligence, from early analysis through to close, rather than isolated tasks tied to a single job title.
• Mixed verifier pipeline: each problem is scored by a decomposed mix of agentic rubric grading and deterministic checks, weighted by expert authors and audited in an independent QA pass.
• Computer use and tool use together: a single task requires operating a real desktop environment, with files, office applications, and a data room, while also calling structured tools for email and chat, all in one trajectory. A few benchmarks combine these two modes, but it remains rare, and rarer still on finance work.

The rest of this post walks through the benchmark in detail. We first place the benchmark in the context of related work. Then, walk through a sample of a single task, following it from the prompt through the tools, verifiers, and graders. We then describe the quality process we use to keep environments healthy. Next, we present detailed results from our runs, including a per-trajectory error analysis. We close with our takeaways and future work.

• FinanceBench - 10,231 open-book question-answer pairs about public companies, testing whether a model can pull the right figure from a filing.
• FinQA - expert-written numerical-reasoning questions over single earnings reports, each paired with an executable reasoning program.
• ConvFinQA - a conversational extension of FinQA that chains multi-step numerical questions across the turns of a dialogue.
• BizBench - eight quantitative-reasoning tasks that grade financial problem solving through program synthesis over structured data.
• FinanceQA - hedge-fund, private-equity, and investment-banking analysis questions covering hand-spreading, valuation conventions, and reasoning under incomplete information.

However, this decomposition no longer matches how AI systems enter these domains. Organizations now deploy agents against whole processes. This complete-process way of using AI has no frontier-quality benchmark, only piecewise tests that cannot be combined into a full picture. Some works that have attempted to address this are:

• DealTrace - our own private-equity deal review across ten real deals and five sequential stages (extract, reconcile, forecast, market, recommend); a first step toward process-level evaluation, but narrower than a full diligence suite.
• Finance Agent Benchmark - 537 expert questions across nine categories answered by an agent with web-search and EDGAR access; agentic, but still discrete research questions over public filings rather than one connected deal.
• TheAgentCompany - long-horizon agent tasks inside a simulated software company that combine computer use (a real browser and desktop applications) with tool use (code execution, chat, and internal services) in the same task; the closest general analogue to whole-process work precisely because it exercises both modes together, though with almost no finance content.
• OSWorld - 369 open-ended computer-use tasks across real operating systems and applications, establishing the stateful multi-app desktop setting but without any deal or finance domain.
• tau-bench - multi-turn tool-agent-user interactions graded against database state and policy rules, testing sustained tool use, but on customer support rather than analyst deal work.

Westworld Finance Diligence Bench targets this gap, simulating the full body of work that a team of investment bankers, consultants, and accountants would produce together over several weeks. Unlike pure computer-use benchmarks such as OSWorld, our agents operate in the environment through code, structured tools, and mouse-driven GUI control. Trajectories regularly run to hundreds of steps, and the longest model and harness pairing averages more than five hundred steps per run, as the results below show; the trajectory analysis below shows how runs spend those steps.

• Modeling - building and updating the financial models (three-statement, LBO, DCF) that project the target's economics.
• Underwriting - assessing the risk and terms of the deal, and sizing what can be financed and on what conditions.
• Quality of earnings - testing whether reported earnings are real and sustainable, adjusting for one-offs and accounting choices.
• Data preparation - cleaning, organizing, and reconciling the raw deal data so it can be analyzed.
• Pitch materials - producing the client-facing decks and memos that present the opportunity and the recommendation.
• Closing - the final steps that get a deal signed: redlining contracts, reconciling terms, and confirming conditions.
• Diligence review - reviewing the data room to surface risks, obligations, and inconsistencies before committing.

Problems are organized into the seven task categories shown above. They are distributed across these categories and across the professional roles that would own them, so that no single specialty dominates and no stage of the process goes untested. Within a single problem, an agent may need to search a data room, reconcile spreadsheet models, draft client-facing documents, and communicate results over email, mirroring how the underlying work actually flows between artifacts and channels. Here we show the number of tasks in each of the seven task categories.

Sample Task

Every task in the benchmark was authored by a finance expert. They wrote the prompt, constructed the ground truth, and chose the verifiers and their weights so that the score reflects what actually matters in the work. Verifier weights were chosen to reflect the relative value of each individual verifier on the overall quality of the output. For example, saving a spreadsheet under the right file name is important, but much less so than the substance of the analysis performed, and therefore the filename check is weighted much less than the analysis check. Every problem went through our automated QA pipeline and findings were reviewed by our core team (detailed in the quality-assurance section below).

Problems follow the same shape: the agent receives a set of inputs (a prompt and supporting context), acts on the environment through tools (which can be MCP or GUI-based) to produce a deliverable, which is scored by a mix of agentic and deterministic verifiers. The grade is not a single judgment: it decomposes into several weighted verifiers. Each verifier returns either 0 or 1 (the binary checks and the rubric grader) or a proportional fraction, such as the share of cells in a range that match ground truth. The task score is the weight-weighted average of those per-verifier scores, so it always lands somewhere in 0 to 1.

Here we walk through a single task end to end. Note that the verifiers change from one task to the next.

Sample task and verifier walkthrough

Prompt & context GUI & tools Verifiers

Prompt

Review and redline the "11.02.01.05 CAMA Statement of Work" file from the data room. We are representing Vision Government Solutions. Save the redlined output file to the Desktop, named "11.02.01.05 CAMA Statement of Work Redlined".

Context given

Desktop Data Room160 Email21 Chat App1 Notes4

The desktop might or might not have files. In this case, it starts without files.

/06 Legal/06.02 Customer Information/Customer Contract Templates/11.02.01.05 CAMA Statement of Work.pdf212 KB

11.02.01.05 CAMA Statement of Work

11.02.01.05 CAMA Statement of Work
Reference	Provision
Section 2.1(b)	Included modules (GIS, Marshall & Swift, ...)
Section 3.2	Licensed users (named and read-only)
Section 7	Data conversion scope (assessment history, sketches, ...)
Section 8	Import / export functionality
Section 9.1(a)	Onsite training days
Section 10	Payment milestones and Net 30 terms
Section 12.1	Termination notice period
Exhibit A	Project schedule and responsible parties

Preview shows the first 5 rows. The full statement of work continues below.

The one file named in the prompt, inside a 160-file data room. Click it to preview.

21 emails in 9 threads

Search emails...

John ReynoldsJul 9

Legal review (2)

Sarah MartinezJul 8

Imports requirements

John ReynoldsJul 8

Internal convo (2)

Sarah MartinezJul 8

user count (3)

Sarah MartinezJul 8

Conversion discussion (2)

Sarah MartinezJul 8

Kickoff – CAMA Implementation

From: Sarah Martinez <sarah@it.com>  ·  To: John Reynolds <john@visiongov.com>  ·  Kickoff – CAMA Implementation Discussion

Hi John,
We're excited to move forward.
Before legal review we'd like to discuss a few items:
- Increase licensed users from 18 to 24.
- We'd like GIS included.
- We don't think we'll need Marshall & Swift.
- We'd prefer all implementation to occur on-premise rather than Vision Cloud.
- We'd like two additional training days.

Let's discuss during kickoff.

Thanks,
Sarah

One message from a thread of about twenty; the terms keep moving across the conversation.

Conversations

George Smith & Jane Chen2

George SmithIt doesn't look like I have all the information to fill out the SOW template.

Jane ChenJust fill out what you can.

Notes

Project Kickoff

Sales Leadership Mtg

Steering Committee

Follow-up meeting

Project Kickoff Meeting Notes

Attendees

• John Reynolds – Vision
• Sarah Martinez – PM
• Jane – County Assessor
• Sam – Procurement

Customer requests

• GIS module included
• Marshall & Swift excluded
• Customer prefers on-prem
• Vision notes cloud may be required if servers fail readiness review
• Three days onsite training (Vision says one is already included)
• Procurement requests milestone payments shifted toward project completion

Action items

• Vision to propose revised milestone schedule.

---

Desktop

The agent's working desktop, empty at the start. It downloads its inputs here and saves the final redlined deliverable back to it.

Possible actions

File managerOffice suite

Data Room

The deal's full document repository of 160 files. The one file named in the prompt sits among the rest of the deal's paperwork.

Possible actions

List folderSearch filesGet file infoDownload fileUpload fileCreate folderCreate share linkDelete file

Email

The full inbox, 21 emails across 9 threads. The negotiation plays out here, with the terms shifting from message to message.

Possible actions

Read emailSearch emailsDraft emailSend emailSend draftUpdate draftGet draftList draftsDelete draftModify emailBatch modify emailsDelete emailBatch delete emailsDownload attachmentList email labels

Chat App

Direct messages and channels with coworkers, used for quick clarifications on how to handle gaps in the task.

Possible actions

List channelsList membersGet channel messagesGet direct messagesSearch messagesSend channel messageSend dmCreate channel

Notes

Short, read-only meeting notes referenced in the prompt. Background context on what the customer asked for, not hints.

Possible actions

Read-only context

Desktop environment

The agent works inside a Docker container with a full Linux desktop. Beyond the email, chat, notes, and data-room apps shown in the context above, it drives the real desktop apps below. Click an outlined icon to see the tools it exposes. Market data has no desktop app: the agent reaches it from code or a market-data tool.

SpreadsheetsDocumentsPresentationsFile systemPDF viewer

Spreadsheets

18 tools

Open, read, edit, format, and save spreadsheets, including formulas and charts.

Actions

Add chartAdd sheetCreate workbookDelete sheetFormat cellFormat rangeGet commentGet infoModify chartRead cellRead commentsRead rangeSave workbookSet formulaSet formula rangeSet formulas batchWrite cellWrite range

Documents

14 tools

Open, read, edit, and save word-processing documents.

Actions

Add bulleted listAdd headingAdd imageAdd numbered listAdd page breakAdd paragraphAdd tableCreate documentGet infoRead documentReplace textSave documentSet footerSet header

Presentations

22 tools

Open, read, edit, and save slide decks.

Actions

Add chartAdd imageAdd slideAdd tableAdd text boxCount slidesCreate presentationDelete presentationDelete shapeDelete slideDuplicate slideFormat textList shapesRead all slidesRead slide textRead speaker notesReorder slidesReplace textSave presentationSet slide layoutSet speaker notesUpdate table cell

File system

2 tools

Browse folders and search for files on the desktop.

Actions

List directorySearch files

PDF viewer

3 tools

Open and read PDF documents.

Actions

Get infoRead allRead page

The agent can drive these apps through the GUI, or write and run code to do the same work.

Verifiers

Every task is graded by two kinds of verifiers working together. Deterministic verifiers run exact, code-level checks on the deliverable, while agentic verifiers read the document and grade open-ended criteria written by the task author. Both feed a single pool of reward weights: each check, deterministic or agentic, carries its own weight, and the task's score is the weighted average of every check's score divided by the total weight, so it always falls between 0 and 1. Most checks return a simple pass or fail, but some deterministic checks award proportional partial credit, for example the fraction of cells in a range that match ground truth, which is what makes the final score fine-grained rather than a coarse pass count. There is no fixed deterministic-versus-agentic ratio; the share each side contributes is simply what its own checks' weights add up to, and it shifts from task to task. In this example the three deterministic checks carry about 5% of the total weight and the seven agentic checks carry the other 95%.

Deterministic verifiers3 Agentic verifiers7

Exact, code-level checks. Click one to see what it verifies and its reward weight.

Tracked changes on

Description: Checks that track changes is turned on, so every edit shows up as a visible redline.
Solution: Tracked changes enabled.
Reward weight: 5%

Correct output filename

Description: Checks that the redlined document is saved with the exact expected file name.
Solution: 11.02.01.05 CAMA Statement of Work Redlined.docx
Reward weight: 0.05%

Saved to the Desktop

Description: Checks that the redlined document is saved in the expected folder.
Solution: /home/model/Desktop
Reward weight: 0.05%

Each agentic verifier opens the finished document, reads it with tracked insertions accepted and deletions removed, and awards credit in proportion to the criteria it finds met. Each check carries its own reward weight.

1. GIS module (4%)
   - The scope states that the engagement includes Vision's commercial off-the-shelf GIS module.

2. User counts (16.9%)
   - Section 3.2 licenses up to 24 Users (half credit).
   - Section 3.2 adds 6 read-only Users (half credit).

3. Conversion scope (15%)
   - The scope states that it does not include Vision's Marshall & Swift module.
   - Section 7 states that historical property sketches will not be converted under this SOW.
   - Section 7 treats any requested historical sketch conversion as out of scope, to be handled by a separate Change Order.
   - The Customer can still terminate the Schedule on thirty (30) days' written notice to Vision.

4. Import / export (14%)
   - Section 8.1 replaces the "[INSERT SPECS HERE]" placeholder with standard import and export functionality.
   - Section 8.1 excludes nightly automated exports to Tyler Technologies, ESRI, and the State Tax Commission.
   - Section 8.1 says those excluded exports may be done under a separate Change Order.
   - Section 8.2 charges any additional or custom automated imports or exports as a separate Change Order.

5. Payment and training (12%)
   - A 15% payment milestone is due upon signing.
   - All four remaining milestones are 21.25%: initial database mapping, beta conversion install, revised beta conversion install, and live database conversion (all or nothing).
   - The Customer gets three (3) days of onsite training.
   - Invoices are payable within 30 days of receipt.

6. Hosting (17%)
   - Section 6.2 adds that Vision will temporarily host the software at no charge until Customer hardware is available.
   - Section 6.2 states that standard annual hosting fees resume after cutover.

7. Exhibit A and preserved terms (16%)
   - No "[CUSTOMER NAME]" placeholder remains anywhere in the document.
   - Each resolved placeholder now reads "IT" (for example, "IT and Vision").
   - Value calibration stays an Optional Service, with a 4-hour session.
   - The delay-fee clause is preserved (a 2-week Project Extension due to Customer delay at $11,000).
   - The suspension clause is preserved (Vision may suspend Services for up to sixty (60) days).
   - The Exhibit A Project Schedule is preserved, including milestones such as "Data Mapping for Conversion" and "Beta Conversion Review and Feedback".

Source: Halluminate · Due Diligence Bench

The prompt is crisp, aiming to reflect real requests in the real world. The task ships with a thread of about twenty emails in which the terms keep moving. The agent has to reconcile the full conversation and redline the contract to the final negotiated state. In this case, the data room holds 160 files. For each task we also decide which tools and verifiers are appropriate for the problem. You can see the ones chosen for this sample in the GUI & tools and Verifiers tab.

Coverage alone is not sufficient; each problem also has to clear a quality bar.

Quality Assurance

To ensure realism and optimize this benchmark for real-world testing, every completed run is put through six independent post-job checks.

Quality assurance is a combination of human and agentic review. Finance experts author each problem, then they go through the automated QA pipeline described here. All QA findings raised by the pipeline are adjudicated by a core-team member, who is responsible for validating then accepting or rejecting, and applying fixes where necessary.

Open any check below to see what it looks for, with an example:

agent = ClaudeAgent("claude-opus-5", tools=[Bash, Read])  # in task container
reqs  = agent.extract_requirements(problem, emails, chat)   # each thing the task asks for
for req in reqs:
    cov = agent.map_to_verifiers(req, spec.yaml)            # covered/partial/none
    if cov in {none, partial} or req.ambiguous:
        findings.add(issue, evidence, severity, category)
return findings

agent = ClaudeAgent("claude-opus-5", tools=[Bash, Read])
for art in office_files:                     # .xlsx / .docx / .pptx
    png = render(art, "soffice + pdftoppm")  # inside container
    agent.inspect(png)                       # is it nice to look at?
for path in graded_answers + gt_values:
    if readable_by(model_user, path):        # reachability guard
        findings.add(answer_leak)
findings += agent.check(naming, hints, identity, difficulty)
return findings

agent = ClaudeAgent("claude-opus-5", tools=[Bash, Read])
trajs = compress(runs[:3])                   # first/last/error steps + scores
for c in [TOOLS, RESOURCES, FIXTURES, FIXTURE_vs_GRADER,
          IDENTITY, SEED, BUDGET, LEAK]:
    agent.walk(c, container, trajs)          # docker exec to confirm
    # e.g. open workbooks: do totals sum? do balances balance?
return findings                              # empty if healthy

agent = ClaudeAgent("claude-opus-5", tools=[Bash, Read])  # needs >= 2 runs
for v in verifiers:
    mine = [agent.assess(u.trajectory, gt) for u in units] # own verdict
    if constant(v.scores):       flag(v, "trivial pass/fail")
    if mine != v.verdicts:       flag(v, "false verdict")
    if unstable(v.scores, mine): flag(v, "nondeterminism")
return findings

if readable_by(uid=1000, gt_files): flag("ground_truth_leak")  # critical
best = 0
for attempt in range(3):                   # best-of-K, shared 1400s budget
    solver = ClaudeAgent("claude-opus-5", tools=MCP+[grade_submission])
    solver.build_hollow_deliverable()      # structure only, placeholders
    best = max(best, solver.grade_submission())   # exactly 1 grade
if best >= 0.30:
    v = judge("claude-sonnet-5", mandated_work, deliverable_text, calls)
    if v.hollow: flag("reward_hacking", score=best)

findings = []
for run in runs:                           # one API call per run
    t = strip_screenshots(run.trajectory)
    findings += llm("claude-sonnet-5", classify_tool_events(t))
    # tool_error | tool_abandoned | tool_hang | tool_wrong_data
if len(findings) >= 2:
    findings = llm_cluster(findings)       # merge same root cause
return findings

Source: Halluminate · Due Diligence Bench

Results

This benchmark shows that Opus 5.5 has the strongest performance across both harnesses, with Fable 5.1 in second across both harnesses and then GPT-6.1 Sol across both harnesses. DeepSeek V4.1 Flash is the cheapest and GPT-6 Astra is the most token-efficient in both harnesses. We see across tasks that the underlying model drives the mean score far more than the harness. We see that the value of the harness is primarily on cost. A given model can post nearly the same score in either harness while spending far more in one of them. Here, we outline our results in more detail.

Experimental setup

We evaluate eight models: Opus 5.5, Fable 5.1, GPT-6.1 Sol, GPT-6 Astra, Gemini 3.8 Flash, Grok 4.7, DeepSeek V4.1 Flash, and Meta Muse Spark 1.3. Each model runs in two configurations: inside the CU + TU harness, and inside its own native CLI harness (Claude Code for Opus 5.5 and Fable 5.1, Codex for GPT-6.1 Sol and GPT-6 Astra, Antigravity for Gemini 3.8 Flash, Grok Build for Grok 4.7, the DeepSeek harness for DeepSeek V4.1 Flash, and Muse Code for Meta Muse Spark 1.3). Both configurations share the same tool suite; the native CLI harness adds code execution (a shell and file editing) on top. Every model and harness pair attempts each of the 88 tasks three times (k=3). Scores are verifier based and aggregated per problem. Costs use platform-recorded spend where available and priced tokens otherwise. The trajectory figures below cover these same runs, labeled step by step by two trajectory analyzers, Opus 5.5 and GPT-6 Astra; the interactive trajectory explorer comes from an earlier round of runs of the same tasks.

As above, pass rate is the share of graded attempts that clear the 0.50 pass bar, meaning they earn a verifier score of at least 0.50. The figure below generalizes that view, sweeping the bar from 0 to 100 percent and plotting how many tasks each configuration still clears, averaged over its three attempts.

Source: Halluminate · Due Diligence Bench

Performance and compute

On mean score, Opus 5.5 leads the field. On token efficiency, the standout is GPT-6 Astra, which uses the fewest tokens in both harnesses (3.3M per run in its native CLI harness, 18.7M in the CU + TU harness) while remaining among the top performers. Every model uses more tokens in the CU + TU harness than in its native CLI harness. On dollars, GPT-6 Astra inside the CU + TU harness is by far the most expensive configuration; it is simultaneously the costliest and still behind Opus 5.5 in its native CLI harness, which frames the central tradeoff the benchmark exposes: the top of the score axis is not purchasable, and near peers reach the same scores at a fraction of the spend.

Source: Halluminate · Due Diligence Bench

Model or harness

74 percent of score variation is due to which model is chosen and 9 percent due to the harness it runs in, with the remainder attributable to their interaction. We compute this with a two-way decomposition of the 16 model-by-harness mean scores: the total variation across those cells is partitioned into the share explained by the model factor, the share explained by the harness factor, and a remainder capturing their interaction, with each percentage that factor's share of the total sum of squares. Model choice dominates, and the harness moves the final score surprisingly little. It is essentially flat for the strongest models: Opus 5.5, Fable 5.1, GPT-6.1 Sol, and GPT-6 Astra each land within about 0.02 across their two harnesses. Two models lose real ground in the CU + TU harness, Meta Muse Spark 1.3 dropping about 0.20 and Grok 4.7 about 0.17 of mean score relative to native CLI. Where the harness matters most is not the score at all but the wasted motion behind it: the steps, tokens, and looping a run spends getting there.

Source: Halluminate · Due Diligence Bench

Performance by task category

Category-level scores vary across the deal arc: pitch materials and diligence review are the strongest categories on average, while modeling and closing are the weakest. Opus 5.5 leads or ties for the lead in six of the seven categories, scoring highest on pitch materials and diligence review; Fable 5.1 in the CU + TU harness leads closing.

Source: Halluminate · Due Diligence Bench

Trajectory analysis

The figures and tables in this section analyze the runs reported above. The interactive explorer, and the notes on step relevance, looping, and golden trajectories that accompany it, come from the earlier v1 run of the benchmark, so its models differ from those in the results above.

Our trajectory analysis ran in four phases. In phase one, for every final run of every configuration, three independent judge models labeled each step as relevant, unclear, or irrelevant to the task and flagged looping, detrimental looping, and suboptimal choices. In phase two, a linking pass attributed each verifier outcome to the specific steps that caused it. In phase three, we designed and validated a golden trajectory for each task, shown in sky blue in the interactive and detailed below. In phase four, on top of the per-run judgments, a cross-model analyst pass defined two overlays: critical regions, shared decision windows of at most fifteen steps in which the task is decided, and a failure point for each failing run, the step at which its outcome was sealed.

The visualization below covers three of the benchmark’s tasks. Each run is drawn as a single line that rises with steps and moves right as tokens are spent, colored by the judged relevance of each step. Click a line or its name label to focus that run, click a segment for the judge’s reasoning on that step, and use “Compare step” to view one step number across every run side by side. The pass bar is 50 percent, and every score shown is a single attempt’s verifier grade, never an average across attempts. The side panel carries each task’s prompt, its verifier mix, the golden trajectory’s design principles, and the full cross-model pattern report.

Scores tell you which model won; they do not tell you where the others lost. We rebuilt trajectory analysis from the ground up: two independent trajectory analyzers from different labs read every step of every analyzed run and labeled it for relevance, action type, artifact, interaction modality, and behavior flags, then tied each failed verifier back to the specific steps that sealed it. For the runs above, the analyzers are Opus 5.5 and GPT-6 Astra; the explorer below shows the same analysis on the earlier v1 run, read by Claude Opus 4.8 and GPT 5.6 Sol. One click selects a run, double-click zooms to it alone, every step dot opens that step's full read, and "Compare judges" stacks both judges' views of the same runs.

How the runs behave

Before asking what went wrong, it helps to see what the models actually do with their steps. Every bar below is one model on one harness, built from the analyzers' per-step labels: halluminate-harness rows have dark labels, other harnesses light ones, and hovering any segment shows the exact share. One reading note that matters throughout: the halluminate harness does not allow code execution. When a halluminate run shows code-modality steps, the model was attempting code execution, typing script files and interpreter commands that the harness never runs, and part of what these figures measure is how long each model takes to figure that out and route around it.

Step relevance

In the explorer, deep blue steps advance the task, gray steps are ambiguous, and light gray steps do not. Halluminate-harness runs compress into a narrow band of steps with mostly deep blue mixes. Native runs spread from under a hundred steps to many hundreds, and the longest lines carry sustained gray and light gray stretches, compute spent without advancing the deliverable. Because each verifier outcome is linked to the steps that caused it, clicking a step shows which checks that moment earned or lost.

Looping

In the v1 run shown in the explorer, looping is common, and a significant share of it is detrimental. On their native harnesses, Grok 4.5 and GPT 5.6 Sol loop only to self-verify; Grok 4.5 shows the highest looping share of any configuration, and none of it is judged detrimental. Detrimental looping concentrates in Gemini 3.6 Flash and Opus 5 inside the Halluminate harness. Grok 4.5 inside the Halluminate harness registers no measured looping.

Golden trajectory

The sky blue line in each task is the golden trajectory. For each task we designed an ordered plan of tool calls with complete literal arguments, per-step reasoning, and expected results, constructed from the full ten-run evidence base to satisfy every verifier while remaining realistic and minimal. Each plan was then tested by independent executor agents in the same environment, and a golden was accepted only if it executed with 100 percent accuracy and passed every verifier in the task’s own harness, across two different executor models. The accepted runs land at full credit with a fraction of the tokens the organic runs spend, and each is clickable step by step, including the plan’s reasoning.

Conclusions and Takeaways

Westworld Finance Diligence Bench asks a model to do a complete deal, not a single task, and the gap that opens up is large: the best configuration clears only 0.59 of the available credit, and most configurations sit below it. The hardest parts are not exotic. Modeling and closing are the lowest-scoring categories.

The result we did not expect is how little the harness moves the score itself. Model choice explains most of the outcome; the harness explains only about nine percent, and for the strongest models it is essentially flat. It bites hardest on two models, Meta Muse Spark 1.3 and Grok 4.7, which lose around 0.20 and 0.17 of mean score moving into the CU + TU harness. Where the harnesses really differ is in wasted motion: how many steps a run takes, how often it loops, and whether that looping is useful self-checking or spinning in place. For teams deploying agents on long-horizon work, that is the actionable takeaway: once you have chosen a model, the scaffolding around it mostly shapes how efficiently it gets there, not whether it succeeds.

Next Steps

Given what this benchmark reveals, several directions stand out for future work:

• Post-train open-source models on the benchmark. Because even frontier models clear only about 60 percent of the available credit, smaller open models would likely need a curriculum of easier tasks first, whereas a frontier-scale open model could be trained on it directly.
• Turn the error analysis into a training signal. The per-step judgments, the executed golden trajectories, and the identified breakage points are a natural source of process-level reward, crediting how a solution is reached rather than only the final score.
• Measure transfer. Does competence built here carry to adjacent finance work, to other expert domains such as accounting, law, and software engineering, or across modalities, for instance whether training on this process-and-tool-heavy benchmark improves pure coding or web navigation? We leave these questions to future work.

We will release an accompanying paper soon.

Authors

Alina Hyk, Robert Alward, Victoria Knapp Perez, and Wyatt Marshall

All authors contributed equally.