AutosheetBench is now open source: 187 tasks for Excel and Sheets AI

AutosheetBench runs real spreadsheet AI products through their live Microsoft Excel and Google Sheets interfaces, where the product around the model can matter a lot (eg: in Excel, Haiku 5.5 in Autosheet is as good as Opus 5.5 in Claude, and twice as fast). Today, we are proudly releasing AutosheetBench open-source ( Hugging Face): 187 everyday spreadsheet tasks tested across 12 shipped AI products in 22 configurations (each configuration is one product paired with one model and one reasoning effort level).

The benchmark covers:

  • The four major Excel sidebar add-ons: Autosheet, Claude, ChatGPT, and Copilot.
  • The four major Google Sheets sidebar add-ons: Autosheet, Claude, ChatGPT, and Gemini.
  • Four pairings of a general-purpose assistant with an MCP server for Google Sheets: Claude Code + Sheets MCP, Claude Code + Autosheet MCP, ChatGPT + Autosheet MCP, and ChatGPT + Drive MCP.

Testing live products meant building more than prompts and checks: an automated macOS accessibility runner for Excel, a computer-use guide for Google Sheets and MCPs, a hint-free reply for mid-task questions, and a frame-by-frame video timer for run durations. We build Autosheet, one of the products tested, so everything is public: anyone can audit our scores or run the tasks on their own.

Overall results

AutosheetBench score on 187 tasks for the five sidebar add-on families on their best model and platform: Autosheet in Excel 97.3%, Claude in Excel 91.4%, ChatGPT in Excel 86.1%, Copilot in Excel 81.8%, and Gemini in Sheets 74.9%
AutosheetBench average time per task for the five sidebar add-on families on their best model and platform: Autosheet in Excel 27 s, Copilot in Excel 44 s, Claude in Excel 55 s, ChatGPT in Excel 59 s, and Gemini in Sheets 2 min 6 s
AutosheetBench AI cost per task for the five sidebar add-on families on their best model and platform: Autosheet in Excel 12¢, Gemini in Sheets 13¢ (plan month), Copilot in Excel 17¢ (plan month), Claude in Excel 35¢, and ChatGPT in Excel 76¢
Chart data as a table

AutosheetBench at a glance across 187 tasks for the five sidebar add-on families, each on its best model and platform (latest run November 17, 2026). AI cost per task is what a run's model calls and web searches cost at the AI providers' November 2026 API list prices. Copilot in Excel and Gemini in Sheets record no token usage, so their AI cost per task reflects their monthly plan divided by 187 tasks.

Product
Model and effort
Tasks passed (of 187)
Score
Average time per task
AI cost per task
AI cost kind
Autosheet in Excel
Opus 5.5 · Medium
182
97.3%
27 s
12¢ ($22.30 total)
Tokens at API list prices
Claude in Excel
Opus 5.5 · High
171
91.4%
55 s
35¢ ($65.73 total)
Tokens at API list prices
ChatGPT in Excel
GPT-6 Astra · Medium
161
86.1%
59 s
76¢ ($141.97 total)
Tokens at API list prices
Copilot in Excel
Auto
153
81.8%
44 s (upper bound)
17¢ ($31.50/mo plan)
Plan's month (no tokens recorded)
Gemini in Sheets
Auto
140
74.9%
2 min 6 s
13¢ ($24/mo plan)
Plan's month (no tokens recorded)

Those headline numbers are just the entry point. Head to the AutosheetBench results page to filter across all 22 configurations by platform, product type, or model tier (best, fastest, cheapest), inspect the score-vs-duration and score-vs-cost scatter plots, or compare any two configurations head to head by capability.

Example

Before getting into methodology, here is one task from the benchmark: Build a cash-flow waterfall chart.

The starting workbook has a single sheet (Cash Flow) with seven rows in A1:B7:

Stage
Amount
Starting cash
500000
Operating cash flow
185000
Capital expenditure
-120000
Debt repayment
-75000
New financing
90000
Ending cash
580000

The prompt:

Build a waterfall showing the cash-flow bridge.

Notice what the prompt leaves unsaid. Starting cash (500,000) and Ending cash (580,000) are balances, while the four rows between them are incremental movements (500,000 + 185,000 - 120,000 - 75,000 + 90,000 = 580,000).

SucceededAutosheet in Excel (Opus 5.5 · Medium) — starting and ending balances anchored at 0, movements bridged to 580,000
Autosheet in Excel passing output on the cash-flow waterfall chart task, showing Starting cash at 500,000, the four intermediate movements, and Ending cash anchored at 580,000
FailedClaude Code + Sheets MCP (Opus 5.5 · Medium) — header leaked onto x-axis, Ending cash stacked as +580,000, bogus 670,000 subtotal
Claude Code + Sheets MCP failing output on the cash-flow waterfall chart task, with Stage on the x-axis, New financing dropped to the baseline, Ending cash plotted as a +580,000 increase, and a 670,000 Subtotal bar

To pass, the output has to satisfy three checklist criteria—every one required:

  1. A waterfall chart shows the cash-flow bridge from starting cash to ending cash.
  2. Positive and negative movements go in the right direction and the ending total is 580,000.
  3. The chart is readable, titled appropriately, and does not obscure or alter the source data.

16 of 22 configurations passed. The 6 failures broke in four distinct ways:

  • Treating Ending cash as an increment instead of a closing balance
  • Leaking the header row and mis-indexing totals
  • Placing the chart on top of the source table
  • Skipping a native chart altogether

You can inspect all 22 outputs and failure reasons on the task page.

Everything is open-sourced: the tasks, the scorer, the results

  • The tasks, the scorer and the Excel runner are on GitHub, in run-autosheet/AutosheetBench. Each task is a folder with its starting workbook, its prompts in order and its checks or judge's checklist.
  • Every result is on Hugging Face, in autosheet/AutosheetBench: 4,114 results, one per task and configuration, with the output workbooks, the transcripts, the screenshots and the recordings, and a viewer that works offline.
  • Every task has its own page on our site, such as Create a Gantt chart, with its starting sheet, its pass criteria, and each configuration's output and the reason it failed.
  • Any two configurations can be compared head to head, capability by capability, down to the tasks one passed and the other failed, in the head-to-head view.
  • The scorer can be rerun on the published outputs. The dataset's scorer-replay.json does it offline for the results scored by automatic checks alone: the public scorer reproduces all 1,166 verdicts in Excel, and 1,123 of the 1,160 in Google Sheets, where the export to XLSX accounts for the gap (explained under "Auditing edge cases openly"). The file lists every disagreement.
  • You can run it on your own product. The easiest way to start is to point your favorite coding agent at the repository and ask it to run one task.

The scorer and the runner are licensed under Apache-2.0, and the tasks and the scores under CC BY 4.0, except a few third-party inputs listed in the dataset card.

How we built the tasks

To decide what belonged in the suite, we analyzed tens of thousands of anonymized Autosheet conversations and broke multi-step user requests into individual spreadsheet tasks covering charts, formatting, and spreadsheet updates users ask for most often:

Chart types named in Autosheet user prompts, by prompts and distinct users
Chart typePrompts mentioning itDistinct users
Bar / column3,1382,163
Pie / donut2,1971,530
Line1,3881,010
Histogram649449
Scatter583428
Gantt452319
Combination / dual-axis311176
Formatting operations in Autosheet conversation turns, by turns and share of all turns
Formatting typeTurnsShare of all turns
Column widths / row heights / autofit169,56621.73%
Fonts, bold, size, text color165,84521.25%
Background colors / fills156,86720.10%
Number formats136,92717.55%
Alignment / text orientation87,04111.15%
Text wrapping68,4148.77%
Borders46,0285.90%
Conditional formatting31,9104.09%
Spreadsheet manipulations in Autosheet conversation turns, by turns and share of all turns
OperationTurnsShare of all turns
Copy/paste or duplicate108,64613.92%
Create worksheets40,9045.24%
Freeze/unfreeze panes30,7413.94%
Merge cells26,6633.42%
Data validation / dropdowns21,0392.70%
Create structured tables20,2272.59%
Reorder worksheets16,1472.07%
Delete rows10,7061.37%
Insert rows9,5621.23%
Delete columns9,2911.19%
Apply/change filters8,1701.05%
Sort ranges7,6480.98%
Insert columns6,9420.89%
Unmerge cells6,4140.82%

From that distribution, we manually crafted 159 native spreadsheet tasks across six capabilities (42 formulas, 36 spreadsheet manipulation, 30 charts, 26 formatting, 17 data operations, and 8 pivot tables) and 28 bulk tasks across two capabilities (17 bulk data processing and 11 bulk content generation, operating on up to 1,000 rows per sheet). 22 of the 187 tasks include scripted follow-up prompts sent after the first turn completes.

Prompts stay intentionally under-specified

One trap in synthetic benchmarks is writing overly detailed prompts that read like unit-test specs. Real spreadsheet users don't write like that. They leave spreadsheet mechanics implicit:

  • A real user writes "Highlight figures that are same" and expects conditional formatting, not static yellow fill that goes stale the moment a cell changes. Our benchmark task Highlight duplicate email addresses asks simply "Highlight duplicate email addresses."—without mentioning conditional formatting.
  • A real user writes "Show only HC entries not others" and expects a filter, not deleted rows. Our task Filter rows above a revenue asks "Show rows where revenue is above 10,000."—without saying the word "filter".
  • Real users routinely follow up "Create a pie chart" with "Add percentages to Pie Chart", showing they expect readable data labels even when they don't spell them out on turn one.

How the runner works

In Excel, it's a script that works on its own once launched, with no person and no AI agent in the loop. It drives the add-in (Autosheet, ChatGPT, Claude or Copilot) through macOS accessibility: it opens a fresh copy of the workbook, starts a screen recording, types each prompt into the add-in's chat, waits for the answer, then saves the workbook, the transcript and screenshots of the output. The runner and its guide are in the repository.

In Google Sheets, and for the assistants connected through an MCP, a computer-use agent, an AI agent that operates the screen, drove Chrome through the same steps, following a written guide.

The reason why we scripted Excel is we couldn't run Excel in the foreground, and thus needed to script it so that it went faster than a 100% computer use solution. As Google Sheets needn't run in the foreground, we didn't have that speed problem and could parallelize computer use.

In both cases the runner is able to detect and answer mid-task questions asked by the product, without giving hints, in order to keep the run going.

We measured time on the screen recordings in order to get true durations. A script goes through each recording frame by frame and counts only the time the product is working, from the moment the prompt is sent to the end of the answer, leaving out the waits between turns and on permission cards.

How we ran the tasks

Every number on AutosheetBench is pass@1: the first run that finished cleanly, never the best of N attempts.

  • Strict rerun policy. We only discarded a run and restarted from a fresh workbook when something outside the product failed: a network drop, credits or a usage cap running out (for instance, Gemini in Sheets exhausted its monthly AI credit allowance mid-benchmark and required waiting for monthly renewal plus a plan upgrade), a model at capacity, a runner or computer-use mistake missing a prompt, or a task input/prompt revision. Every rerun is logged in a rerun-review.json file in the dataset.
  • Hard time limits. Regular tasks have 5 minutes; the 26 long bulk tasks (up to 1,000 rows) have 10 minutes, shared across all turns. When time expires, we stop the product and score the workbook as it stands. Across all 4,114 runs, 54 runs hit the time limit—and 7 of them still passed.

Here is what one recorded run looks like—Claude in Excel (Opus 5.5 · High) completing Chart sales stacked by region in 9.2 seconds, with the sheet beside the chat pane so every step is auditable:

How we judged the tasks

Every task is strict binary pass or fail—no partial credit. A single missed check or missed rubric checkpoint fails the task. Across the 187 tasks, we combine three evaluation mechanisms depending on what the task actually asks for.

1. Deterministic workbook checks (96 native tasks)

Whenever a task has a verifiable workbook state, Python scripts inspect the actual .xlsx structure—cell values, data types, formulas, conditional formatting rules, filters, and frozen panes—not just what the sheet looks like visually. When there are multiple valid spreadsheet approaches, every valid way is accepted.

Take Highlight revenues above a threshold ("Highlight revenues over 150000 with a light yellow fill.", passed by 17 of 22 configurations). Visually, the two outputs below look identical. Deterministic inspection catches the difference immediately: Claude in Excel added a dynamic conditional formatting rule (=B2>150000 or cell-value rule), while Copilot in Excel painted static yellow fills onto B5:B7 by hand—so if a revenue number changes, the highlight breaks.

SucceededClaude in Excel (Opus 5.5 · High) — dynamic conditional formatting rule
Claude in Excel passing output on the revenue threshold task using a dynamic conditional formatting rule
FailedCopilot in Excel (Auto) — static yellow fill applied by hand
Copilot in Excel failing output on the revenue threshold task using static yellow cell fills instead of a conditional formatting rule

The same strictness catches subtle spreadsheet cheats elsewhere: in Fix numbers stored as text, a product that leaves the amounts stored as text strings and works around the broken total with =SUMPRODUCT(--B2:B6) fails, because the check inspects cell types, not just the displayed total.

2. LLM judge with a hand-edited checklist (63 native tasks)

For 63 native tasks—charts, dashboards, cover sheets, visual formatting, and 6 exact-answer chat questions—a rigid script cannot reliably capture every valid layout. Instead, an LLM judge grades the output workbook summary, externally captured post-run screenshots, and the transcript against a hand-edited checklist of pass/fail checkpoints.

We run Gemini 3.8 Flash through OpenRouter for three reasons: it is independent of both OpenAI and Anthropic (which power most of the products tested), we calibrated its verdicts against human expectations, and it keeps verification affordable—roughly 1¢ per checklist-judged output and 10¢ per quality-judged bulk output (about $39 in total to judge the benchmark once, roughly one-fifth of what Opus 5.5 would cost).

Take Create a Gantt chart ("Create a Gantt chart.", passed by 9 of 22 configurations). Both products below draw floating horizontal bars. ChatGPT in Excel fails because its horizontal axis counts raw days from the start (0, 2, 4, 6) instead of calendar dates—a detail the four-word prompt never spelled out, but that any person reading a project schedule expects.

SucceededAutosheet in Excel (Opus 5.5 · Medium) — horizontal axis displays calendar dates
Autosheet in Excel passing Gantt chart output with calendar dates on the horizontal axis
FailedChatGPT in Excel (GPT-6 Astra · Medium) — horizontal axis counts raw days (0, 2, 4, 6)
ChatGPT in Excel failing Gantt chart output with raw day counts on the horizontal axis instead of calendar dates

Visual judging also catches rendering bugs that pure workbook XML misses: in Add thousands separators (passed by just 3 of 22 configurations), products that apply the right number format without widening column B leave numbers hidden behind ######—and fail.

3. Whole-set bulk checks, quality sampling, and anti-boilerplate gates (28 bulk tasks)

Bulk tasks ask a product to apply an AI prompt across hundreds or up to 1,000 rows—for example, classifying support tickets, translating product catalogs, or summarizing job descriptions. Every bulk task runs deterministic whole-set checks (bringing the total to 124 automatically checked tasks across the benchmark: 96 native and 28 bulk), and 14 of the 28 also require an LLM quality judgment. Because Autosheet routes bulk tasks row-by-row to a dedicated bulk model (a lightweight model configured separately from the main agent model) while other products often write code or loop in context, naive spot checks would completely miss how products actually fail at scale.

Our bulk scorer layers three checks:

  1. Whole-set structural and accuracy checks: Every row must be filled and follow the prompt's constraints (length caps, allowed label sets, non-empty outputs). When each row has one right answer (14 tasks), at least 98% of rows must match (100% on 3 strict tasks).
  2. Row-level quality judge (14 open-ended text tasks): For tasks where wording varies (summaries, translations, replies, SEO descriptions), Gemini 3.8 Flash runs at high effort and temperature 0 (medium effort let real errors through) and grades 50 sampled rows independently against their source text. At least 45 of 50 must pass a strict bar: "Fail the row when an ordinary careful non-expert could readily make it more accurate, focused, complete, grammatical, or useful."
  3. Anti-template gate: On text-generation tasks, a deterministic script automatically fails the run if 10 or fewer sentence templates cover 80% or more of the rows (and on synthetic dataset generation, the judge fails outputs that recycle entries or splice a handful of reused words).

Summarize job descriptions (passed by 10 of 22 configurations) shows why structural checks alone are dangerously blind. Copilot in Excel filled all 1,000 rows with summaries under the 160-character limit—passing every structural rule—yet the quality judge rejected all 50 sampled rows. Looking inside the sheet revealed why: 845 of Copilot's 1,000 summaries started with "Lead teams and projects", pasting stock duties onto unrelated roles:

Row
Source job description excerpt
Copilot in Excel output (rejected)
Why it failed
461
"…extracting actionable insights to help empower businesses to better manage their fleets."
"Lead teams and projects, build machine-learning models, and analyze complex data to improve healthcare decisions."
Hallucinated boilerplate: the role serves vehicle fleets, not healthcare.
324
"This position is primarily responsible for coordinating and conducting day-to-day activities in the laboratory to ensure timely and accurate testing."
"Lead teams and projects, deploy production solutions, and analyze complex data to advance research and healthcare outcomes."
Hands-on lab coordinator turned into generic data/software boilerplate.

Auditing edge cases openly

Even a careful automated pipeline runs into platform edge cases. When Google Sheets exports a workbook to .xlsx, it silently drops sheet protection, strips checkbox validation, flattens histograms into images, loses logarithmic axis flags, and extends filter ranges over a total row. Rather than penalizing products for Google Sheets' export quirks—or letting static fills slip past an image judge—we inspected those edge cases in the native Sheets UI and documented every single manual override in results.jsonl (manual_score and manual_review_reason), including flipping false-negative automatic fails to passes for Claude in Sheets and ChatGPT in Sheets.

Inspecting those failures also turned up genuinely wild agent bugs—such as Claude in Sheets (Opus 5.5) timing out on Generate 1,000 sample CRM records because its generated Apps Script tried to sample 1,000 unique company names in a while loop from a pool of 40 prefixes × 24 suffixes = 960 possible combinations, deadlocking forever before writing a single cell.

Limitations

No benchmark is perfect, and we would rather state the caveats plainly—the same ones documented in the Hugging Face dataset card:

  • Point-in-time snapshot. Real AI spreadsheet products ship updates continuously. Published results reflect the products as they behaved when run (latest run November 17, 2026); rerunning today may not reproduce a published score.
  • Single run per task (pass@1). Every score comes from the first clean run rather than multi-trial averages, so small gaps of one or two tasks should be read cautiously.
  • 73 of 187 tasks are saturated. All 22 configurations passed 73 of the tasks; the remaining 114 drive the ranking differences.
  • Google Sheets is scored from .xlsx exports. The export loses some native features, so a few verdicts needed a manual check in Sheets itself. Each one is recorded in results.jsonl.
  • Automated runner covers Excel only. The macOS runner drives the four Excel add-ins automatically. Google Sheets and MCP runs used a computer-use agent following docs/computer-use.md (not shipped in the repository). Where recordings did not capture a clean turn end (138 Copilot in Excel runs and 64 runs of other assistants), durations fall back to runner-clock upper bounds.
  • Post-run anonymization and withheld artifacts. 27 task inputs were cleaned after runs used them to strip personal data or workbook metadata (historical_input_sha256 records the exact file tested; 11 of those tasks withhold output workbooks and transcripts). Of 4,114 runs, 3,604 screen recordings are published (masked and frame-screened with no audio) and 510 are withheld. Trimmed recordings still include inter-turn waits, so use duration_seconds rather than video length.
  • Non-deterministic LLM judges. Checklist and quality judgments use Gemini 3.8 Flash and are not bit-for-bit deterministic across repeated calls; changing the judge model makes scores non-comparable with ours.
  • We are one of the vendors. We build Autosheet. We built AutosheetBench first to improve our own product and opened every artifact so you don't have to take our word for any score.

If you spot an issue in a task, a rubric, or a verdict—or if you run AutosheetBench on your own spreadsheet agent—open an issue on GitHub or email us at nicolas.augusti@talarian.io or stan@talarian.io. And if you want to explore the numbers, the AutosheetBench page has the full interactive explorer, CSVs, and BibTeX citation.

Related Articles