All AutosheetBench tasks

Pivot revenue by region with % of total

Pivot tablesMediumLLM judge scoringLatest run

Passed by 11 of 12 products on their best model. Failed: ChatGPT in Sheets.

Prompt and starting spreadsheet

Task files

Prompt

In a new Pivot sheet at A3, create a pivot table showing revenue by region and include percent of total revenue.

Data

Starting sheet
ABC
1DateRegionRevenue
22025-01-03North8200
32025-01-05South6400
42025-01-07East9100
52025-01-09West7300
62025-01-12North8700
72025-01-14South6900
82025-01-18East9400
92025-01-21West7600

Pivot

Starting sheet
A
1

Expected result

How each AI configuration did

19 of 22 passRun files
Excel and Sheets · All types · All models · All outputs

Pass criteria

All must be met, checked by an LLM judge.
  • A real pivot table on Pivot!A3
  • Revenue summed by region
  • A second value: % of the grand total

Passed 19

Autosheet in Excel Opus 5.5 · Medium

Passed17s · 14¢

Autosheet in Excel GPT-6 Astra · Low

Passed16s · 12¢

Autosheet in Sheets Opus 5.5 · Medium

Passed17s · 4¢

Autosheet in Excel Sonnet 5.5 · Medium

Passed11s · 3¢

Claude in Excel Opus 5.5 · High

Passed26s · 20¢

Autosheet in Excel Haiku 5.5 · High

Passed18s · 0.3¢

Claude in Sheets Opus 5.5 · Medium

Passed29s · 6¢

Claude Code + Sheets MCP Opus 5.5 · Medium

Passed35s · N/A

Autosheet in Excel GPT-6.1 Sol · Medium

Passed22s · 8¢

Claude Code + Autosheet MCP Opus 5.5 · Medium

Passed40s · N/A

ChatGPT + Autosheet MCP GPT-6 · Medium

Passed39s · N/A

Autosheet in Sheets Haiku 5.5 · High

Passed34s · 0.3¢

ChatGPT in Excel GPT-6 Astra · Medium

Passed34s · 60¢

Claude in Excel Haiku 5.5 · High

Passed10s · 0.4¢

Claude in Sheets Haiku 5.5 · High

Passed1m 7s · 3¢

Autosheet in Excel GPT-6 Luna · High

Passed35s · 0.7¢

Copilot in Excel Auto

Passed37s · N/A

ChatGPT + Drive MCP GPT-6 · Medium

Passed48s · N/A

Gemini in Sheets Auto

Passed1m 46s · N/A

Failed 3

ChatGPT in Excel GPT-6 Luna · High

Failed1m 11s · 0.8¢

Why it failed: The percent of total is not configured as a second displayed value in the PivotTable; it is a manual side formula calculation (=B4/$B$8) in column C.

ChatGPT in Sheets GPT-6 Astra · Medium

Failed39s · 53¢

Why it failed: A real pivot table was not created; a manual summary table using SUMIF formulas was created instead.

ChatGPT in Sheets GPT-6 Luna · High

Failed1m 9s · 0.7¢

Why it failed: A real pivot table was not created; the agent built a manual summary table using SUMIF formulas.