All AutosheetBench tasks

Group transactions by month and quarter

Pivot tablesEasyLLM judge scoringLatest run

Passed by 11 of 12 products on their best model. Failed: ChatGPT in Sheets.

Prompt and starting spreadsheet

Task files

Prompt

In a new Pivot sheet at A3, group the transaction data into months and quarters in a pivot table.

Data

Starting sheet
ABCD
1Transaction DateQuarterMonthRevenue
22024-01-03Q1Jan3200
32024-02-08Q1Feb4100
42024-03-11Q1Mar3800
52024-04-05Q2Apr4600
62024-05-09Q2May5100
72024-06-12Q2Jun4900
82024-07-07Q3Jul5400
92024-08-15Q3Aug5600
102024-09-18Q3Sep6000
112024-10-10Q4Oct6200
122024-11-14Q4Nov6400
132024-12-19Q4Dec6800
142025-01-06Q1Jan3500
152025-02-13Q1Feb4200
162025-03-20Q1Mar4500

Pivot

Starting sheet
A
1

Expected result

How each AI configuration did

20 of 22 passRun files
Excel and Sheets · All types · All models · All outputs

Pass criteria

All must be met, checked by an LLM judge.
  • A real pivot table on Pivot!A3
  • Revenue summed by quarter, then month
  • The groups and sums match the data

Passed 20

Autosheet in Excel Opus 5.5 · Medium

Passed13s · 5¢

Autosheet in Excel GPT-6 Astra · Low

Passed12s · 10¢

Autosheet in Sheets Opus 5.5 · Medium

Passed38s · 10¢

Autosheet in Excel Sonnet 5.5 · Medium

Passed8s · 2¢

Claude in Excel Opus 5.5 · High

Passed39s · 27¢

Autosheet in Excel Haiku 5.5 · High

Passed38s · 0.6¢

Claude in Sheets Opus 5.5 · Medium

Passed59s · 22¢

Claude Code + Sheets MCP Opus 5.5 · Medium

Passed1m 13s · N/A

Autosheet in Excel GPT-6.1 Sol · Medium

Passed23s · 4¢

Claude Code + Autosheet MCP Opus 5.5 · Medium

Passed54s · N/A

ChatGPT + Autosheet MCP GPT-6 · Medium

Passed1m 8s · N/A

Autosheet in Sheets Haiku 5.5 · High

Passed50s · 0.6¢

ChatGPT in Excel GPT-6 Astra · Medium

Passed45s · 64¢

Claude in Excel Haiku 5.5 · High

Passed28s · 0.7¢

Claude in Sheets Haiku 5.5 · High

Passed56s · 2¢

Autosheet in Excel GPT-6 Luna · High

Passed27s · 0.5¢

Copilot in Excel Auto

Passed46s · N/A

ChatGPT + Drive MCP GPT-6 · Medium

Passed33s · N/A

ChatGPT in Excel GPT-6 Luna · High

Passed1m 11s · 0.9¢

Gemini in Sheets Auto

Passed1m 34s · N/A

Failed 2

ChatGPT in Sheets GPT-6 Astra · Medium

Failed1m 1s · 60¢

Why it failed: No real pivot table was created.

ChatGPT in Sheets GPT-6 Luna · High

Failed1m 50s · 0.9¢

Why it failed: No real pivot table was created; a formula-driven summary table was created instead.