All AutosheetBench tasks

Pivot revenue by region, then filter it

Pivot tablesMediumLLM judge scoringLatest run

Passed by 11 of 12 products on their best model. Failed: ChatGPT in Sheets.

Prompt and starting spreadsheet

Task files

Prompt

Break down our revenue by region in a pivot at I2

Now filter it to just North and East — those are our focus markets

Starting spreadsheet

Sheet1 · rows 1–21 of 42
ABCDEF
1OrderIDSalesRepRegionProductQuantityRevenue
2ORD-001Alice ChenNorthWidget Pro121440
3ORD-002Bob SmithSouthGadget Plus5625
4ORD-003Carol WhiteEastWidget Pro8960
5ORD-004David LeeWestDataHub31050
6ORD-005Alice ChenNorthGadget Plus101250
7ORD-006Eve JohnsonSouthSyncBox7910
8ORD-007Frank BrownEastDataHub41400
9ORD-008Grace KimWestWidget Pro6720
10ORD-009Bob SmithNorthSyncBox91170
11ORD-010Carol WhiteSouthPanelKit111540
12ORD-011David LeeEastWidget Pro141680
13ORD-012Eve JohnsonWestGadget Plus3375
14ORD-013Frank BrownNorthPanelKit81120
15ORD-014Grace KimSouthDataHub62100
16ORD-015Alice ChenEastSyncBox5650
17ORD-016Bob SmithWestWidget Pro101200
18ORD-017Carol WhiteNorthGadget Plus7875
19ORD-018David LeeSouthPanelKit4560
20ORD-019Eve JohnsonEastDataHub93150
21ORD-020Frank BrownWestSyncBox2260

Expected result

How each AI configuration did

19 of 22 passRun files
Excel and Sheets · All types · All models · All outputs

Pass criteria

All must be met, checked by an LLM judge.
  • A real pivot table of revenue by region
  • Then filtered to North and East
  • The source data is unchanged

Passed 19

Autosheet in Excel Opus 5.5 · Medium

Passed19s · 7¢

Autosheet in Excel GPT-6 Astra · Low

Passed15s · 14¢

Autosheet in Sheets Opus 5.5 · Medium

Passed20s · 6¢

Autosheet in Excel Sonnet 5.5 · Medium

Passed10s · 3¢

Claude in Excel Opus 5.5 · High

Passed16s · 18¢

Autosheet in Excel Haiku 5.5 · High

Passed16s · 0.3¢

Claude in Sheets Opus 5.5 · Medium

Passed37s · 9¢

Claude Code + Sheets MCP Opus 5.5 · Medium

Passed44s · N/A

Autosheet in Excel GPT-6.1 Sol · Medium

Passed34s · 7¢

Claude Code + Autosheet MCP Opus 5.5 · Medium

Passed1m 5s · N/A

ChatGPT + Autosheet MCP GPT-6 · Medium

Passed51s · N/A

Autosheet in Sheets Haiku 5.5 · High

Passed34s · 0.3¢

ChatGPT in Excel GPT-6 Astra · Medium

Passed32s · 80¢

Claude in Excel Haiku 5.5 · High

Passed23s · 2¢

Claude in Sheets Haiku 5.5 · High

Passed42s · 2¢

Autosheet in Excel GPT-6 Luna · High

Passed33s · 0.8¢

Copilot in Excel Auto

Passed57s · N/A
Sheet18 cells changed
ABCDEFGHIJ
1OrderIDSalesRepRegionProductQuantityRevenue
2ORD-001Alice ChenNorthWidget Pro121440RegionTotal Revenue
3ORD-002Bob SmithSouthGadget Plus5625East11,600
4ORD-003Carol WhiteEastWidget Pro8960North12,700
5ORD-004David LeeWestDataHub31050Grand Total24,300
6ORD-005Alice ChenNorthGadget Plus101250
7ORD-006Eve JohnsonSouthSyncBox7910
8ORD-007Frank BrownEastDataHub41400
9ORD-008Grace KimWestWidget Pro6720
10ORD-009Bob SmithNorthSyncBox91170
11ORD-010Carol WhiteSouthPanelKit111540
12ORD-011David LeeEastWidget Pro141680
13ORD-012Eve JohnsonWestGadget Plus3375
14ORD-013Frank BrownNorthPanelKit81120
15ORD-014Grace KimSouthDataHub62100
16ORD-015Alice ChenEastSyncBox5650
17ORD-016Bob SmithWestWidget Pro101200
18ORD-017Carol WhiteNorthGadget Plus7875
19ORD-018David LeeSouthPanelKit4560
20ORD-019Eve JohnsonEastDataHub93150
21ORD-020Frank BrownWestSyncBox2260

ChatGPT + Drive MCP GPT-6 · Medium

Passed55s · N/A

Gemini in Sheets Auto

Passed2m 1s · N/A

Failed 3

ChatGPT in Excel GPT-6 Luna · High

Failed4m 57s · 7¢

Why it failed: Still working at the 5-minute time limit. Workbook metadata confirms no native pivot table exists (pivotTableCount is 0); cells I2:J5 were populated with manual text and SUMIF formulas.

ChatGPT in Sheets GPT-6 Astra · Medium

Failed59s · 96¢

Why it failed: No pivot table: the summary is built with SUMIF formulas instead.

ChatGPT in Sheets GPT-6 Luna · High

Failed1m 2s · 1¢

Why it failed: No pivot table: the summary is built with SUMIF formulas instead.