All AutosheetBench tasks

Pivot hotel rates and nights by room

Pivot tablesEasyLLM judge scoringLatest run

Passed by 11 of 12 products on their best model. Failed: ChatGPT in Sheets.

Prompt and starting spreadsheet

Task files

Prompt

I need to see average nightly rate and total nights by room type in a pivot table. Anchor it in J2

Starting spreadsheet

Sheet1 · rows 1–21 of 32
ABCDEFG
1BookingIDGuestNameRoomTypeCheckInCheckOutNightlyRateNights
2BKG-001Sarah ConnorStandard2024-01-052024-01-071202
3BKG-002John CarterDeluxe2024-01-062024-01-101854
4BKG-003Emily ClarkeSuite2024-01-082024-01-113203
5BKG-004Michael FoxPenthouse2024-01-102024-01-145504
6BKG-005Rachel AdamsStandard2024-01-122024-01-131201
7BKG-006Tom HughesDeluxe2024-01-142024-01-171953
8BKG-007Linda ParkSuite2024-01-152024-01-203405
9BKG-008Kevin MooreStandard2024-01-182024-01-211153
10BKG-009Amy ScottPenthouse2024-01-202024-01-235803
11BKG-010Brian WalshDeluxe2024-01-222024-01-251853
12BKG-011Carla DiazStandard2024-01-242024-01-261202
13BKG-012David KimSuite2024-01-252024-01-293104
14BKG-013Nancy LeeDeluxe2024-01-272024-01-301903
15BKG-014Oscar RuizStandard2024-01-282024-02-011254
16BKG-015Patricia ChenPenthouse2024-01-302024-02-035604
17BKG-016Quentin BellSuite2024-02-022024-02-053253
18BKG-017Rebecca YoungStandard2024-02-042024-02-061152
19BKG-018Steven HillDeluxe2024-02-062024-02-091803
20BKG-019Tanya BrooksSuite2024-02-082024-02-133305
21BKG-020Ulrich BauerStandard2024-02-102024-02-141204

Expected result

How each AI configuration did

20 of 22 passRun files
Excel and Sheets · All types · All models · All outputs

Pass criteria

All must be met, checked by an LLM judge.
  • A real pivot table of the bookings
  • Rows by room type
  • Average nightly rate and total nights
  • The source data is unchanged

Passed 20

Autosheet in Excel Opus 5.5 · Medium

Passed15s · 5¢

Autosheet in Excel GPT-6 Astra · Low

Passed10s · 8¢

Autosheet in Sheets Opus 5.5 · Medium

Passed17s · 4¢

Autosheet in Excel Sonnet 5.5 · Medium

Passed11s · 3¢

Claude in Excel Opus 5.5 · High

Passed37s · 29¢

Autosheet in Excel Haiku 5.5 · High

Passed17s · 0.3¢

Claude in Sheets Opus 5.5 · Medium

Passed19s · 4¢

Claude Code + Sheets MCP Opus 5.5 · Medium

Passed39s · N/A

Autosheet in Excel GPT-6.1 Sol · Medium

Passed22s · 4¢

Claude Code + Autosheet MCP Opus 5.5 · Medium

Passed32s · N/A

ChatGPT + Autosheet MCP GPT-6 · Medium

Passed31s · N/A

Autosheet in Sheets Haiku 5.5 · High

Passed19s · 0.1¢

ChatGPT in Excel GPT-6 Astra · Medium

Passed17s · 45¢

Claude in Excel Haiku 5.5 · High

Passed28s · 2¢

Claude in Sheets Haiku 5.5 · High

Passed42s · 2¢

Autosheet in Excel GPT-6 Luna · High

Passed22s · 0.4¢

Copilot in Excel Auto

Passed39s · N/A

ChatGPT + Drive MCP GPT-6 · Medium

Passed29s · N/A

ChatGPT in Excel GPT-6 Luna · High

Passed1m 37s · 1¢

Gemini in Sheets Auto

Passed1m 4s · N/A

Failed 2

ChatGPT in Sheets GPT-6 Astra · Medium

Failed41s · 50¢

Why it failed: No native pivot table object was created in the workbook; the agent created a formula-based summary table.

ChatGPT in Sheets GPT-6 Luna · High

Failed55s · 0.6¢

Why it failed: No pivot table: the summary is built with AVERAGEIF and SUMIF formulas instead.