All AutosheetBench tasks

Add a matching table of failed tool calls

FormattingEasyLLM judge scoringLatest run

Passed by all 12 products on their best model.

Prompt and starting spreadsheet

Task files

Prompt

Add a similar table to "# turns with tool call" ones below it, but only with the ones were the tool failed at least once (use dataset col J)

Dataset

Starting sheet · rows 1–21 of 95,215 and columns A–J of 14
ABCDEFGHIJ
1agent_run_idagent_idproductfirst_pack_typestart_event_timestampDatenb_loopsdistinct_tools_usedok_toolsfailed_tools
23c4d6ccb-5366-4267-84b2-342b574b2e6d663bb40f-ea62-4899-8862-e6c72c880f39excel2026-06-08 14:00:58.574 UTCJun 8, 2026
3f61a43a5-ffd8-4267-8e15-e783b8169f81e6725675-7a0c-4e09-80e3-1bd707ea8070excelFREE_TRIAL2026-06-08 14:00:48.485 UTCJun 8, 20261write_rangewrite_range
4ef030721-83f3-4080-87e9-535c3692182245500c7e-a84e-4d3b-8bcc-9ed1e73e22a3excelPAID_PACK2026-06-08 14:00:45.646 UTCJun 8, 20262get_sheet_overviewget_sheet_overview
511a7dd6a-f619-4132-877b-791a6f5b62511cff89f2-0e27-400e-80c9-156ef9e4b5e1excelFREE_TRIAL2026-06-08 14:00:43.926 UTCJun 8, 20263get_sheet_overview, read_rangeget_sheet_overview, read_range
6dce3f6d4-aa4b-4ee7-8fa3-24cf06eb7332618d474d-df15-4804-8fd0-e14816bb124aexcelFREE_TRIAL2026-06-08 14:00:40.372 UTCJun 8, 20261
753026ebf-abb7-4b57-8b71-1fc37d974a2474c40571-26d6-4b5b-8bb3-00b7b7c9688fexcelFREE_TRIAL2026-06-08 14:00:37.32 UTCJun 8, 20262write_rangewrite_range
827e8df97-3d67-4cc4-882b-1c33b8c6c644ad185af1-ae1e-4b69-8d68-c433ee609ee5excelFREE_TRIAL2026-06-08 14:00:37.046 UTCJun 8, 20262get_sheet_overviewget_sheet_overview
918c9f6d6-96dd-4c20-817a-0230af92c50533b5af13-5ec4-4f41-8011-e34463d85d47excelFREE_TRIAL2026-06-08 14:00:36.807 UTCJun 8, 20262write_rangewrite_range
103421c317-dcc2-4212-887d-e1c822f7ff1403f46d2a-cf5c-49c7-8bcf-88f4a0c5e424excelFREE_TRIAL2026-06-08 14:00:35.732 UTCJun 8, 20262execute_officejs_scriptexecute_officejs_script
11c3f939f7-6431-4c5e-8104-f1d005c2af2f46013fa2-66a1-4ee6-85a2-01e1033a7641excelFREE_TRIAL2026-06-08 14:00:34.792 UTCJun 8, 20261
122862f8a6-1813-4db7-8776-451cf4001a3f5b7fd5b9-05ab-4800-8163-92544482b53dexcelFREE_TRIAL2026-06-08 14:00:34.011 UTCJun 8, 20261
13452ac514-73d1-48e4-8076-2b98a42fca088b1e03b1-893c-4bbf-8f5a-96a46c7d488cexcelFREE_TRIAL2026-06-08 14:00:31.659 UTCJun 8, 20263execute_officejs_script, write_rangeexecute_officejs_script, write_range
14139e1032-f4e5-4ebf-802f-8697d61a9ec147819f2d-f5b7-4ed2-8e73-4165e10524bdexcelFREE_TRIAL2026-06-08 14:00:25.992 UTCJun 8, 20262copy_paste_rangecopy_paste_range
159b0b3c72-59e2-4243-8195-f116cb878db908c0a702-e756-4540-8075-4db55a3101f8excelFREE_TRIAL2026-06-08 14:00:23.718 UTCJun 8, 20262write_rangewrite_range
16155a398f-f11c-4bcc-8992-b9b627501f68414044b1-aee9-4101-8b34-c244d2e4dbcfexcelFREE_TRIAL2026-06-08 14:00:21.593 UTCJun 8, 20263copy_paste_range, write_rangecopy_paste_range, write_range
176876a140-9e3e-452f-8d37-b59335d06d322aee2b37-cf6c-4a9e-8e57-754521cd0af7excelFREE_TRIAL2026-06-08 14:00:21.159 UTCJun 8, 20263execute_officejs_script, write_rangeexecute_officejs_script, write_range
181c5c2f4f-72e2-4bdb-825a-cb618c4d750bcbfc142a-a975-4bfe-8a46-218c0b9aa848sheetsPAID_PACK2026-06-08 14:00:20.569 UTCJun 8, 20262ai_bulk_process_rowsai_bulk_process_rows
19a9459dc5-f8c0-455e-8305-8fd946f8a71b128e14a1-207a-4471-81ba-3f1b581c0195excelFREE_TRIAL2026-06-08 14:00:19.823 UTCJun 8, 20263execute_officejs_script, get_sheet_overviewexecute_officejs_script, get_sheet_overview
20f1f09115-309c-4933-86e0-ddd7c04a0ecfa9a8ea07-82f3-4489-8199-08c3509d97abexcelPAID_PACK2026-06-08 14:00:14.355 UTCJun 8, 20262get_sheet_overview, read_rangeget_sheet_overview, read_range
21a71b6f3f-7a31-493c-832a-523f1031ce7a56296678-05c0-44c3-8e31-4be77f891066excelFREE_TRIAL2026-06-08 14:00:08.165 UTCJun 8, 20264execute_officejs_script, write_rangeexecute_officejs_script, write_range

Analysis

Starting sheet
BCDEFGHIJ
1
2PackFREE_TRIAL
3
4
5# turns with tool call
6DateJun 2, 2026Jun 3, 2026Jun 4, 2026Jun 5, 2026Jun 6, 2026Jun 7, 2026Jun 8, 2026
7read_range2,8673,0072,5941,9341,2041,1421,452
8write_range4,5884,7884,0573,4872,8742,8063,182
9copy_paste_range1,9822,1091,8201,4441,1211,0541,270
10ai_generate_list0000000
11ai_bulk_process_rows501646514394242304373
12get_sheet_overview1,4191,3981,2681,075700666823
13spreadsheet_batch_update159152167126797679
14execute_officejs_script6,4646,7155,8684,8723,7873,5974,292
15read_spreadsheet_fields147161146124726468
16load_skill886922853697578538568

Expected result

How each AI configuration did

19 of 22 passRun files
Excel and Sheets · All types · All models · All outputs

Pass criteria

All must be met, checked by an LLM judge.
  • A new table of failed tool calls by tool and date
  • Formatted like the original table
  • The original data is kept

Passed 19

Autosheet in Excel Opus 5.5 · Medium

Passed24s · 12¢

Autosheet in Excel GPT-6 Astra · Low

Passed20s · 17¢

Autosheet in Sheets Opus 5.5 · Medium

Passed44s · 14¢
Analysis89 cells changed

D20=D6

ABCDEFGHIJ
1
11ai_bulk_process_rows501646514394242304373
12get_sheet_overview1,4191,3981,2681,075700666823
13spreadsheet_batch_update159152167126797679
14execute_officejs_script6,4646,7155,8684,8723,7873,5974,292
15read_spreadsheet_fields147161146124726468
16load_skill886922853697578538568
17
18
19# turns with failed tool call
20DateJun 2, 2026Jun 3, 2026Jun 4, 2026Jun 5, 2026Jun 6, 2026Jun 7, 2026Jun 8, 2026
21read_range458148133
22write_range951077979545660
23copy_paste_range29212722242931
24ai_generate_list0000000
25ai_bulk_process_rows232275221173102128157
26get_sheet_overview3223125
27spreadsheet_batch_update5122130
28execute_officejs_script9810712383666975
29read_spreadsheet_fields1001000
30load_skill0000000

Claude in Excel Opus 5.5 · High

Passed22s · 20¢

Autosheet in Excel Haiku 5.5 · High

Passed40s · 0.7¢

Claude in Sheets Opus 5.5 · Medium

Passed50s · 14¢

Claude Code + Sheets MCP Opus 5.5 · Medium

Passed1m 28s · N/A

Claude Code + Autosheet MCP Opus 5.5 · Medium

Passed2m 16s · N/A

ChatGPT + Autosheet MCP GPT-6 · Medium

Passed1m 37s · N/A

Autosheet in Sheets Haiku 5.5 · High

Passed2m 34s · 2¢

ChatGPT in Excel GPT-6 Astra · Medium

Passed30s · 62¢

Claude in Sheets Haiku 5.5 · High

Passed47s · 1¢

Autosheet in Excel GPT-6 Luna · High

Passed59s · 1¢

Copilot in Excel Auto

Passed56s · N/A

ChatGPT + Drive MCP GPT-6 · Medium

Passed51s · N/A

ChatGPT in Excel GPT-6 Luna · High

Passed1m 45s · 2¢

Gemini in Sheets Auto

Passed2m 58s · N/A

ChatGPT in Sheets GPT-6 Astra · Medium

Passed55s · 60¢

ChatGPT in Sheets GPT-6 Luna · High

Passed1m 54s · 1¢

Failed 3

Autosheet in Excel Sonnet 5.5 · Medium

Failed21s · 6¢

Why it failed: No new table was created below the existing table on the Analysis sheet.

Autosheet in Excel GPT-6.1 Sol · Medium

Failed39s · 12¢

Why it failed: The formatting does not match the original table: the title header lacks the yellow background (it is styled with a pale blue fill), and the header background is pale blue rather than the…

Claude in Excel Haiku 5.5 · High

Failed39s · 2¢

Why it failed: The table title in C18 is '# turns with tool call (tools that failed at least once)' rather than the required '# turns with failed tool call'.