SimplifyAItools: routing & sequential planning

Measured sequential-v2 run

Protocol dataset: 20 routing cases and six preselected multi-step planning cases, evaluated independently of routing. No real tools are executed. A plan is proposed one action at a time, with up to three actions followed by stop.

This report contains 20 unique cases and 156 case attempts across the selected tracks/providers/orders/repeats.

Models: jev: jev-1.13.0, openai: gpt-4.1-mini-2025-04-14

ProviderTrackCompleted / attemptedStrict routeOperational routeFirst actionExact sequenceMedian ms / caseAPI calls
jevplan8/18N/AN/A100.0% (n=18)33.3% (n=18)1195.33468
jevroute60/6085.0% (n=60)90.0% (n=60)N/AN/A341.25960
openaiplan9/18N/AN/A100.0% (n=18)16.7% (n=18)4533.70271
openairoute60/6086.7% (n=60)86.7% (n=60)N/AN/A900.71060

Cost breakdown (USD, original order)

A planning case includes ALL its API calls, including the stop decision. Jev output is free at the saved rate. Missing usage remains unknown. Projected cost is per 1,000 cases/plans, not per 1,000 individual API calls.

ProviderTrackInput costOutput costTotal costProjected / 1,000 casesCost coverage
jevplan0.0021880320.0000000000.0021880320.12155718/18
jevroute0.0016334640.0000000000.0016334640.02722460/60
openaiplan0.0140436000.0007584000.0148020000.82233318/18
openairoute0.0090696000.0006240000.0096936000.16156060/60

Actual predictions

CaseProviderTrackOrderRepeatPrediction / partial planStatusFirst correctSequence correctTotal msCalls
R11jevplanoriginal1['google_drive', 'gmail']okTrueTrue1049.8923
R11openaiplanoriginal1['google_drive', 'gmail', 'general_reasoning', 'general_reasoning']max_stepsTrueFalse4264.7084
R11jevplanoriginal2['google_drive', 'gmail', 'google_drive', 'gmail']max_stepsTrueFalse1352.2634
R11openaiplanoriginal2['google_drive', 'gmail', 'general_reasoning']okTrueFalse4602.4324
R11jevplanoriginal3['google_drive', 'gmail']okTrueTrue1046.9993
R11openaiplanoriginal3['google_drive', 'gmail', 'general_reasoning']okTrueFalse4223.5594
R13jevplanoriginal1['crm', 'general_reasoning', 'general_reasoning', 'general_reasoning']max_stepsTrueFalse1328.6944
R13openaiplanoriginal1['crm', 'general_reasoning', 'general_reasoning']okTrueFalse4666.0304
R13jevplanoriginal2['crm', 'general_reasoning', 'general_reasoning']okTrueFalse1330.8474
R13openaiplanoriginal2['crm', 'general_reasoning']okTrueTrue3386.0413
R13jevplanoriginal3['crm', 'general_reasoning', 'general_reasoning', 'general_reasoning']max_stepsTrueFalse1377.3514
R13openaiplanoriginal3['crm', 'general_reasoning', 'general_reasoning', 'general_reasoning']max_stepsTrueFalse4040.8644
R17jevplanoriginal1['gmail', 'calendar']okTrueTrue1059.8223
R17openaiplanoriginal1['gmail', 'calendar', 'general_reasoning', 'gmail']max_stepsTrueFalse5327.0484
R17jevplanoriginal2['gmail', 'calendar']okTrueTrue1006.1613
R17openaiplanoriginal2['gmail', 'calendar', 'general_reasoning', 'gmail']max_stepsTrueFalse4184.7564
R17jevplanoriginal3['gmail', 'calendar', 'general_reasoning']okTrueFalse1626.3184
R17openaiplanoriginal3['gmail', 'calendar', 'general_reasoning', 'general_reasoning']max_stepsTrueFalse5006.9194
R18jevplanoriginal1['google_drive', 'general_reasoning', 'general_reasoning', 'general_reasoning']max_stepsTrueFalse1649.6504
R18openaiplanoriginal1['google_drive', 'general_reasoning', 'general_reasoning']okTrueFalse6333.6484
R18jevplanoriginal2['google_drive', 'general_reasoning', 'general_reasoning', 'general_reasoning']max_stepsTrueFalse1395.6324
R18openaiplanoriginal2['google_drive', 'general_reasoning', 'general_reasoning']okTrueFalse5211.9314
R18jevplanoriginal3['google_drive', 'general_reasoning', 'general_reasoning', 'general_reasoning']max_stepsTrueFalse1359.0704
R18openaiplanoriginal3['google_drive', 'general_reasoning', 'general_reasoning']okTrueFalse4476.0574
R19jevplanoriginal1['gmail', 'crm', 'general_reasoning', 'general_reasoning']max_stepsTrueFalse1383.9944
R19openaiplanoriginal1['gmail', 'crm', 'general_reasoning']okTrueTrue3880.2814
R19jevplanoriginal2['gmail', 'crm', 'general_reasoning']okTrueTrue1476.1384
R19openaiplanoriginal2['gmail', 'crm', 'general_reasoning']okTrueTrue4533.7024
R19jevplanoriginal3['gmail', 'crm', 'general_reasoning']okTrueTrue1379.1074
R19openaiplanoriginal3['gmail', 'crm', 'general_reasoning', 'general_reasoning']max_stepsTrueFalse4532.3974
R20jevplanoriginal1['database', 'general_reasoning', 'general_reasoning', 'general_reasoning']max_stepsTrueFalse1485.1944
R20openaiplanoriginal1['database', 'crm', 'general_reasoning', 'general_reasoning']max_stepsTrueFalse4398.9104
R20jevplanoriginal2['database', 'general_reasoning', 'general_reasoning', 'general_reasoning']max_stepsTrueFalse1311.9634
R20openaiplanoriginal2['database', 'crm', 'general_reasoning', 'general_reasoning']max_stepsTrueFalse4525.2484
R20jevplanoriginal3['database', 'general_reasoning', 'general_reasoning', 'general_reasoning']max_stepsTrueFalse1298.2814
R20openaiplanoriginal3['database', 'crm', 'general_reasoning', 'general_reasoning']max_stepsTrueFalse4188.5664
R01jevrouteoriginal1google_driveokNoneNone335.6871
R01openairouteoriginal1databaseokNoneNone790.5361
R01jevrouteoriginal2google_driveokNoneNone327.3071
R01openairouteoriginal2databaseokNoneNone1033.8871
R01jevrouteoriginal3google_driveokNoneNone374.8591
R01openairouteoriginal3databaseokNoneNone763.5341
R02jevrouteoriginal1gmailokNoneNone324.3271
R02openairouteoriginal1gmailokNoneNone873.4601
R02jevrouteoriginal2gmailokNoneNone341.7671
R02openairouteoriginal2gmailokNoneNone1219.0361
R02jevrouteoriginal3gmailokNoneNone326.5421
R02openairouteoriginal3gmailokNoneNone989.0001
R03jevrouteoriginal1calendarokNoneNone354.7021
R03openairouteoriginal1calendarokNoneNone902.8131
R03jevrouteoriginal2calendarokNoneNone311.8331
R03openairouteoriginal2calendarokNoneNone773.9611
R03jevrouteoriginal3calendarokNoneNone402.0371
R03openairouteoriginal3calendarokNoneNone1128.2871
R04jevrouteoriginal1crmokNoneNone377.5711
R04openairouteoriginal1crmokNoneNone808.7811
R04jevrouteoriginal2crmokNoneNone317.2341
R04openairouteoriginal2crmokNoneNone832.1111
R04jevrouteoriginal3crmokNoneNone334.5931
R04openairouteoriginal3crmokNoneNone924.4171
R05jevrouteoriginal1databaseokNoneNone316.4451
R05openairouteoriginal1databaseokNoneNone875.2721
R05jevrouteoriginal2databaseokNoneNone354.4851
R05openairouteoriginal2databaseokNoneNone737.8801
R05jevrouteoriginal3databaseokNoneNone340.7501
R05openairouteoriginal3databaseokNoneNone827.8631
R06jevrouteoriginal1general_reasoningokNoneNone392.6461
R06openairouteoriginal1general_reasoningokNoneNone766.2321
R06jevrouteoriginal2general_reasoningokNoneNone338.1541
R06openairouteoriginal2general_reasoningokNoneNone818.6991
R06jevrouteoriginal3general_reasoningokNoneNone351.3721
R06openairouteoriginal3gmailokNoneNone819.6721
R07jevrouteoriginal1gmailokNoneNone348.9521
R07openairouteoriginal1gmailokNoneNone986.9771
R07jevrouteoriginal2gmailokNoneNone366.4681
R07openairouteoriginal2gmailokNoneNone1036.1941
R07jevrouteoriginal3gmailokNoneNone313.6651
R07openairouteoriginal3gmailokNoneNone907.4011
R08jevrouteoriginal1gmailokNoneNone347.3201
R08openairouteoriginal1google_driveokNoneNone958.7901
R08jevrouteoriginal2gmailokNoneNone332.6881
R08openairouteoriginal2google_driveokNoneNone854.8131
R08jevrouteoriginal3gmailokNoneNone372.2831
R08openairouteoriginal3google_driveokNoneNone930.3431
R09jevrouteoriginal1crmokNoneNone391.9381
R09openairouteoriginal1crmokNoneNone1270.4001
R09jevrouteoriginal2crmokNoneNone356.4491
R09openairouteoriginal2crmokNoneNone837.5101
R09jevrouteoriginal3crmokNoneNone405.5681
R09openairouteoriginal3crmokNoneNone1591.9591
R10jevrouteoriginal1databaseokNoneNone369.3571
R10openairouteoriginal1databaseokNoneNone962.2321
R10jevrouteoriginal2databaseokNoneNone309.8941
R10openairouteoriginal2databaseokNoneNone1589.2731
R10jevrouteoriginal3databaseokNoneNone339.7891
R10openairouteoriginal3databaseokNoneNone906.0111
R11jevrouteoriginal1multi_toolokNoneNone381.2071
R11openairouteoriginal1multi_toolokNoneNone734.5331
R11jevrouteoriginal2multi_toolokNoneNone321.2711
R11openairouteoriginal2multi_toolokNoneNone869.9471
R11jevrouteoriginal3multi_toolokNoneNone355.5881
R11openairouteoriginal3multi_toolokNoneNone928.5751
R12jevrouteoriginal1calendarokNoneNone306.2471
R12openairouteoriginal1calendarokNoneNone2159.9981
R12jevrouteoriginal2calendarokNoneNone360.4571
R12openairouteoriginal2calendarokNoneNone767.1451
R12jevrouteoriginal3calendarokNoneNone337.3161
R12openairouteoriginal3calendarokNoneNone930.1071
R13jevrouteoriginal1crmokNoneNone343.0281
R13openairouteoriginal1multi_toolokNoneNone2295.7001
R13jevrouteoriginal2crmokNoneNone324.9811
R13openairouteoriginal2crmokNoneNone738.1931
R13jevrouteoriginal3crmokNoneNone336.7771
R13openairouteoriginal3multi_toolokNoneNone993.7941
R14jevrouteoriginal1ambiguousokNoneNone334.4811
R14openairouteoriginal1ambiguousokNoneNone1116.3361
R14jevrouteoriginal2ambiguousokNoneNone337.6541
R14openairouteoriginal2ambiguousokNoneNone965.1621
R14jevrouteoriginal3ambiguousokNoneNone481.3071
R14openairouteoriginal3ambiguousokNoneNone2972.5831
R15jevrouteoriginal1gmailokNoneNone322.8991
R15openairouteoriginal1google_driveokNoneNone994.4471
R15jevrouteoriginal2multi_toolokNoneNone334.3041
R15openairouteoriginal2google_driveokNoneNone810.5461
R15jevrouteoriginal3gmailokNoneNone330.4001
R15openairouteoriginal3google_driveokNoneNone879.3621
R16jevrouteoriginal1databaseokNoneNone320.7861
R16openairouteoriginal1databaseokNoneNone865.8891
R16jevrouteoriginal2databaseokNoneNone330.0501
R16openairouteoriginal2databaseokNoneNone1776.3881
R16jevrouteoriginal3databaseokNoneNone333.2051
R16openairouteoriginal3databaseokNoneNone1121.9511
R17jevrouteoriginal1multi_toolokNoneNone584.4091
R17openairouteoriginal1multi_toolokNoneNone815.3461
R17jevrouteoriginal2multi_toolokNoneNone341.8621
R17openairouteoriginal2multi_toolokNoneNone879.8301
R17jevrouteoriginal3multi_toolokNoneNone354.4431
R17openairouteoriginal3multi_toolokNoneNone811.7181
R18jevrouteoriginal1multi_toolokNoneNone330.6811
R18openairouteoriginal1multi_toolokNoneNone898.6081
R18jevrouteoriginal2multi_toolokNoneNone352.8541
R18openairouteoriginal2multi_toolokNoneNone834.2141
R18jevrouteoriginal3multi_toolokNoneNone341.9441
R18openairouteoriginal3multi_toolokNoneNone854.3011
R19jevrouteoriginal1multi_toolokNoneNone479.4241
R19openairouteoriginal1multi_toolokNoneNone963.0481
R19jevrouteoriginal2multi_toolokNoneNone324.9981
R19openairouteoriginal2multi_toolokNoneNone855.4151
R19jevrouteoriginal3multi_toolokNoneNone343.8261
R19openairouteoriginal3multi_toolokNoneNone1905.8511
R20jevrouteoriginal1multi_toolokNoneNone362.9891
R20openairouteoriginal1multi_toolokNoneNone1040.5891
R20jevrouteoriginal2multi_toolokNoneNone330.2791
R20openairouteoriginal2multi_toolokNoneNone874.5621
R20jevrouteoriginal3multi_toolokNoneNone354.7151
R20openairouteoriginal3multi_toolokNoneNone833.6001

Primary metrics use original order. Errors and step-limit failures count as incorrect, but a validated first action is scored independently. Median latency sums SDK-call times for each completed case (network included). Repeats are correlated; only six independent planning cases. See summary.json for consistency/order-stability coverage and results.jsonl for full prompts and raw responses. This revised harness is a new experiment, not directly comparable with the previous parallel-slot latency. No calibration or universal superiority claim is supported.

Back to article