Refund policy: Development API evaluation

Synthetic development cases. AI-drafted expected answers; no independent review.

Agreement with draft labels is not evidence of business trust or unseen accuracy. No refund is executed. Errors do not count as model escalations; production must handle them separately.

Latency and estimated API cost

USD for this run. Rules have zero API cost, not zero engineering or compute cost. Model latency includes network and SDK processing; rules run locally. These are different execution paths, not equivalent compute measurements.

Case decisions

Provider	Accuracy	Wrong eligibility	Wrong rejection	Required-review misses	Review rate	Automation coverage
jev	92.2% (83/90)	16.0% (4/25)	12.5% (3/24)	2.4% (1/42)	45.6% (41/90)	54.4% (49/90)
openai	87.8% (79/90)	15.8% (3/19)	25.0% (8/32)	7.1% (3/42)	43.3% (39/90)	56.7% (51/90)
rules	100.0% (90/90)	0.0% (0/24)	0.0% (0/24)	0.0% (0/42)	46.7% (42/90)	53.3% (48/90)

Provider	Median ms	Input USD	Output USD	Total USD	Known cost / attempted	API USD / correct definitive outcome
jev	350.17115000	0.00637207	0.00000000	0.00637207	90/90	0.00015172
openai	930.55910000	0.05062800	0.00102880	0.05165680	90/90	0.00129142
rules	0.05360000	0.00000000	0.00000000	0.00000000	90/90	0.00000000

Case	Provider	Status	Draft expected answer	Actual	Correct	Rules clause	Error
D21	rules	ok	human_review	human_review	True	G2	None
D21	jev	ok	human_review	human_review	True	None	None
D21	openai	ok	human_review	human_review	True	None	None
D25	rules	ok	human_review	human_review	True	G2	None
D25	openai	ok	human_review	human_review	True	None	None
D25	jev	ok	human_review	human_review	True	None	None
D09	rules	ok	eligible	eligible	True	E1	None
D09	jev	ok	eligible	not_eligible	False	None	None
D09	openai	ok	eligible	not_eligible	False	None	None
D01	rules	ok	eligible	eligible	True	E1	None
D01	openai	ok	eligible	eligible	True	None	None
D01	jev	ok	eligible	eligible	True	None	None
D19	rules	ok	human_review	human_review	True	G1	None
D19	jev	ok	human_review	human_review	True	None	None
D19	openai	ok	human_review	human_review	True	None	None
D13	rules	ok	not_eligible	not_eligible	True	N1	None
D13	openai	ok	not_eligible	not_eligible	True	None	None
D13	jev	ok	not_eligible	not_eligible	True	None	None
D30	rules	ok	human_review	human_review	True	G1	None
D30	jev	ok	human_review	human_review	True	None	None
D30	openai	ok	human_review	human_review	True	None	None
D14	rules	ok	eligible	eligible	True	E0	None
D14	openai	ok	eligible	eligible	True	None	None
D14	jev	ok	eligible	eligible	True	None	None
D11	rules	ok	not_eligible	not_eligible	True	N1	None
D11	jev	ok	not_eligible	not_eligible	True	None	None
D11	openai	ok	not_eligible	not_eligible	True	None	None
D10	rules	ok	not_eligible	not_eligible	True	N2	None
D10	openai	ok	not_eligible	not_eligible	True	None	None
D10	jev	ok	not_eligible	not_eligible	True	None	None
D06	rules	ok	not_eligible	not_eligible	True	N2	None
D06	jev	ok	not_eligible	eligible	False	None	None
D06	openai	ok	not_eligible	not_eligible	True	None	None
D12	rules	ok	eligible	eligible	True	E0	None
D12	openai	ok	eligible	eligible	True	None	None
D12	jev	ok	eligible	eligible	True	None	None
D18	rules	ok	human_review	human_review	True	G1	None
D18	jev	ok	human_review	human_review	True	None	None
D18	openai	ok	human_review	human_review	True	None	None
D07	rules	ok	eligible	eligible	True	E1	None
D07	openai	ok	eligible	eligible	True	None	None
D07	jev	ok	eligible	eligible	True	None	None
D04	rules	ok	not_eligible	not_eligible	True	N2	None
D04	jev	ok	not_eligible	not_eligible	True	None	None
D04	openai	ok	not_eligible	not_eligible	True	None	None
D22	rules	ok	human_review	human_review	True	G2	None
D22	openai	ok	human_review	human_review	True	None	None
D22	jev	ok	human_review	human_review	True	None	None
D02	rules	ok	eligible	eligible	True	E1	None
D02	jev	ok	eligible	eligible	True	None	None
D02	openai	ok	eligible	not_eligible	False	None	None
D20	rules	ok	human_review	human_review	True	G1	None
D20	openai	ok	human_review	human_review	True	None	None
D20	jev	ok	human_review	human_review	True	None	None
D28	rules	ok	not_eligible	not_eligible	True	N2	None
D28	jev	ok	not_eligible	not_eligible	True	None	None
D28	openai	ok	not_eligible	not_eligible	True	None	None
D03	rules	ok	eligible	eligible	True	E1	None
D03	openai	ok	eligible	eligible	True	None	None
D03	jev	ok	eligible	eligible	True	None	None
D24	rules	ok	human_review	human_review	True	G2	None
D24	jev	ok	human_review	eligible	False	None	None
D24	openai	ok	human_review	eligible	False	None	None
D27	rules	ok	not_eligible	not_eligible	True	N2	None
D27	openai	ok	not_eligible	not_eligible	True	None	None
D27	jev	ok	not_eligible	not_eligible	True	None	None
D29	rules	ok	human_review	human_review	True	G1	None
D29	jev	ok	human_review	human_review	True	None	None
D29	openai	ok	human_review	human_review	True	None	None
D26	rules	ok	human_review	human_review	True	G0	None
D26	openai	ok	human_review	human_review	True	None	None
D26	jev	ok	human_review	human_review	True	None	None
D17	rules	ok	human_review	human_review	True	G1	None
D17	jev	ok	human_review	human_review	True	None	None
D17	openai	ok	human_review	human_review	True	None	None
D23	rules	ok	human_review	human_review	True	G2	None
D23	openai	ok	human_review	human_review	True	None	None
D23	jev	ok	human_review	human_review	True	None	None
D08	rules	ok	not_eligible	not_eligible	True	N2	None
D08	jev	ok	not_eligible	not_eligible	True	None	None
D08	openai	ok	not_eligible	not_eligible	True	None	None
D05	rules	ok	eligible	eligible	True	E1	None
D05	openai	ok	eligible	not_eligible	False	None	None
D05	jev	ok	eligible	eligible	True	None	None
D16	rules	ok	human_review	human_review	True	G1	None
D16	jev	ok	human_review	human_review	True	None	None
D16	openai	ok	human_review	human_review	True	None	None
D15	rules	ok	human_review	human_review	True	G1	None
D15	openai	ok	human_review	human_review	True	None	None
D15	jev	ok	human_review	human_review	True	None	None
D15	rules	ok	human_review	human_review	True	G1	None
D15	jev	ok	human_review	human_review	True	None	None
D15	openai	ok	human_review	human_review	True	None	None
D03	rules	ok	eligible	eligible	True	E1	None
D03	openai	ok	eligible	eligible	True	None	None
D03	jev	ok	eligible	eligible	True	None	None
D04	rules	ok	not_eligible	not_eligible	True	N2	None
D04	jev	ok	not_eligible	not_eligible	True	None	None
D04	openai	ok	not_eligible	not_eligible	True	None	None
D17	rules	ok	human_review	human_review	True	G1	None
D17	openai	ok	human_review	human_review	True	None	None
D17	jev	ok	human_review	human_review	True	None	None
D21	rules	ok	human_review	human_review	True	G2	None
D21	jev	ok	human_review	human_review	True	None	None
D21	openai	ok	human_review	human_review	True	None	None
D01	rules	ok	eligible	eligible	True	E1	None
D01	openai	ok	eligible	eligible	True	None	None
D01	jev	ok	eligible	eligible	True	None	None
D11	rules	ok	not_eligible	not_eligible	True	N1	None
D11	jev	ok	not_eligible	not_eligible	True	None	None
D11	openai	ok	not_eligible	not_eligible	True	None	None
D28	rules	ok	not_eligible	not_eligible	True	N2	None
D28	openai	ok	not_eligible	not_eligible	True	None	None
D28	jev	ok	not_eligible	not_eligible	True	None	None
D29	rules	ok	human_review	human_review	True	G1	None
D29	jev	ok	human_review	human_review	True	None	None
D29	openai	ok	human_review	human_review	True	None	None
D06	rules	ok	not_eligible	not_eligible	True	N2	None
D06	openai	ok	not_eligible	not_eligible	True	None	None
D06	jev	ok	not_eligible	eligible	False	None	None
D10	rules	ok	not_eligible	not_eligible	True	N2	None
D10	jev	ok	not_eligible	not_eligible	True	None	None
D10	openai	ok	not_eligible	not_eligible	True	None	None
D23	rules	ok	human_review	human_review	True	G2	None
D23	openai	ok	human_review	human_review	True	None	None
D23	jev	ok	human_review	human_review	True	None	None
D26	rules	ok	human_review	human_review	True	G0	None
D26	jev	ok	human_review	human_review	True	None	None
D26	openai	ok	human_review	human_review	True	None	None
D12	rules	ok	eligible	eligible	True	E0	None
D12	openai	ok	eligible	eligible	True	None	None
D12	jev	ok	eligible	eligible	True	None	None
D25	rules	ok	human_review	human_review	True	G2	None
D25	jev	ok	human_review	human_review	True	None	None
D25	openai	ok	human_review	human_review	True	None	None
D08	rules	ok	not_eligible	not_eligible	True	N2	None
D08	openai	ok	not_eligible	not_eligible	True	None	None
D08	jev	ok	not_eligible	not_eligible	True	None	None
D19	rules	ok	human_review	human_review	True	G1	None
D19	jev	ok	human_review	human_review	True	None	None
D19	openai	ok	human_review	human_review	True	None	None
D18	rules	ok	human_review	human_review	True	G1	None
D18	openai	ok	human_review	human_review	True	None	None
D18	jev	ok	human_review	human_review	True	None	None
D09	rules	ok	eligible	eligible	True	E1	None
D09	jev	ok	eligible	not_eligible	False	None	None
D09	openai	ok	eligible	not_eligible	False	None	None
D24	rules	ok	human_review	human_review	True	G2	None
D24	openai	ok	human_review	eligible	False	None	None
D24	jev	ok	human_review	human_review	True	None	None
D20	rules	ok	human_review	human_review	True	G1	None
D20	jev	ok	human_review	human_review	True	None	None
D20	openai	ok	human_review	human_review	True	None	None
D16	rules	ok	human_review	human_review	True	G1	None
D16	openai	ok	human_review	human_review	True	None	None
D16	jev	ok	human_review	human_review	True	None	None
D13	rules	ok	not_eligible	not_eligible	True	N1	None
D13	jev	ok	not_eligible	not_eligible	True	None	None
D13	openai	ok	not_eligible	not_eligible	True	None	None
D30	rules	ok	human_review	human_review	True	G1	None
D30	openai	ok	human_review	human_review	True	None	None
D30	jev	ok	human_review	human_review	True	None	None
D07	rules	ok	eligible	eligible	True	E1	None
D07	jev	ok	eligible	eligible	True	None	None
D07	openai	ok	eligible	eligible	True	None	None
D27	rules	ok	not_eligible	not_eligible	True	N2	None
D27	openai	ok	not_eligible	not_eligible	True	None	None
D27	jev	ok	not_eligible	not_eligible	True	None	None
D05	rules	ok	eligible	eligible	True	E1	None
D05	jev	ok	eligible	eligible	True	None	None
D05	openai	ok	eligible	not_eligible	False	None	None
D14	rules	ok	eligible	eligible	True	E0	None
D14	openai	ok	eligible	eligible	True	None	None
D14	jev	ok	eligible	eligible	True	None	None
D02	rules	ok	eligible	eligible	True	E1	None
D02	jev	ok	eligible	eligible	True	None	None
D02	openai	ok	eligible	not_eligible	False	None	None
D22	rules	ok	human_review	human_review	True	G2	None
D22	openai	ok	human_review	human_review	True	None	None
D22	jev	ok	human_review	human_review	True	None	None
D17	rules	ok	human_review	human_review	True	G1	None
D17	jev	ok	human_review	human_review	True	None	None
D17	openai	ok	human_review	human_review	True	None	None
D02	rules	ok	eligible	eligible	True	E1	None
D02	openai	ok	eligible	not_eligible	False	None	None
D02	jev	ok	eligible	eligible	True	None	None
D24	rules	ok	human_review	human_review	True	G2	None
D24	jev	ok	human_review	human_review	True	None	None
D24	openai	ok	human_review	eligible	False	None	None
D13	rules	ok	not_eligible	not_eligible	True	N1	None
D13	openai	ok	not_eligible	not_eligible	True	None	None
D13	jev	ok	not_eligible	not_eligible	True	None	None
D30	rules	ok	human_review	human_review	True	G1	None
D30	jev	ok	human_review	human_review	True	None	None
D30	openai	ok	human_review	human_review	True	None	None
D07	rules	ok	eligible	eligible	True	E1	None
D07	openai	ok	eligible	eligible	True	None	None
D07	jev	ok	eligible	eligible	True	None	None
D06	rules	ok	not_eligible	not_eligible	True	N2	None
D06	jev	ok	not_eligible	eligible	False	None	None
D06	openai	ok	not_eligible	not_eligible	True	None	None
D23	rules	ok	human_review	human_review	True	G2	None
D23	openai	ok	human_review	human_review	True	None	None
D23	jev	ok	human_review	human_review	True	None	None
D29	rules	ok	human_review	human_review	True	G1	None
D29	jev	ok	human_review	human_review	True	None	None
D29	openai	ok	human_review	human_review	True	None	None
D15	rules	ok	human_review	human_review	True	G1	None
D15	openai	ok	human_review	human_review	True	None	None
D15	jev	ok	human_review	human_review	True	None	None
D18	rules	ok	human_review	human_review	True	G1	None
D18	jev	ok	human_review	human_review	True	None	None
D18	openai	ok	human_review	human_review	True	None	None
D03	rules	ok	eligible	eligible	True	E1	None
D03	openai	ok	eligible	eligible	True	None	None
D03	jev	ok	eligible	eligible	True	None	None
D25	rules	ok	human_review	human_review	True	G2	None
D25	jev	ok	human_review	human_review	True	None	None
D25	openai	ok	human_review	human_review	True	None	None
D26	rules	ok	human_review	human_review	True	G0	None
D26	openai	ok	human_review	human_review	True	None	None
D26	jev	ok	human_review	human_review	True	None	None
D04	rules	ok	not_eligible	not_eligible	True	N2	None
D04	jev	ok	not_eligible	not_eligible	True	None	None
D04	openai	ok	not_eligible	not_eligible	True	None	None
D14	rules	ok	eligible	eligible	True	E0	None
D14	openai	ok	eligible	eligible	True	None	None
D14	jev	ok	eligible	eligible	True	None	None
D19	rules	ok	human_review	human_review	True	G1	None
D19	jev	ok	human_review	human_review	True	None	None
D19	openai	ok	human_review	human_review	True	None	None
D08	rules	ok	not_eligible	not_eligible	True	N2	None
D08	openai	ok	not_eligible	not_eligible	True	None	None
D08	jev	ok	not_eligible	not_eligible	True	None	None
D27	rules	ok	not_eligible	not_eligible	True	N2	None
D27	jev	ok	not_eligible	not_eligible	True	None	None
D27	openai	ok	not_eligible	not_eligible	True	None	None
D05	rules	ok	eligible	eligible	True	E1	None
D05	openai	ok	eligible	eligible	True	None	None
D05	jev	ok	eligible	eligible	True	None	None
D01	rules	ok	eligible	eligible	True	E1	None
D01	jev	ok	eligible	eligible	True	None	None
D01	openai	ok	eligible	eligible	True	None	None
D09	rules	ok	eligible	eligible	True	E1	None
D09	openai	ok	eligible	not_eligible	False	None	None
D09	jev	ok	eligible	not_eligible	False	None	None
D20	rules	ok	human_review	human_review	True	G1	None
D20	jev	ok	human_review	human_review	True	None	None
D20	openai	ok	human_review	human_review	True	None	None
D10	rules	ok	not_eligible	not_eligible	True	N2	None
D10	openai	ok	not_eligible	not_eligible	True	None	None
D10	jev	ok	not_eligible	not_eligible	True	None	None
D16	rules	ok	human_review	human_review	True	G1	None
D16	jev	ok	human_review	human_review	True	None	None
D16	openai	ok	human_review	human_review	True	None	None
D11	rules	ok	not_eligible	not_eligible	True	N1	None
D11	openai	ok	not_eligible	not_eligible	True	None	None
D11	jev	ok	not_eligible	not_eligible	True	None	None
D21	rules	ok	human_review	human_review	True	G2	None
D21	jev	ok	human_review	human_review	True	None	None
D21	openai	ok	human_review	human_review	True	None	None
D28	rules	ok	not_eligible	not_eligible	True	N2	None
D28	openai	ok	not_eligible	not_eligible	True	None	None
D28	jev	ok	not_eligible	not_eligible	True	None	None
D22	rules	ok	human_review	human_review	True	G2	None
D22	jev	ok	human_review	human_review	True	None	None
D22	openai	ok	human_review	human_review	True	None	None
D12	rules	ok	eligible	eligible	True	E0	None
D12	openai	ok	eligible	eligible	True	None	None
D12	jev	ok	eligible	eligible	True	None	None

Repeat consistency

Cases with the same successful outcome on every repeat / cases attempted more than once. A request failure counts as inconsistent; an identical incorrect recommendation can still be consistent. Stable does not mean correct. Repeats are not independent new cases.

Rates use the denominators shown. Wrong eligibility is wrong eligible outputs / all eligible outputs; wrong rejection is wrong not_eligible outputs / all not_eligible outputs. Required-review misses count definitive outputs where review was required. Latency uses successful responses only; failures reduce accuracy and remain in the raw records. Confidence is captured for Jev only, not calibrated or used to automate. Prices are frozen configuration estimates, not invoices.

Provider	Consistency
jev	96.7% (29/30)
openai	96.7% (29/30)
rules	100.0% (30/30)