Refund policy: API evaluation with human-reviewed expected answers

Synthetic cases. Expected answers reviewed by one human reviewer. H13 revised after policy clarification.

Reviewer saw prior development results; cases overlap earlier families. This is not an independently authored holdout or production certification. No refund is executed. Errors do not count as model escalations; production must handle them separately.

Latency and estimated API cost

USD for this run. Rules have zero API cost, not zero engineering or compute cost. Model latency includes network and SDK processing; rules run locally. These are different execution paths, not equivalent compute measurements.

Case decisions

Provider	Accuracy	Wrong eligibility	Wrong rejection	Required-review misses	Review rate	Automation coverage
jev	93.3% (42/45)	20.0% (3/15)	0.0% (0/15)	16.7% (3/18)	33.3% (15/45)	66.7% (30/45)
openai	73.3% (33/45)	0.0% (0/3)	44.4% (12/27)	16.7% (3/18)	33.3% (15/45)	66.7% (30/45)
rules	100.0% (45/45)	0.0% (0/12)	0.0% (0/15)	0.0% (0/18)	40.0% (18/45)	60.0% (27/45)

Provider	Median ms	Input USD	Output USD	Total USD	Known cost / attempted	API USD / correct definitive outcome
jev	318.46730004	0.00318074	0.00000000	0.00318074	45/45	0.00011781
openai	980.91129994	0.02584080	0.00054240	0.02638320	45/45	0.00146573
rules	0.07590000	0.00000000	0.00000000	0.00000000	45/45	0.00000000

Case	Provider	Status	Reviewed expected answer	Actual	Correct	Rules clause	Error
H04	rules	ok	human_review	human_review	True	G2	None
H04	jev	ok	human_review	eligible	False	None	None
H04	openai	ok	human_review	not_eligible	False	None	None
H10	rules	ok	not_eligible	not_eligible	True	N2	None
H10	openai	ok	not_eligible	not_eligible	True	None	None
H10	jev	ok	not_eligible	not_eligible	True	None	None
H01	rules	ok	eligible	eligible	True	E1	None
H01	jev	ok	eligible	eligible	True	None	None
H01	openai	ok	eligible	not_eligible	False	None	None
H11	rules	ok	human_review	human_review	True	G1	None
H11	openai	ok	human_review	human_review	True	None	None
H11	jev	ok	human_review	human_review	True	None	None
H06	rules	ok	human_review	human_review	True	G1	None
H06	jev	ok	human_review	human_review	True	None	None
H06	openai	ok	human_review	human_review	True	None	None
H07	rules	ok	eligible	eligible	True	E0	None
H07	openai	ok	eligible	eligible	True	None	None
H07	jev	ok	eligible	eligible	True	None	None
H02	rules	ok	not_eligible	not_eligible	True	N2	None
H02	jev	ok	not_eligible	not_eligible	True	None	None
H02	openai	ok	not_eligible	not_eligible	True	None	None
H05	rules	ok	eligible	eligible	True	E1	None
H05	openai	ok	eligible	not_eligible	False	None	None
H05	jev	ok	eligible	eligible	True	None	None
H14	rules	ok	eligible	eligible	True	E1	None
H14	jev	ok	eligible	eligible	True	None	None
H14	openai	ok	eligible	not_eligible	False	None	None
H13	rules	ok	human_review	human_review	True	G0	None
H13	openai	ok	human_review	human_review	True	None	None
H13	jev	ok	human_review	human_review	True	None	None
H03	rules	ok	not_eligible	not_eligible	True	N2	None
H03	jev	ok	not_eligible	not_eligible	True	None	None
H03	openai	ok	not_eligible	not_eligible	True	None	None
H08	rules	ok	not_eligible	not_eligible	True	N1	None
H08	openai	ok	not_eligible	not_eligible	True	None	None
H08	jev	ok	not_eligible	not_eligible	True	None	None
H12	rules	ok	human_review	human_review	True	G2	None
H12	jev	ok	human_review	human_review	True	None	None
H12	openai	ok	human_review	human_review	True	None	None
H15	rules	ok	not_eligible	not_eligible	True	N2	None
H15	openai	ok	not_eligible	not_eligible	True	None	None
H15	jev	ok	not_eligible	not_eligible	True	None	None
H09	rules	ok	human_review	human_review	True	G1	None
H09	jev	ok	human_review	human_review	True	None	None
H09	openai	ok	human_review	human_review	True	None	None
H15	rules	ok	not_eligible	not_eligible	True	N2	None
H15	openai	ok	not_eligible	not_eligible	True	None	None
H15	jev	ok	not_eligible	not_eligible	True	None	None
H03	rules	ok	not_eligible	not_eligible	True	N2	None
H03	jev	ok	not_eligible	not_eligible	True	None	None
H03	openai	ok	not_eligible	not_eligible	True	None	None
H01	rules	ok	eligible	eligible	True	E1	None
H01	openai	ok	eligible	not_eligible	False	None	None
H01	jev	ok	eligible	eligible	True	None	None
H13	rules	ok	human_review	human_review	True	G0	None
H13	jev	ok	human_review	human_review	True	None	None
H13	openai	ok	human_review	human_review	True	None	None
H05	rules	ok	eligible	eligible	True	E1	None
H05	openai	ok	eligible	not_eligible	False	None	None
H05	jev	ok	eligible	eligible	True	None	None
H09	rules	ok	human_review	human_review	True	G1	None
H09	jev	ok	human_review	human_review	True	None	None
H09	openai	ok	human_review	human_review	True	None	None
H04	rules	ok	human_review	human_review	True	G2	None
H04	openai	ok	human_review	not_eligible	False	None	None
H04	jev	ok	human_review	eligible	False	None	None
H07	rules	ok	eligible	eligible	True	E0	None
H07	jev	ok	eligible	eligible	True	None	None
H07	openai	ok	eligible	eligible	True	None	None
H11	rules	ok	human_review	human_review	True	G1	None
H11	openai	ok	human_review	human_review	True	None	None
H11	jev	ok	human_review	human_review	True	None	None
H08	rules	ok	not_eligible	not_eligible	True	N1	None
H08	jev	ok	not_eligible	not_eligible	True	None	None
H08	openai	ok	not_eligible	not_eligible	True	None	None
H14	rules	ok	eligible	eligible	True	E1	None
H14	openai	ok	eligible	not_eligible	False	None	None
H14	jev	ok	eligible	eligible	True	None	None
H12	rules	ok	human_review	human_review	True	G2	None
H12	jev	ok	human_review	human_review	True	None	None
H12	openai	ok	human_review	human_review	True	None	None
H02	rules	ok	not_eligible	not_eligible	True	N2	None
H02	openai	ok	not_eligible	not_eligible	True	None	None
H02	jev	ok	not_eligible	not_eligible	True	None	None
H10	rules	ok	not_eligible	not_eligible	True	N2	None
H10	jev	ok	not_eligible	not_eligible	True	None	None
H10	openai	ok	not_eligible	not_eligible	True	None	None
H06	rules	ok	human_review	human_review	True	G1	None
H06	openai	ok	human_review	human_review	True	None	None
H06	jev	ok	human_review	human_review	True	None	None
H08	rules	ok	not_eligible	not_eligible	True	N1	None
H08	jev	ok	not_eligible	not_eligible	True	None	None
H08	openai	ok	not_eligible	not_eligible	True	None	None
H10	rules	ok	not_eligible	not_eligible	True	N2	None
H10	openai	ok	not_eligible	not_eligible	True	None	None
H10	jev	ok	not_eligible	not_eligible	True	None	None
H05	rules	ok	eligible	eligible	True	E1	None
H05	jev	ok	eligible	eligible	True	None	None
H05	openai	ok	eligible	not_eligible	False	None	None
H04	rules	ok	human_review	human_review	True	G2	None
H04	openai	ok	human_review	not_eligible	False	None	None
H04	jev	ok	human_review	eligible	False	None	None
H02	rules	ok	not_eligible	not_eligible	True	N2	None
H02	jev	ok	not_eligible	not_eligible	True	None	None
H02	openai	ok	not_eligible	not_eligible	True	None	None
H07	rules	ok	eligible	eligible	True	E0	None
H07	openai	ok	eligible	eligible	True	None	None
H07	jev	ok	eligible	eligible	True	None	None
H01	rules	ok	eligible	eligible	True	E1	None
H01	jev	ok	eligible	eligible	True	None	None
H01	openai	ok	eligible	not_eligible	False	None	None
H09	rules	ok	human_review	human_review	True	G1	None
H09	openai	ok	human_review	human_review	True	None	None
H09	jev	ok	human_review	human_review	True	None	None
H13	rules	ok	human_review	human_review	True	G0	None
H13	jev	ok	human_review	human_review	True	None	None
H13	openai	ok	human_review	human_review	True	None	None
H12	rules	ok	human_review	human_review	True	G2	None
H12	openai	ok	human_review	human_review	True	None	None
H12	jev	ok	human_review	human_review	True	None	None
H14	rules	ok	eligible	eligible	True	E1	None
H14	jev	ok	eligible	eligible	True	None	None
H14	openai	ok	eligible	not_eligible	False	None	None
H11	rules	ok	human_review	human_review	True	G1	None
H11	openai	ok	human_review	human_review	True	None	None
H11	jev	ok	human_review	human_review	True	None	None
H03	rules	ok	not_eligible	not_eligible	True	N2	None
H03	jev	ok	not_eligible	not_eligible	True	None	None
H03	openai	ok	not_eligible	not_eligible	True	None	None
H06	rules	ok	human_review	human_review	True	G1	None
H06	openai	ok	human_review	human_review	True	None	None
H06	jev	ok	human_review	human_review	True	None	None
H15	rules	ok	not_eligible	not_eligible	True	N2	None
H15	jev	ok	not_eligible	not_eligible	True	None	None
H15	openai	ok	not_eligible	not_eligible	True	None	None

Repeat consistency

Cases with the same successful outcome on every repeat / cases attempted more than once. A request failure counts as inconsistent; an identical incorrect recommendation can still be consistent. Stable does not mean correct. Repeats are not independent new cases.

Rates use the denominators shown. Wrong eligibility is wrong eligible outputs / all eligible outputs; wrong rejection is wrong not_eligible outputs / all not_eligible outputs. Required-review misses count definitive outputs where review was required. Latency uses successful responses only; failures reduce accuracy and remain in the raw records. Confidence is captured for Jev only, not calibrated or used to automate. Prices are frozen configuration estimates, not invoices.

Provider	Consistency
jev	100.0% (15/15)
openai	100.0% (15/15)
rules	100.0% (15/15)

H13 adjudication record

Original case: Please assess refund eligibility. Product type: hardware. The original structured case is included in the supporting-data download.

Original answer: not_eligible.

The product is hardware rather than a subscription. The policy terms (Plan A / Plan B cancellation windows and usage limits) strictly govern subscriptions; hardware is not covered. (Note: If your operating procedures route out-of-policy/unsupported product categories to manual escalation, this could alternatively be handled under `human_review`, but strictly under the policy rules it is not eligible).

Ambiguity identified: The initial reviewer-facing policy summary omitted the G0 scope rule. Unsupported hardware was interpreted as ineligible rather than requiring review. The assistant flagged this conflict after the original submission.

Final interpretation: If product_type is not `subscription`, return `human_review`. H13 therefore requires human_review.

Approval: The original human reviewer, recorded as user-reviewer-1, explicitly accepted the revision after G0 clarification, according to the saved adjudication record. This was assisted clarification, not a second independent review.

Model-output separation: The saved record confirms that model outputs were not consulted for the adjudication. No new adjudication or experiment was performed for this editorial update.