An AI system can recognize that a customer is asking for a refund and still make the wrong refund decision. The second task requires the company’s policy, reliable account records, exception rules and a clear instruction to stop when the evidence is insufficient.

That distinction became the central question in our Jev AI research at SimplifyAItools. We began with routing messages, moved to selecting workflow steps, and then tested a fictional refund policy. Jev had lower median latency than GPT-4.1 mini in our routing and refund-policy runs. Both models also made mistakes that prevented correct completion or violated the test policy.

Research status: These are findings from our initial routing, planning and synthetic refund-policy tests, collected September 27–30, 2026. This is an ongoing investigation, not a production reliability assessment. No refunds or other business actions were executed.

• What looks promising: fast, bounded judgments within a defined workflow.
• What our tests exposed: failures in stopping, exact policy boundaries and mandatory review conditions.
• What remains unproven: performance on independently reviewed unseen cases, calibrated automation thresholds and real business outcomes.

1 What Jev does and what it needs from a business

TypeSafe introduced Jev as a System One Model for structured decisions. The interface returns bounded values rather than general written responses. The company describes parallel output sampling and specialized training; these are vendor descriptions, separate from our own performance measurements. TypeSafe launch and vendor evaluation methodology [1]

A large language model, or LLM, can write a reply, explain an issue or produce structured output. A decision model focuses on a constrained judgment. In TypeSafe’s interface, the application supplies state—the information available for the decision—and typed questions. TypeSafe typed questions [2]

Figure 1. Two output paths, with application validation required for both. Generative LLMs can also return schema-constrained JSON; a correctly shaped answer can still be wrong.

Jev does not automatically know your company’s current policy. A business application must supply the relevant criteria and evidence. Recognizing “refund requested” does not establish eligibility, verify a transaction or authorize a payment.

For readers of our AI email agent tutorial, the distinction is practical: category selection is one job; writing a response is another; applying a refund policy is a third. A decision layer could be evaluated for the first job, without handing it authority over the others.

2 What trustworthy behavior would look like

For a bounded business task, trust should mean observable behavior under an agreed policy, not a general impression of intelligence. Before comparing speed, define what a correct outcome requires.

• Correct evidence: use records for the right account and transaction.
• Correct policy: apply the intended version, including exceptions and rule priority.
• Correct fallback: request review when information is missing, conflicting or out of scope.
• Correct execution: enforce permissions, verify tool results and stop when the task is complete.

When the relevant facts are already structured and the policy is explicit, ordinary code is an essential baseline. The harder language problem may be extracting those facts from documents. Our policy study tested supplied facts; it did not test extraction or real-world verification.

Figure 2. Proposed separation of judgment, validation and action. Missing or invalid evidence leads to review; a model recommendation must still pass policy and permission checks. This execution architecture was not tested in our pilots.

A model can recommend a next step, while the application checks whether it is permitted and necessary. Our Make.com assistant article provides a practical companion on the limits of a bounded workflow. Another common shortcut is to treat the model’s confidence as permission to act. That needs a separate test.

3 Confidence is a signal to validate

TypeSafe’s Choice and Score answers include distributions and a confidence value derived from them. Noul has no separate confidence field. A distribution concentrated on one answer indicates certainty within the model’s output; it does not by itself establish the observed probability of correctness on a company’s cases. TypeSafe confidence documentation [3]

Figure 3. Proposed confidence-based review flow. The acceptance rule must be evaluated on representative data; no numeric threshold is recommended by this diagram.

Calibration concerns whether predicted probabilities align with observed outcomes. To evaluate an automation rule, also measure coverage: how many cases qualify, how often accepted decisions are wrong, and how much review costs. Sending almost everything to a person can improve accepted accuracy while providing little automation.

Our tests recorded Jev confidence but did not validate a deployment threshold. The studies below therefore examine correctness and required review first, then use confidence as a diagnostic signal.

4 Our research so far

We tested Jev 1.13.0 and GPT-4.1 mini dated 2025-04-14. The policy study also included deterministic rules. Product descriptions above come from official documentation; the findings below come from our own recorded runs. The explanatory architecture is a proposal, not a measurement of proprietary internals.

The routing smoke test

On September 27, both models correctly routed all 20 authored support messages across three repeats: 60/60 each. Median latency was 326 ms for Jev and 815 ms for GPT-4.1 mini. This was a smoke test: it checked the integrations and easy classifications. It did not establish difficult decision-making ability. SimplifyAItools routing smoke test September 27 [4]

Routing and planning exposed different problems

The September 28 routing study used 20 cases with three repeats. Jev matched the strict label in 51/60 attempts; GPT-4.1 mini in 52/60. Allowing a documented alternative changed Jev’s score to 54/60. All three additional credits came from repeats of one case. The ranking depended on the rubric. SimplifyAItools routing and planning study September 28 [5]

• In six preselected planning cases, each repeated three times, both models chose the correct first action in 18/18 attempts.
• Jev produced the exact sequence and stopped in 6/18 attempts; GPT-4.1 mini did so in 3/18. Neither reliably completed this setup.
• Each model continued after a correct full sequence prefix in 12 attempts. Only action labels were selected; tools were not executed.

The finding is a gap between selecting useful actions and finishing a workflow. Failures could reflect the model, prompt, limited state or controller; these results do not isolate the cause. On a separate routing case, Jev disagreed with the rubric at confidence around 0.92–0.94. Neither observation establishes reliable autonomous operation. SimplifyAItools routing and planning study September 28 [5]

This left a narrower question to test next: could the models apply an explicit business policy correctly when the relevant facts were already supplied?

5 Testing an explicit refund policy

The September 30 study asked a more business-relevant question: given a policy and structured records, should the recommendation be eligible, not_eligible or human_review? The policy and every account record were fictional. A “verified” field meant verified within the scenario, not independently authenticated customer data. Fictional refund policy tested in both API evaluations [8]

• The policy defined plan-specific cancellation windows, usage limits and a request deadline.
• Scope, evidence and validity checks came before eligibility decisions.
• A duplicate-charge exception could override certain thresholds, but not the earlier gates or an already-refunded transaction.
• Boundaries were inclusive, and elapsed time had to be calculated exactly.

Two API evaluations using synthetic cases

We evaluated Jev and GPT-4.1 mini through their APIs using synthetic test cases and recorded their responses, latency and estimated API costs. Expected answers are the reference outcomes used to score those responses; their preparation and review are described below.

There were 270 model API calls across these two evaluations. All cases were synthetic; API evaluation does not mean production use. Three repeats are not three independent cases. Refund policy development API evaluation September 30 [6] Refund policy API evaluation with human-reviewed expected answers September 30 [7]

Development results

Jev agreed with the draft labels in 83/90 attempts, GPT-4.1 mini in 79/90, and rules in 90/90. However, the development runner required rules to agree with the draft labels before proceeding. Its perfect rules score therefore is not independent validation. Refund policy development API evaluation September 30 [6]

Two failures illustrate why exact arithmetic matters: Jev accepted a Plan B cancellation one second beyond its deadline in all three repeats. Both models rejected a request exactly at the allowed 30-day boundary in all three repeats.

The reviewed results

Methodology disclosure: The cases are synthetic and were created during this investigation. One human reviewer checked the final 15-case set. This was not an independently authored or blind benchmark.

The next set comprised 15 assistant-authored synthetic cases labeled by one human reviewer. The reviewer had seen earlier development results, and the cases overlapped prior families. H13 was revised after a disclosed clarification of the scope rule, without consulting model outputs. This was not an independently authored, blind holdout. Each case was run three times per approach, giving 45 recommendations. Refund policy API evaluation with human-reviewed expected answers September 30 [7]

A definitive output means eligible or not_eligible, not an executed action. Jev had higher agreement with the reviewed reference answers and lower median latency than this GPT configuration on these cases. Nevertheless, three of its 15 eligible recommendations should have gone to review. GPT’s zero wrong eligible outputs came from only three eligible recommendations; it also rejected valid cases. One headline accuracy or safety percentage misses that tradeoff.

The case both models mishandled

In H04, cancellation was dated August 10 and the refund request August 9. Clause G2 explicitly required review because the request preceded cancellation. Jev said eligible in all three repeats; GPT said not_eligible in all three. Rules returned human_review every time.

Figure 4. Observed H04 outcomes across three repeats per approach. Both models bypassed the required review, despite giving opposite recommendations. No refund was executed.

All three approaches had 100% repeat consistency on the reviewed set. H04 shows why that is not the same as correctness: the models consistently repeated an error. Jev’s confidence on those errors ranged from 0.45 to 0.69. That is a useful diagnostic observation, not a validated rule for accepting other decisions. Refund policy API evaluation with human-reviewed expected answers September 30 [7]

What speed and cost do not settle

Estimated API cost for all 45 reviewed attempts was $0.00318 for Jev and $0.02638 for GPT-4.1 mini, using configured token rates. Those are API estimates, not invoices or complete operating costs. Rules incurred no API charge but still require development, maintenance and compute. Their local latency is also a different execution path from a hosted model call.

The dollar difference is small at this test scale. Review effort, mistaken eligibility decisions and customer rework may matter much more. We have not measured those costs or actual financial consequences. Refund policy API evaluation with human-reviewed expected answers September 30 [7]

6 What we can conclude today

In these small synthetic samples, Jev showed higher agreement with our reviewed reference answers than the tested GPT-4.1 mini configuration. Both models nevertheless missed an explicit mandatory-review condition. The rules implementation handled it correctly and matched every reviewed reference answer.

Where the evidence is already structured and the policy can be expressed exactly, the decision may belong in code rather than in any AI model. Language interpretation should be separated from exact arithmetic, mandatory review gates, policy enforcement and authorization.

• Supported: lower observed Jev latency and API cost in our runs, plus concrete examples of policy and workflow failures.
• Not established: general model superiority, production accuracy, a safe confidence threshold or reliable autonomous execution.
• Practical implication: separate language interpretation from exact arithmetic, mandatory review gates and action authorization.

Jev may be useful as a fast, inexpensive component for bounded language judgments. But it should not receive business authority. When facts and policy are structured, deterministic controls should enforce exact rules, mandatory review and permissions. Model confidence, consistent output and typed responses did not ensure correct policy enforcement in our tests.

7 What our continuing research needs to answer

This article records the evidence available so far as an initial research report. The following work is the next research phase, not a condition for publishing the current findings. It is planned, not completed, and does not strengthen today’s results until it is carried out.

• Independent evaluation: freeze policy, prompts and scoring, then evaluate 200–500 unseen cases with independent labeling and explicit adjudication.
• Appropriate baselines: add a traditional or embedding classifier where language classification is the task; retain rules for exact policy calculations.
• Validated review rules: choose confidence thresholds on development data, then measure errors and coverage on new cases.
• Policy-change pairs: test changed rules and unchanged controls to detect both adaptation and regressions.
• Workflow evidence: use controlled tool results to test completion and permissions, then evaluate shadow recommendations on representative real cases if available.
• Outcome cost: include retries, human review and the consequences of wrong decisions rather than only token charges.

These stages may change our conclusions. Future revisions should identify new datasets, model and policy versions, changed findings and remaining limitations. The September 27–30 results should remain traceable rather than being silently replaced.

Methods and references

Research disclosure: No complimentary credits were received for this investigation.

Two additional setup-check reports validated rules and prepared model requests without calling Jev or GPT. The development check used two cases that were also included in the 30-case evaluation. The reviewed-case check used the same 15 cases as the reviewed API evaluation. These checks add no model-performance observations. In the saved evidence filenames, offline denotes a setup check and live denotes an API evaluation; those historical filenames are retained for traceability.

Both models received the semantic task and supplied evidence. Jev used Choice responses; GPT-4.1 mini used strict JSON, temperature zero and a 64-token output limit. Policy runs alternated model order, used sequential calls and no retries. All 270 policy API calls completed successfully. No tool execution, real verification, policy-change experiment or production traffic was tested.

Policy-run summaries were recalculated from raw records. Reviewer provenance is taken from the manifests and adjudication record. Repeated cases are correlated; percentages describe this sample and are not production risk estimates. Confidence was diagnostic only. The references below link directly to the evaluation reports. Each policy report includes an optional supporting-data download containing the raw records and unchanged original report.

1. TypeSafe launch and vendor evaluation methodology
2. TypeSafe typed questions
3. TypeSafe confidence documentation
4. SimplifyAItools routing smoke test September 27
5. SimplifyAItools routing and planning study September 28
6. Refund policy development API evaluation September 30
7. Refund policy API evaluation with human-reviewed expected answers September 30
8. Fictional refund policy tested in both API evaluations

Evidence cutoff September 30, 2026. Research update 1 adds the synthetic refund-policy study to the earlier routing and planning findings. This draft has not been published.