In an illustrative failure, the demo works on tidy examples, then a long attachment triggers a second model call, the parser rejects the output, and someone spends six minutes repairing the result. The provider invoice records usage; it does not tell you whether the product job was worth completing.
Start with a release question, not a model contest
Choose one product job: classify an inbound request, extract fields from a document, draft a reply, or flag an exception. Write the decision before the experiment: ship the job as designed, narrow its scope, change the configuration, keep a mandatory human step, or reject the feature.
Do not compare models across changing prompts and call the winner “better.” Freeze a configuration card with the provider, exact model identifier, modality, prompt version, parser schema, tool permissions, retrieval snapshot, retry rule, data settings and review date. OpenAI’s official evaluation guidance, observed 28 July 2026, supports task-specific criteria, representative cases and explicit edge cases; it does not supply a transferable quality score.
Build one ledger around the product request
A provider attempt is not the same thing as a customer job. Give the original job a durable product_request_id, then attach every retry, fallback and repair action to it.
| Record | Fields that change the decision | Why it exists |
|---|---|---|
| Configuration card | Feature version, provider/model, prompt, schema, endpoint, data settings, rate-card date | Stops two different systems being compared as if they were one. |
| Attempt log | Request ID, attempt number, billed units by category, tool/media charges, latency, provider outcome | Keeps retries and expensive tails visible. |
| Review log | Reviewer action, minutes, correction type, final route | Shows whether automation moved work or merely hid it. |
| Failure label | Missing fact, unsupported addition, parse failure, boundary breach, prohibited data, fallback | Turns “quality” into defects attached to the job. |
| Decision line | Product limit, result, owner, next trigger | Forces an operating choice instead of another demo. |
OpenAI’s usage and cost APIs, documented as of 28 July 2026, can support reconciliation by dimensions such as model, project or key. They are provider aggregates, not a product-level request ledger. Keep both records and investigate mismatches.
Use a configurable cost model
The provider’s dated rate card is the source for the exact model, modality, cached input category, tool and service tier tested. Rates, plans and billing categories are mutable, so save the rate card used for the run and check the current page before repeating it.
illustrative_request_cost =
sum(provider charge for every attempt under product_request_id)
+ (measured review minutes × your loaded review rate)
+ documented recovery or support allocation
illustrative_feature_period_cost =
sum(illustrative_request_cost for the selected product jobs)These are decision formulas, not universal accounting definitions. If you allocate revenue to a bundled feature, record the rule and test more than one plausible allocation. Do not manufacture precision from a disputed assumption.
Label the cases that can stop the release
Create cases from authorised product material or mark them synthetic. Each case needs required facts, forbidden claims, the expected route, the consequence of being wrong and a reviewer rubric written before the run.
Illustrative scenario: a document-expiry feature extracts a document type and date, then proposes a reminder. The deck includes a clear date, two competing dates, no date, an unsupported document type, handwritten text and a file containing data the endpoint is not approved to receive. This scenario is fictional and carries no measured pass rate.
| Observed outcome in your run | Product question | Possible decision |
|---|---|---|
| Required fact is unsupported or missing | Can the job safely pause for clarification? | Narrow the feature or keep mandatory review. |
| Parser fails but the answer is readable | Is repair predictable and cheaper than fallback? | Fix schema handling before changing models. |
| Prohibited data crosses the boundary | Can the input be reduced before any provider call? | Block release until the data path changes. |
| Retries dominate expensive cases | Is the retry solving a transient failure or repeating bad input? | Route invalid cases out; cap safe retries. |
| Human review remains consequential | Does removing it create an unacceptable action? | Keep the manual exit as product design. |
Inspect the exact data path
OpenAI’s data-control documentation, observed 28 July 2026, describes behaviour that varies by endpoint, storage setting, account eligibility and region. Do not turn a general documentation page into a claim about your deployment. Record the endpoint, retention and storage choices actually approved before sending product data. Check the current provider documentation after any relevant endpoint, storage, account or regional configuration change.
A leaderboard cannot answer the release question
It cannot decide whether your parser survives, your forbidden cases route correctly, your reviewers can recover the job, or your cost boundary holds on your workload. Also ignore the deprecated OpenAI Evals API as an architectural dependency: the provider documentation reviewed on 28 July 2026 announces read-only status on 31 October 2026 and shutdown on 30 November 2026. Keep the dataset and rubric portable.
The release card
Complete one decision, not a generic score
Where the cheap demo gets expensive
The first call is the easy number to see. The second call, the parse repair and the person checking whether the product did the right thing are easier to lose. Put them on the same request ID before anyone calls the feature cheap.
Keep the feature behind review until the evidence exists
Do not ship the AI feature from a clean demo. Freeze one configuration, run an authorised labelled set, count every attempt and recovery step, and keep a manual exit. If those records are unavailable, the concrete decision is to keep the feature behind human review—not to invent a benchmark.
Pricing and model sources
- OpenAI, API pricing, observed 28 July 2026; check the current rate card for the exact model and service tier used.
- OpenAI, Usage API reference, observed 28 July 2026; confirm current fields and account eligibility before relying on an export.
- OpenAI, Evaluation best practices, observed 28 July 2026; adapt the guidance to the product’s own labelled examples and failure costs.
- OpenAI, Your data, observed 28 July 2026; verify again for the exact endpoint and account configuration.