Once an AI feature has a clear product role, evaluate it with two linked records: a request-cost log and a fixed set of labelled cases. Provider price, output quality, retries, and review effort all belong in the same decision.
Illustrative sheet design and formula arithmetic. No provider prices, model economics, latency benchmarks, quality rates, or product results are supplied here. Enter dated values from the provider you are testing and measurements from your own workload.
Freeze what you are evaluating
A quality score cannot be compared cleanly when the prompt, model identifier, tools, retrieval data, or output parser changes between runs. Give each evaluation a versioned configuration record.
| Feature version | Internal release or experiment identifier |
|---|---|
| Provider and model identifier | Exact value sent to the provider, plus observation date |
| Prompt version | Immutable prompt or template identifier |
| Structured output schema | Schema version and parser version |
| Retrieval or tool inputs | Data snapshot, search scope, and tool permissions |
| Generation settings | Every configurable value used in the request |
| Retry policy | Which failures retry, maximum attempts, delay, and idempotency behavior |
| Fallback | Manual path or non-AI behavior when the request fails or is rejected |
Request-level cost log
Store one row per original product request and separate provider attempts underneath it. Otherwise retries disappear into an average and make the feature look cheaper than it is.
| Field | What to record |
|---|---|
request_id | Internal identifier shared by all attempts for the same product action. |
account_id or segment | A privacy-appropriate identifier or cohort needed to understand usage distribution. |
feature_version | The frozen configuration evaluated. |
provider_model | Provider and exact model identifier used for the attempt. |
input_units and output_units | The provider-billed units reported for the attempt, split by billed category where relevant. |
provider_rate_snapshot | Dated input, output, cached, tool, image, or other applicable rates and billing units. |
attempt_number | Original call or retry. |
latency | Measured request duration for the attempt and end-to-end feature duration. |
outcome | Accepted, edited, rejected, failed, fallback, or abandoned, using definitions written in advance. |
review_minutes | Measured human review and correction time for the product request. |
support_or_recovery_cost | Product-specific amount assigned only when the method for estimating it is documented. |
Spreadsheet formulas without invented rates
Put provider rates in a dated assumptions tab. Keep raw usage in a request log. The formulas below describe the sheet; they do not assert that a provider bills every model in the same units or categories.
input_cost
= (input_units / provider_input_billing_unit) * provider_input_rate
output_cost
= (output_units / provider_output_billing_unit) * provider_output_rate
attempt_provider_cost
= input_cost + output_cost + applicable_tool_or_media_cost
request_provider_cost
= SUM(attempt_provider_cost for every attempt sharing request_id)
review_cost
= (review_minutes / 60) * loaded_reviewer_hourly_cost
request_variable_cost
= request_provider_cost + review_cost + support_or_recovery_cost
account_feature_cost
= SUM(request_variable_cost for the account and billing period)
feature_contribution_after_variable_cost
= revenue_allocated_to_feature
- account_feature_cost
- other_feature_specific_variable_cost“Revenue allocated to feature” is a product assumption, not a fact created by the formula. If the feature is bundled, show the allocation rule and run the decision with more than one plausible allocation rather than treating one answer as precise.
Use the distribution, not one average
A short input and a long document may have very different cost and review profiles. Group requests by the characteristics that drive work in your product: input size, output type, account segment, language, attachment count, retrieval use, or escalation path.
| Slice | Request share | Provider cost distribution | Review-time distribution | Failure and fallback count |
|---|---|---|---|---|
| Short, routine inputs | Enter observed share | Enter median and tail values from the log | Enter measured values | Enter counts |
| Long or attachment-heavy inputs | Enter observed share | Enter median and tail values from the log | Enter measured values | Enter counts |
| Ambiguous or exception cases | Enter observed share | Enter median and tail values from the log | Enter measured values | Enter counts |
| Requests requiring retry or fallback | Enter observed share | Include every attempt | Include investigation and recovery | Enter counts |
Choose slices from actual traffic or a clearly labelled pre-launch dataset. Do not claim they represent customers if they were constructed by the team.
Build a labelled evaluation set
The set should reflect the approved feature boundary from the AI product-fit decision. Include routine cases, hard cases, and prohibited cases that must route to a human or fallback. The necessary size depends on the variety and consequence of the workflow; there is no universal call count or correction-rate threshold here.
| Case field | Definition |
|---|---|
case_id | Stable identifier that survives reruns. |
input_source | Permission and provenance: approved historical item, synthetic case, or constructed edge case. |
required_facts | Facts the output must preserve or extract. |
forbidden_claims | Statements or actions the output must not introduce. |
expected_route | Generate, ask for clarification, refuse, or fall back to a person. |
severity_if_wrong | Product-defined consequence category with an owner and recovery path. |
reviewer_label | Pass, edit, reject, or unsafe according to the written rubric. |
review_notes | The exact missing fact, unsupported addition, formatting defect, or harmful action. |
A rubric tied to the job
Illustrative rubric, not a quality benchmark. Change the criteria before running the set, then keep them fixed for comparisons.
| Criterion | Pass | Edit | Reject |
|---|---|---|---|
| Required facts | All required facts are present and traceable to the permitted input. | A non-consequential fact is missing but can be corrected quickly. | A required fact is wrong, invented, or omitted in a way that changes the action. |
| Boundary | The output stays within the approved draft, extraction, classification, or suggestion role. | The output adds harmless material outside the preferred format. | The output makes a decision or promise reserved for a person. |
| Uncertainty | The output asks for clarification or routes to fallback when the input is insufficient. | The uncertainty is visible but awkwardly expressed. | The output presents an unsupported answer as settled. |
| Format | The product can parse and display it without repair. | A person can repair it without changing substance. | Parsing fails or the structure could trigger the wrong product action. |
| Data handling | The request and output stay inside the approved data boundary. | Non-sensitive excess data appears and the input design should be tightened. | Prohibited or unapproved data is sent, exposed, or retained. |
Cost and quality decision sheet
Feature version and evaluation date: ____________________
Provider/model and dated rate source: ____________________
Traffic slices represented: ____________________
User-defined maximum variable cost per successful job: ____________________
User-defined unacceptable outcomes: ____________________
Observed provider-cost distribution: ____________________
Observed review-time distribution: ____________________
Failures, retries, and fallbacks included in cost: yes / no / incomplete
Decision: ship / narrow / change configuration / retain manual path / reject
Reason tied to evidence: ____________________
Re-evaluation trigger: provider/model change / prompt change / traffic shift / new failure mode / price change / other
Read provider spend beside review and recovery work. Then compare the measured total with the product’s own cost boundary and the labelled cases with its own error boundary. Re-run both records whenever the frozen configuration or workload changes.