Test Whether an AI Summary Preserves Unresolved Questions

A short handover can sound decisive while losing the one question that still blocks the work. If a source says a delivery date is proposed but unconfirmed, a summary that calls it scheduled changes the meaning. The test is not simply whether the paragraph reads well.

For an AI summary used in business operations, define which unresolved facts must survive compression. The following exercise uses fictional notes and manually written examples of acceptable and unacceptable wording. It is a proposed evaluation method, not a benchmark result or a claim about a particular model.

AI-generated conceptual illustration; not a client screenshot.

Define the handover decision first

Choose one task, such as summarizing a service-request conversation for the next operator. State what the reader should know before taking action: what is confirmed, what is proposed, which statements conflict and what still needs a decision.

Anthropic's evaluation guidance recommends specific, measurable criteria aligned with the application's purpose. For this exercise, turn “accurate summary” into a reviewable requirement: no unresolved approval may become an approved action, and every designated blocking question must remain visible.

Agree on those blocking questions before reviewing the output. Otherwise, reviewers may reward a polished paragraph and overlook the fact that it answers a different operational question.

Create a small source packet with deliberate uncertainty

Use clearly labeled fictional notes, with stable line references:

• Note A: The customer requests a visit on Thursday. The service coordinator has not confirmed availability.
• Note B: One message says the access gate opens at 08:00; another says 09:00. Neither is marked as a correction.
• Note C: A replacement part may be needed. Inspection has not happened and no purchase is approved.
• Note D: The coordinator will check availability. Nobody is assigned to resolve the gate-time discrepancy.

Record the expected interpretation separately: Thursday is requested, the access time is contradictory, the part requirement is conditional and one follow-up lacks an owner. Do not choose the later message as authoritative unless the source or agreed workflow establishes that rule.

Review support and coverage separately

Google Cloud's question-answering evaluation templates distinguish groundedness from completeness. A response should be supported by its context, while also covering the required information. That distinction helps explain why a summary can avoid invented facts and still omit an important blocker.

Use a compact review sheet with one row per required point. Record its source reference, the summary's corresponding wording and a result: preserved, omitted, contradicted or strengthened beyond the source. Treat “not mentioned” differently from “not required.”

An acceptable illustrative sentence is: “Thursday is requested, pending availability confirmation; gate access time remains disputed.” An unacceptable sentence is: “The visit is booked for Thursday at 09:00.” The second wording invents confirmation and resolves a contradiction without evidence.

Keep next actions separate from source facts

A summary may suggest asking someone to confirm the gate time. Label that as a proposed next step. It must not claim that a person accepted responsibility when the source never assigns one.

Ask reviewers to check certainty words: requested, expected, confirmed, approved, possible and completed have different meanings. Also check negation. “No purchase is approved” must not become “Purchase approved” or disappear if that restriction matters to the handover.

If the output format includes an owner or due date, allow an explicit “not assigned” or “not provided” value. Requiring a filled field must not pressure the system into inventing a name or deadline.

Test small changes without changing the question

1. Move the unresolved note earlier or later in the source packet and check whether it remains represented.
2. Add a clear correction that resolves the gate time. Verify that the summary updates the fact and no longer presents that specific conflict as unresolved.
3. Remove the availability owner's assignment. Confirm that the output does not preserve an owner from an earlier version.
4. Request a shorter summary. Check whether designated blockers survive even when background detail is removed.

Keep source version, instructions, model identifier and available settings with each observed output. Review several outputs when the system can vary between runs. Report the cases actually tested and the failures found; one passing example is not evidence of dependable behavior across all requests.

Decide what happens after a failed review

For this workflow, require a human check before an uncertain summary drives scheduling or purchasing. Preserve the source references so that the reviewer can resolve discrepancies rather than guess from the condensed text. Record the correction and add a representative failure to the next evaluation set.

Contact Dynelink to define a focused evaluation for AI summaries used in your team's operational handovers.