Version an AI Prompt Change with a Repeatable Before-and-After Test

A small instruction change can alter how an AI feature handles cases that previously worked. Before replacing a prompt, make the old and new versions comparable and give a reviewer enough evidence to decide whether the change is acceptable.

This is a proposed release exercise for a fictional request-routing assistant. It describes a test plan, not a test performed for a client or measured improvements. The assistant suggests a queue; a person retains responsibility for the routing decision.

Describe the change in one sentence

Suppose the current prompt assigns requests to billing, technical support or general review. A proposed revision clarifies what to do when a message contains two unrelated requests. Record that specific intention before editing. Keep unrelated changes to tone, output format and model choice in separate comparisons.

Google describes prompt design as an iterative process and recommends refining instructions against observed responses. That supports checking a revision against examples; it does not establish that a particular wording will improve your application. See the Google prompt design guide.

Keep a release record that identifies both versions

• Prompt: save the complete old and proposed instructions, their version identifiers and the reason for the change.
• Execution context: record the model identifier, available version information, generation settings and application configuration. Note anything the provider does not expose or let you hold constant.
• Inputs: freeze a named set of synthetic or approved, suitably redacted cases. Keep the same input and supporting context for each comparison.
• Judgment: version the acceptance rules, expected handling and reviewer notes alongside the cases.

A repeatable procedure does not promise identical generated wording. Record variations across attempts and keep the attempt policy consistent for both versions. If the model or context changes at the same time, label the result as a comparison of combined changes rather than attributing it solely to the prompt.

Compare cases, not just a single total

The following are proposed checks for the fictional routing change. Their expected outcomes are business decisions to approve before collecting outputs, not provider guarantees.

• One clear billing request: preserve the established billing suggestion.
• One clear technical request: preserve the established technical suggestion.
• Two unrelated requests: follow the newly approved handling rule, such as suggesting general review with both requests identified.
• Insufficient information: retain the approved review path instead of inventing a missing detail.

Save both outputs for every case, the applicable rule and the reviewer's decision. Separate improvements in the targeted case from regressions elsewhere. Anthropic distinguishes evaluations of new capabilities from regression evaluations that protect existing behavior in its guide to evaluating AI agents. The same distinction is useful when planning this smaller prompt comparison.

Decide what blocks release before seeing the results

For this exercise, propose blocking a revision that loses a required review path or breaks an established output contract. Have the responsible owner approve those criteria first. Do not change the rubric afterward just to make the new version pass. Record disputed cases and resolve them explicitly.

Keep the previously approved prompt and configuration available, identify who can approve a rollout or rollback, and define what evidence would trigger that decision. A passing example set supports a bounded release decision; it does not prove performance on every future request or demonstrate business savings.

Prepare a reviewable change brief

Bring the current instruction, proposed change, frozen cases, acceptance rules and unresolved questions to a technical discussion. Contact Dynelink to discuss the scope of your AI integration.

AI-generated conceptual illustration; not a client screenshot.