Before changing your AI assistant, keep a small test set
Compare your current and proposed AI assistant on the same customer cases, with clear expected results and a record of what improved or broke.
By Noe Guerrero III
A shorter answer can look like an improvement until it leaves out something the customer needs. Imagine a print shop changing its AI assistant's instructions to make replies more concise. The new reply sounds better, but does it still ask for the poster quantity before sending the request to staff?
Before changing a customer AI assistant, keep a set of repeatable cases with clear expected results. Run the current version and the proposed version against those cases, then compare what happened. A saved comparison gives the owner and developer something concrete to review before the change reaches customers.
Turn a customer task into a test case
For a fictional McAllen print shop, say the approved quoting instructions ask for poster size, quantity and the date the customer needs them. Staff must confirm the price and production schedule. The assistant can collect a request and pass it along, but it can't promise that the order is accepted.
Write a case using those rules. A customer gives the size and needed date but leaves out the quantity. The expected result is a request for the missing quantity, with no invented price or promised completion date. Several different replies could satisfy that requirement. Decide which facts and actions matter before judging how polished an answer sounds.
Keep the input, the starting conversation, the approved information used for the answer and the criteria for passing together. Name who confirmed the business rule. If an expected answer has gone out of date, correct the case against the current approved information before comparing versions. Keep an AI assistant's business information current
Google Cloud's evaluation guidance recommends examples aligned with the task being evaluated and human review alongside metrics. A small business can apply that idea with a review sheet and saved conversations. Google Cloud guidance on evaluating a generative AI application
Start with a handful of tasks your staff already checks. Include a previous failure when you have one, and label invented scenarios as test examples. Keep cases that should produce different outcomes. An assistant that always asks another question might pass a missing-information case and still frustrate a customer who already provided everything.
Compare the same inputs and starting state
Record what the proposed change is meant to improve. For the print shop, it could be shorter replies that still collect the required quote information. Give the current and candidate versions the same input and approved information snapshot. Start each run with a fresh copy of the required conversation and test data so one version doesn't benefit from the other's earlier work.
Ask the developer to record the assistant instructions, model, information source version, connected tools and relevant settings for each run. Change the intended part while keeping the comparison conditions consistent. If you deliberately change several parts together, record them as one candidate configuration so the team knows what it evaluated.
Run each case more than once using the same procedure for both versions. Keep each result, including the awkward ones. AI outputs can vary between attempts, and Anthropic's evaluation guidance discusses repeated trials as a way to understand that variation. The number of repeats you choose belongs in the record. Anthropic's guide to evaluations for AI agents
For the print shop, a starter set could include these cases.
A request with all required details. The assistant recognizes the size, quantity and needed date, then follows the approved next step without asking for information already supplied.
A request missing the quantity. It asks for that detail and avoids inventing a price or production promise.
A question the approved information doesn't answer. It explains what staff needs to confirm and follows the agreed route for getting an answer.
A request to speak with a person. It follows the configured handoff. If it says a request was sent, check the test inbox or task record to confirm delivery.
The same customer task in Spanish. It preserves the business conditions and reaches the same permitted next step in natural Spanish.
Use a test environment for cases that create records or send messages. Check the outcome as well as the reply. A sentence saying the request was delivered needs evidence in the destination. Set the handoff expectations with staff before using them as a grading rule. Plan an AI assistant's handoff to staff
Review failures before approving the change
Have someone who knows the service read both sets of results against the written criteria. For each attempt, record a pass or fail and a short reason. For example, “Asked for quantity, preserved the requested date, made no price promise” gives the reviewer a basis for passing the missing-quantity case. “Polite reply” leaves the business task unresolved.
Keep correctness and style visible as separate observations. A candidate can improve brevity while getting the quote process wrong. Agree beforehand which failures should stop this particular change from going live. For the fictional shop, claiming that staff accepted an order without confirmation would be one such failure.
When a case fails, save the conversation and the observed outcome with its version information. Decide whether the problem is in the assistant, the available information or an ambiguous test expectation. Fix it, then rerun the full set so you can see whether the correction affected another case.
Keep the comparison with the release decision, including known failures and who reviewed them. Add cases as real customer questions reveal gaps. The saved set becomes a repeatable way to discuss the next change, using the tasks the business expects its assistant to handle.