Scenarios to test
- A direct question about a fact present in the description.
- A short continuation containing a pronoun or omitted product name.
- A new product name that is absent from active listings.
- A question about the full assortment rather than one card.
- An objection that needs a persuasive but honest answer.
- A question about a discount, order, or another seller decision.
- A follow-up after an inaccurate answer.
Signs of a correct reply
- The opening sentence answers the latest question.
- Facts belong to the correct item and do not move between listings.
- A newly named model is not replaced with a similar item from stock.
- Already stated attributes are not repeated without a reason.
- The reply contains no invented check, personal experience, guarantee, or promise.
- An unknown private decision is handed to the seller with a specific reason.
Fix the source instead of one sentence
If an answer is wrong because the description is outdated, correct the listing. If a stable private condition is missing, add it to the product information. If the issue belongs to one conversation, add context for that chat. Do not create a separate prepared answer for every buyer phrase, because the next buyer will express the same need differently.
When to switch to automatic mode
After testing different scenarios on one listing, connect the next listings. The activity log should show more than the final text: it should expose the handoff reason, selected listing, and timing of each stage. This reveals a systemic cause instead of reducing quality review to one screenshot.
A review system that reflects real quality
One successful reply does not prove that automation is consistently reliable. Review must cover short and long chats, several similar listings, objections, follow-ups without a product name, and seller-owned decisions. The most valuable outcome is not a style score in isolation but evidence that the answer helped the buyer take the next step without invention or repetition.
Which conversations should be reviewed daily?
Review new products, long threads, answers after price or note changes, seller handoffs, and cases where a buyer sent several brief messages. Add a random sample of ordinary chats so the review is not limited to known problems. Together these groups provide an honest view of everyday performance.
What counts as a correct answer?
It answers the newest intent within full history, uses facts from the right item, does not invent seller decisions, avoids repeating known information, and sounds natural in the buyer's language. For an assortment request it names all relevant active options instead of using a vague phrase such as “other models.”
How do facts differ from decisions?
A listed price, documented set, and public specification are facts. A discount, reservation, delivery exception, or individual warranty may require the seller's decision. Review should test that boundary: automation must be confident when data exists and involve a human only for choices the seller genuinely owns.
How can harmful repetition be found?
Read several seller responses in sequence rather than reviewing each in isolation. Different words may still repeat the same argument and frustrate a buyer. A later response should advance the conversation by resolving the new concern, adding another relevant fact, or offering a concrete next step.
What should follow a discovered error?
First inspect its source: listing, note, conversation history, or general seller rule. Correct the data or universal instruction rather than adding a prepared answer for one phrase. Then rerun the scenario beside neighbouring cases so one improvement does not make another product or category worse.
How do you know a change is safe?
The new scenario works with realistic conversational forms while core functions such as price, condition, included items, assortment, handoff, reminders, and language remain correct. Tests must reproduce full context and state between turns. A second turn with blank memory cannot validate the conversation production actually runs.
What creates a false sense of quality
- Testing only one perfectly written request without history, informal language, errors, or similar products in the catalogue.
- Treating a green technical status as proof of a good reply without reading the actual text as a real buyer would.
- Fixing every poor example with a separate template, dictionary, or filter that unpredictably damages other categories.
- Testing the second turn with empty state even though production remembers previous facts, actions, and seller messages.
- Judging politeness alone while ignoring subject accuracy, factual completeness, persuasion, and a practical next step.
Maintain a compact scenario matrix by capability rather than by individual buyer words. It can cover product selection, listing facts, public research, objections, assortment requests, seller decisions, consecutive messages, and continuation after a human response. Store full realistic history and an expected outcome for each scenario, but never a required prepared sentence. This gives the model room for natural language while keeping meaning testable. Run every change with the baseline set, not only the case that was just corrected. Regularly add anonymized real question shapes containing typos and short follow-ups. Analyze a bad result from source to final text: which data arrived, what semantic decision was produced, and whether the writer preserved it. This finds root causes without accumulating filters that make one test green while damaging unpredictable live conversations.
Review successful complex conversations as well as failures. They reveal which data and instructions already work and must survive the next change. Before release, compare old and new results with the same seller pack and history. After release, verify technical health without sending tests to real buyers, then read the first natural conversations carefully. Quality control is a continuous evidence cycle, not a one-time reaction to the loudest complaint.