Journal / Kirill Naumov
A proposed evaluation method for retail AI: waiting time, conversion, repeat visits, team adoption and the cost of checking mistakes. Includes an illustrative calculation.
By Kirill Naumov ·
A pilot can produce impressive answers and still make a shop harder to run. The useful question is not how many prompts the team sent. It is whether one customer moment became better without moving the work somewhere else. This is a proposed evaluation method, not a report of a client deployment.
Take a sales colleague answering questions about product compatibility. Define the eligible questions, the approved product information, the person responsible for updates and the situations that must go to a specialist. Compare AI assistance with the current process, including a simpler searchable reference sheet. AI should earn its place against a realistic alternative.
Before launch, record a baseline across comparable shifts. Keep the definition of an eligible interaction unchanged. Note promotions, staffing, product availability and footfall: a conversion change during a sale is not automatically an AI effect. Where feasible, compare similar shifts or groups running at the same time. Set the observation period and required sample with someone able to assess the variability in your data.
Customer effort: measure median waiting time and the slowest tenth of interactions. An improving average can hide a worse experience for customers with difficult questions. Record abandoned interactions separately.
Commercial outcome: define conversion as purchases divided by eligible interactions, using the same denominator in both groups. For repeat visits, use comparable customer cohorts and a stated observation window. A customer who has not had time to return should not be counted as a failure.
Answer quality: have a qualified reviewer check a sample against approved product facts. Track critical errors separately from wording preferences. Set stop conditions before launch; do not trade unsafe advice for a faster response.
Team adoption: compare eligible interactions where the tool was used with all eligible interactions. Record when colleagues override it and why. Low usage may point to missing information, a slow interface or a workflow that needs another login—not a team that needs another motivational speech.
Operating effort: include review time, corrections, training, information updates, integration support and software costs. Time saved on the shop floor is not a saving if another team spends more time correcting it.
Suppose 1,000 eligible interactions take two minutes each today. An assistant reduces the initial task to one minute, but checking and corrections add an average of 30 seconds. Net time released is 500 minutes, not 1,000. This does not automatically reduce payroll: the time must be usable for another task or change a real cost before it becomes financial value.
Likewise, conversion moving from 20% to 22% is a two-percentage-point change, not proof of causation. Compare the groups, uncertainty and changes in traffic or stock before attributing the difference to the tool. These numbers are hypothetical.
Before starting, name the decision owner and agree on three possible outcomes: expand, revise or stop. Expand only when the customer measure improves, quality remains acceptable and the ongoing work is affordable. A stopped pilot can be a good decision if it prevents an expensive rollout.
A 45-minute diagnostic with Kirill Naumov costs $500 and helps identify three priorities and an initial hypothesis. Instrumentation, implementation and an evaluated pilot are separate work. Bring one process, the available baseline and the person who owns the outcome.