Six months after a small business launches an AI tool, there is enough operational history to ask better questions than those posed during procurement. Has the tool completed the job it was purchased to support? Do employees trust it for the right reasons? Have the supplier's features, model behaviour or data practices changed? A useful review must answer these questions from verifiable records rather than usage charts or enthusiasm. Its purpose is to decide what the business should retain, adjust, restrict, replace or stop.
Recover the original purpose and approval
Find the buying brief, pilot acceptance criteria and the tasks the assistant was authorised to perform. Compare them with what it actually does today. A product introduced to draft internal text may now answer customers directly or access a booking system. That expansion cannot automatically inherit the risk decision made for a lower-consequence use case. Record every material change in purpose, audience, data category or action authority and identify who approved it.
Ask whether the original business problem still exists. A redesigned website, new staff member or changed supplier process may have removed the bottleneck that justified the tool. Continuing a subscription solely because it was purchased six months ago is not an operational argument.
Measure actual customer and staff outcomes
Start with the same outcome definition used in the pilot. If the goal was faster handling of enquiries, distinguish a quick acknowledgement from a useful first answer and a verified completed task. If the goal was reduced document drafting time, include employee editing and correction effort. Where a tool changes real systems, inspect those records rather than accepting its own summary that the task succeeded.
Compare similar work cohorts and note differences in seasonality, staffing and business activity. No claim of cost reduction should be drawn simply from higher conversation volume. A small number of excellent use cases can justify continued use; a high-usage tool that produces extensive hidden repair work may require restriction.
Review exceptions, corrections and near misses
Collect the cases where staff corrected false information, a customer contacted the business again or a connected service failed. Categorise causes such as stale content, misunderstood requests, inadequate permissions, system outages and inappropriate scope. Include near misses, because a dangerous proposed action that a staff member intercepted reveals an important control weakness even if the customer was not harmed.
Prioritise by consequence as well as frequency. An awkward greeting is not equivalent to an incorrect account disclosure or a fictional booking confirmation. Decide whether the root cause needs better documentation, model evaluation, a technical restriction or a human-only process. Record the corrective action and retest the affected scenario.
Recheck the knowledge sources used in production
Identify which policies, service descriptions and operational records the assistant consults. Confirm that each source has an owner and a known current version. Outdated pricing, changed cancellation terms and expired contact details can turn a previously accurate assistant into a source of customer confusion. Where sources conflict, the correct behaviour should be explicit rather than letting the model choose whatever wording sounds plausible.
Examine recent updates and ask whether the approval and publication process actually reached the live AI system. A policy edited on the website may not have been refreshed in a separate knowledge index. Test the known difficult cases after the source is updated.
Audit tools, permissions and human approvals
A product may acquire additional CRM, calendar or messaging integrations after launch. List every accessible system, credential, operation and responsible owner. Check which actions are read-only, which change business records and which need human approval. Remove permissions that no longer serve the approved purpose.
Do not rely solely on a prompt telling the assistant to avoid prohibited actions. Verify technical scopes and rejected operations in a controlled test. Confirm that an actual failed booking or refund is reported as unsuccessful rather than rephrased as a completed request.
Reassess privacy and retention
Review what personal information enters prompts, vendor logs, case summaries and analytics. Decide which records remain necessary for unresolved customer work and which were retained by default without a continued purpose. The ICO's data-minimisation guidance and storage-limitation guidance support collecting and keeping only personal data justified by its use.
Examine access by former employees, unused administrator accounts and external processors added during implementation. Ensure customers' account data does not appear in general analytics reports merely because it was available to the model during a service conversation.
Review supplier changes and dependencies
Ask whether the vendor changed its model, pricing tiers, service limits, subprocessors or integration behaviour. Identify which changes were material to the company's risk controls and whether users were notified. The NCSC secure AI system development guidance encourages managing security through deployment, operation and maintenance. A six-month review should therefore include supplier security and continuity, not only the quality of generated prose.
Test how staff would export essential records or operate manually if the supplier were unavailable. A feature is not dependable merely because it worked during the review meeting.
Ask employees who carry the downstream work
Speak to staff using the tool, supervising its output and receiving the cases it cannot handle. An employee who sees only polished AI summaries may miss the extra work another colleague performs correcting them. A specialist may find that the assistant gathers useful facts but misclassifies important exceptions. Record these experiences and check them against real cases.
Discuss whether changed duties require training or updated cover arrangements. A review should not become an opaque productivity ranking of individuals. Focus on the workflow and the conditions needed for staff to deliver accurate service.
Include the customer's perspective
Examine repeat contact, complaints, incorrect promises, accessibility problems and difficulty reaching a person. Customers may be satisfied with an immediate answer to a routine question and frustrated by an automated loop when the matter becomes unusual. Both outcomes matter. Use proportionate feedback and case review rather than interpreting silence as proof that the service is good.
Where customers need a human, check whether the transfer retains the important context. A handoff that makes somebody repeat their information can erase the benefit of faster automated intake.
Re-run realistic acceptance tests
Use a small but representative set of tests from before launch, updated for current policies and integrations. Include ambiguous questions, unavailable information, disputed prices, unauthorised account changes, duplicate bookings, provider timeouts and attempts to treat untrusted documents as instructions. Define the expected result before running each case. Inspect actual operational records and the escalation destination, not only the chat reply.
Test a narrow mobile layout, keyboard navigation and the visible route to human assistance where the customer interface is affected. A text-based audit cannot prove that a website remains accessible after a design change; the relevant rendered environment must be checked.
Compare the full cost of continuing
Include the current subscription, usage charges, integrations, employee monitoring, source maintenance, correction effort and recovery from faults. Compare those with a credible non-AI alternative delivering the same verified task. The software may be valuable even if it does not eliminate headcount, but the business needs a transparent account of what it is buying.
Where benefits are intangible, such as clearer staff handoffs or better after-hours information, describe them separately and obtain appropriate evidence. Avoid converting every hypothetical minute saved into profit without establishing a financial consequence.
Decide whether to retain, improve, restrict, replace or stop
A six-month review should end with a specific decision for each workflow. Retain a bounded function that works reliably and has clear ownership. Improve one with remediable data or process problems. Restrict or pause capabilities that repeatedly produce consequential errors. Replace a supplier when important needs cannot be met under acceptable controls. Stop a use case when the business problem no longer justifies the risk or operating cost.
Assign an owner and date for each action. A vague decision to continue monitoring is insufficient when a specific access or customer-harm issue has been identified. Record what evidence would permit an affected function to resume.
Set triggers for review before another six months
The next evaluation need not wait for an anniversary. Significant changes in model behaviour, data flow, tool permissions, business policy or customer-facing use can justify earlier reassessment. Define thresholds for repeated incorrect confirmations, data exposure and failed escalations that lead to immediate review.
Maintain a short, usable decision record: what was evaluated, which evidence was inspected, what changed and who approved the result. The goal is an operating tool whose benefits and limits are understood by the people responsible for the business.
Keep a balanced view of success
An AI system may deliver value through routine knowledge access while remaining unsuitable for refunds or specialist advice. The review should not force one overall success label across unrelated tasks. Inspect each capability on its own evidence and recognise that a narrow reliable service can be preferable to an ambitious uncontrolled platform.
The most useful six-month evaluation translates actual experience into a better operating decision. It reconnects the technology to the original business goal, verifies outcomes, examines risk and makes ownership visible. That is how a small team can learn from deployed AI without mistaking continued use for proof of value.
Retain, improve or pause: what should a six-month review decide?
A review must lead to different actions depending on the evidence. Continuing a workflow because staff use it is not the same as proving that it remains safe, useful and worth its operating cost.
| Check | Retain or improve | Restrict, replace or stop |
|---|---|---|
| Observed value | Verified task outcomes justify continued use with manageable supervision. | Benefits are unproven or offset by substantial correction and maintenance work. |
| Errors | Known low-consequence defects have owners and effective fixes. | Serious false actions, privacy incidents or unmanaged failures remain. |
| Data access | Only necessary records and permitted operations remain available. | Over-broad access, unexplained retention or supplier risk cannot be justified. |
| Supplier continuity | Updates, incident response and export arrangements remain acceptable. | Critical dependencies, missing controls or exit risks undermine reliable operation. |
| Next action | Assign improvements, owners and a repeat testing date. | Pause risky capabilities, preserve service fallback and define remediation criteria. |
Practical recommendation: record a decision for each meaningful capability, not one blanket thumbs-up for the whole product. Reassess earlier than six months if risk or permissions change materially.
Source A — evaluating AI risk: UK Government, Introduction to AI Assurance (UK government guidance, checked 27 October 2026), relevant to evaluating and communicating trustworthiness; not a prescribed six-month certification.
Source B — operating controls: NCSC, Secure operation and maintenance (AI security guidance, checked 27 October 2026), supporting continuous monitoring, updates and incident response. Neither source specifies a universal retention threshold or a quantitative AI return.
Frequently Asked Questions
Why review an AI tool after six months?
Actual usage, corrections, supplier changes and customer outcomes provide stronger evidence than the original demonstration. Six months is a useful checkpoint, not a universal regulatory deadline.
Which evidence matters most?
Start with the original business task, verified completion outcomes, staff correction effort, unresolved exceptions and any consequential errors. Compare actual operation with the approved scope.
Should we review personal data and permissions again?
Yes. Check connected systems, user access, subprocessors, transcript retention, exports and whether each data flow remains necessary and authorised.
When should we pause an AI workflow?
Pause or restrict capabilities when serious unauthorised actions, exposed information or repeated false confirmations cannot be controlled. Keep an alternative service route.
What should the review produce?
A documented retain, improve, restrict, replace or stop decision, with evidence, owners, corrective actions and the next test or review trigger.