Practical AI: How to Design Around Its Weaknesses

The most dangerous AI failure is often a result that looks completely normal. A confident answer can cite a non-existent policy, an agent can follow malicious instructions embedded in a document, or a workflow can duplicate a payment after a timeout. Organisations should not assume that an apparently intelligent interface is intrinsically reliable. A practical pre-mortem imagines that an AI project has failed and asks which safeguards were missing. Five recurrent failure modes deserve particular attention: incorrect information, sensitive-data exposure, excessive agent authority, prompt injection and silent cost or quality deterioration. NIST's AI Risk Management Framework and OWASP's GenAI Security Project offer useful approaches for identifying and managing these risks.

Failure one: a convincing but incorrect answer

Hallucination includes more than invented quotations. A model may confuse customers, combine figures from different accounting periods, overlook a contract exclusion or infer a technical root cause from incomplete logs. Fluency makes the error easy to miss. The control is an evidence route: show which trusted document supports a material claim and verify the relevant detail. For regulated or financially consequential use, checks must be stronger than simply asking another model whether the answer looks accurate. A supporting reference can also be wrong, so reviewers need access to the original document and its current version.

Make abstention an acceptable outcome

A system that must answer everything will eventually fabricate certainty. Define which queries it can answer from approved knowledge and what should happen when facts are missing. The assistant may ask for further context or route the case to an authorised employee. Test questions that intentionally lack answers in the source corpus. A useful refusal describes the limitation accurately rather than suggesting that unavailable facts were checked. The goal is not to maximise response completion, but to support correct decisions without unsupported claims.

Failure two: unintended data disclosure

Workers may paste contracts, confidential files, passwords or personal information into an apparently private interface. A retrieval application may also expose documents beyond the requester's permissions. Before connecting a model, classify the information involved and confirm the permitted processing system, retention rules and access controls. Minimise prompt data, use secure secret management and restrict retrieval according to the user's actual rights. Do not assume the model will reliably hide information simply because an instruction says it should. Confidentiality must be enforced by surrounding systems.

Consider cross-client and tenant boundaries

Service providers may hold documents for multiple organisations. A shared vector index or agent tool must not allow customer A's users to retrieve customer B's records. Enforce tenant isolation in queries, storage and tool identity, then test with examples designed to cross that boundary. Logging should preserve enough evidence to investigate improper access without itself becoming a store of uncontrolled secrets. The platform operator should know exactly which system supplies authorisation and what happens when account privileges change.

Failure three: excessive agency

Models that can send email, update databases or deploy code create real operational consequences. An assistant asked to draft a reply should not automatically receive unrestricted sending rights. An agent reading tickets should not have broad deletion or closure permissions. Assign the smallest action set required by the task and separate proposal from execution. A person or deterministic business control should approve consequential changes. The more systems an agent can access, the more important it becomes to review not only whether each individual tool works, but whether a chain of actions can cause unintended effects.

Design safe transaction handling

An agent may repeat an action after an ambiguous timeout. Without idempotency and read-back, one intended refund can become two, or one message can reach a customer several times. Use stable transaction identifiers, explicit state checks and bounded retry policies. Confirm what the target system actually did before another write. A model-generated statement that an update succeeded is not equivalent to authoritative evidence. The system should distinguish prepared, submitted, accepted and completed outcomes and preserve an audit path for human investigation.

Failure four: prompt injection from untrusted content

A website, email or retrieved document can contain instructions intended to redirect a tool-using model. The text may ask the assistant to ignore its rules, disclose confidential information or send data to an attacker-controlled destination. Such material is task data, not an authorised instruction. Keep system-level permissions outside the retrieved content and validate every requested action against actual business policy. An agent should not gain a new destination or privilege because a webpage says it is required. Test with benign simulated attacks and record how the system responds.

Apply layered defences rather than prompt magic

Writing a stronger prompt is not sufficient to guarantee resistance to injection. Limit network and filesystem access, restrict tool parameters, require approval for sensitive operations and maintain destination allowlists where appropriate. Structured validation can reject unauthorised parameters before an API call. The application should be safe even when the language model misclassifies a document or follows persuasive untrusted wording. Defence in depth prevents a single interpretation error from producing a large incident.

Failure five: uncontrolled consumption

An automated workflow may retrieve entire document collections, recurse through tools or keep retrying a failing task. Costs can grow while outputs become less useful. Set ceilings on input size, tool calls, concurrency, execution steps and retries. Monitor both latency and actual bills. A lower-cost model may still be expensive if poor answers trigger repeated attempts and lengthy human correction. Financial governance should include engineering maintenance, monitoring and evaluation, not simply the provider's token prices.

Monitor drift in model and knowledge behaviour

Even without visible code changes, models can change and source documents can become stale. A business may publish new service hours or contract terms while its assistant continues using earlier material. Maintain a representative evaluation set covering normal questions, missing facts, sensitive data and adversarial examples. Test after changes to prompts, models, retrieval settings or important sources. Record baseline results and investigate worsening quality rather than treating continued fluent responses as evidence of consistency.

Distinguish errors by severity

Not every wrong summary has the same consequence. A mistaken internal draft that an employee corrects is different from a false delivery promise sent to customers or an unauthorised database modification. Classify incidents by confidentiality, integrity, availability, legal exposure and affected people. Set escalation thresholds and incident ownership in advance. Teams should not hide material failures inside a generic model feedback score. A clear severity model directs effort towards risks with real business consequences.

Make human supervision real

The phrase human in the loop can conceal weak controls if the reviewer sees only an approval button and a polished final answer. Provide the relevant original source, proposed action, expected result and uncertainty. Give staff time and authority to reject unsuitable output. Avoid requiring dozens of trivial approvals that create fatigue. The best review point is immediately before a consequential action, when the person can still prevent harm. Train reviewers to distinguish evidence from model inference.

Preserve forensic and operational records

When an agent behaves unexpectedly, investigators need to understand the inputs, retrieved documents, tool calls, decisions and results. Log these in a restricted, proportionate way, without exposing confidential content to unnecessary staff. Version prompts, model choices, evaluation datasets and integration settings. Keep source identifiers where authorised. An audit trail allows the team to identify whether the failure began in stale knowledge, unsafe permissions or incorrect agent planning rather than repeatedly adjusting prompts without understanding the cause.

Practise controlled failures before launch

Use a safe test environment to simulate missing data, conflicting instructions, network timeouts, forged source text and excessive action requests. Check that the agent abstains, escalates or stops within configured limits. Confirm that a failed tool call does not silently mark the business task completed. Rehearse how operators disable an integration or revoke permissions when the system behaves unexpectedly. Recovery capability matters as much as ideal-path accuracy, particularly for systems connected to client data or operational infrastructure.

Assign an owner to continuing risk

A deployed AI workflow needs a process owner, technical custodian, security escalation route and regular review of source freshness and cost. The builder may move to another project while the tool continues to handle customers. Define who can change prompts, approve new tool permissions and respond to a complaint. Avoid distributing responsibility so broadly that no one can stop a failing system. Ownership should include authority to suspend automation when an important control has broken.

Improve the architecture when a failure repeats

Repeated fabrication of order statuses often indicates missing verified integration, not insufficiently elaborate wording. Repeated exposure concerns suggest overbroad retrieval access. Duplicated actions suggest missing transaction controls. Treat recurring incidents as evidence about system design. Correct the underlying weakness, test the revised process and preserve the earlier outcome record. A growing archive of prompt patches without architectural change is unlikely to create a dependable service.

The pre-mortem outcome

A responsible AI project assumes that the model will sometimes misunderstand information and asks whether the surrounding system can keep that mistake bounded. Evidence checks address invented answers, data boundaries address confidentiality, least privilege limits actions, architecture contains injection, and cost plus quality monitoring detect slow degradation. These controls do not guarantee zero incidents. They make the system easier to understand, supervise and recover when something goes wrong.

Sources: NIST AI Risk Management Framework; OWASP GenAI Security Project. Controls must be adapted to the actual system and risk, not claimed as independent security certification.