# Catch AI mistakes before your client does: build a practical quality gate in 90 minutes Source: https://getstartedwithai.net/tutorials/red-team-ai-output Site: Get Started With AI Tagline: Build AI automations that actually run your business. Audience: solo professionals, freelancers, consultants and AI implementers in training Contact: hello@getstartedwithai.co.uk This Markdown companion is generated from the prerendered page for AI crawlers. # Catch AI mistakes before your client does: build a practical quality gate in 90 minutes AI outputs look confident even when they are wrong. A hallucinated statistic, a mis-attributed quote, a process step that works in one context but not the client actual stack. The output arrives in fluent prose with no visible crack in it, and that is the problem. Fluency is not accuracy. A quality gate changes that. It is a short, repeatable set of tests you run on every AI output before it leaves your hands. Consultants who build one stop relying on gut feel and start shipping work they can defend. This tutorial shows you how to build one in 90 minutes. You will leave with three things: a test-case bank for your most common AI tasks, a review checklist that runs in under five minutes per output, and a confidence protocol that tells you when to run a second pass before delivery. ## Prep (5 minutes) - The three AI tasks you use most in client work. Pick the ones you run every week: meeting note summaries, first-draft reports, research briefs, data extracts from uploaded documents, or similar. You will build test cases from these. - A sample output for each task. Pull a real output from the last two weeks, not one you crafted carefully. Pick ordinary ones. - A blank document or Notion page for the checklist. Name it AI Quality Gate and date it. ## Step 1: Name the failure modes for each task (20 minutes) Every AI task has a characteristic failure mode. The point of this step is to name yours before a client finds them for you. Open each sample output. Read it as a sceptic, not as someone who wrote the prompt. Ask: if this is wrong, what kind of wrong would it be? There are four categories worth checking on almost any output. - Factual errors. Statistics, dates, names, version numbers. AI models are trained on data that goes out of date, and they often interpolate plausible-sounding figures when the real one is not in the training set. Every number in the output needs a source you can actually open. - Scope drift. The output answers a slightly different question than the one you asked. Particularly common in summarisation tasks where the model loses the specific angle requested. - Omission. The output is accurate about what it includes but misses something material. A legal clause, a pricing condition, a named exception the client already knows about. Absence of error is not evidence of completeness. - Instruction drift. The model followed the prompt on the first output and then drifted on a follow-up. Common in chain prompting where later calls do not re-read the original brief. - Write these four, plus any task-specific failure modes you have already encountered, as a numbered list under the task name. That list becomes the test-case bank for that task. ## Step 2: Build the test-case bank (20 minutes) A test case is a minimum check. It is not a full re-do of the task. It answers one question: did this output get this specific thing right? For each task, write three to five test cases. The format is: claim, source, pass or fail. For a meeting summary output, test cases might look like this. - Test 1: The summary lists four agreed actions. When you check the original transcript (lines 14 to 22), the actual count is three. Fail. - Test 2: The price agreed in the meeting is stated as 8,500 pounds. Source: transcript line 47. Pass. - Test 3: The next meeting date is listed as Thursday the 12th. When you check the transcript (line 61), the date agreed was Thursday the 19th. Fail. - Two failures in three test cases tells you the output is not ready. It tells you before you send it. - Build the same structure for each of your three priority tasks. The bank does not need to be exhaustive. Five test cases per task, run every time, catches more errors than a vague read-through catches in an hour. ## Step 3: Write the five-minute review checklist (15 minutes) The checklist is the gate every output passes through before it leaves your screen. It runs in under five minutes and it does not require you to research everything in the output. It is a prompt list, not an audit. A practical starting point for most AI consulting outputs: 1. Does the output answer the question as asked, or a similar but different question? If the scope has drifted, send it back. 2. Does every number have an identifiable source? List any number you cannot immediately verify, and check each one before delivery. 3. Does the output name any tools, products, or versions your client uses? If so, confirm each one is current for their stack. 4. Is anything missing that the client briefed you on? Run through the original brief and tick off each item. 5. Does the output include any absolute claims (always, never, all, none)? Absolute claims are almost always wrong or unprovable. Replace them with scoped claims. 6. If you ran a second prompt on the same input, would the output be materially different? If you are unsure, run the second-pass protocol. ## Step 4: Build the second-pass protocol (15 minutes) A second pass is a prompt you run specifically to find errors in a first output. It is not asking the AI to redo the work. It is asking it to act as a reviewer. Run it in a separate chat window with no prior context, asking the model to list any claims that could be factually wrong, note any place where the scope seems to have drifted from the brief, and flag any absolutes with whether each is defensible. Tell it to return findings as numbered items and not to rewrite the output. - The second-pass protocol is not triggered on every output. It is triggered by three signals: the task is high-stakes (board presentation, pitch deck, legal brief), the output is longer than 600 words, or you noticed something in the five-minute checklist that you cannot immediately resolve. - Keep a note of how often the second pass catches a real error versus returning a clean result. After 20 uses, you will know whether your first-pass checklist is calibrated correctly or whether certain tasks routinely need the second pass regardless. ## Step 5: Embed the gate in every client handover (10 minutes) A quality gate only protects a client if it runs before delivery, not as an afterthought. The way to make it stick is to attach it to the handover step, not to the drafting step. 1. In your project folder or Notion page for each client, add one line to the handover checklist: AI Quality Gate run, date, pass or fail summary. 2. When a deliverable fails one item on the checklist, fix it and note the failure mode. After 10 engagements you will see whether a particular task type produces errors reliably. That tells you where to tighten the prompt, not just the review. 3. In the WIRE Method Embed phase, the quality gate is the governance asset that makes the system trustworthy to run without the consultant in the room. Clients who see a named gate process trust the system more than clients who take the consultant word for it. ## What you ship A test-case bank with three to five checks per task, built on real outputs rather than theory. A five-minute review checklist that runs on every deliverable before it leaves your hands. A second-pass prompt you can trigger on high-stakes or long-form outputs. A handover note that shows clients the gate ran. Most consultants who build this report catching at least one material error in the first ten uses that would otherwise have reached a client. ## What to build next Pair this gate with the AI readiness assessment to set client expectations before any AI system goes live. The governance light tutorial extends the gate into a repeatable bus-test protocol for automations that run without a human in the loop. ## Related builds More tutorials in Become an Implementer. Each one runs in 60 to 90 minutes and ships a finished asset. ### Land your first paid AI engagement in 90 days: the 2026 consultant playbook 30 to 60 minutes. Implementer. [Open tutorial](/tutorials/how-to-become-an-ai-consultant-2026) ### Ship five repeatable client engagements: the AI automation agency playbook 30 to 60 minutes. Implementer. [Open tutorial](/tutorials/ai-automation-agency-practitioner-playbook) ### Win client work with a free first call: the AI readiness assessment template 30 to 60 minutes. Implementer. [Open tutorial](/tutorials/ai-readiness-assessment-free-template) ### Set rates that pay back in week one: pricing AI implementation work 30 to 60 minutes. Implementer. [Open tutorial](/tutorials/pricing-ai-implementation-work) ## Questions people ask before they start ### Does this add significant time to each deliverable? The five-minute checklist adds five minutes. The second-pass protocol adds ten to fifteen minutes when triggered. On a two-hour piece of client work, that is under 15 percent overhead in exchange for removing the most likely client-trust failure. Most consultants find the checklist pays for itself the first time it catches something. ### How should I price a quality gate into client work? If the gate is built as a standalone deliverable, price it as a half-day process design session at your standard rate, typically 400 pounds to 900 pounds, plus the ongoing cost of the review time embedded in each output. If you build it as part of a larger implementation, add 10 to 15 percent to your delivery estimate and name the gate as a named deliverable in the proposal. ### Can I show clients the checklist? Yes and that is often worth doing. A one-page named process signals to the client that the AI outputs they are receiving have passed a structured review, not just a read-through. Several consultants use the gate document as a trust-building asset in discovery calls. ### What if the second pass returns errors the first pass missed? Fix the output, then look at why the first checklist missed it. Add a test case to the bank. The bank is a living document. After ten to fifteen engagements it will carry more weight than any generic quality framework because it reflects the specific failure modes you have actually encountered. ### Does this work for automations as well as one-off AI outputs? For automations, the governance light tutorial covers the recurring check protocol. This tutorial covers one-off outputs: reports, summaries, briefs, analysis documents. Both are worth having.