Designing content moderation and guardrail systems

Story

Hide story

In the Signal Box, the false order from the dispatch bag now hangs in a frame beside the levers. The brass plate under it says REMEMBER. Fenn dusts it every morning.

The Network Office wants one rulebook for a thousand platforms. It must stop what is harmful and still let honest travellers through. The red lantern sits on its hook, unswung.

Where does Fenn draw the line, and who decides?

All counts, rates and capacities below are illustrative assumptions chosen for easy arithmetic. They are not measurements of any real classifier.

Concept#

A moderation systemModerationDetecting harmful content such as hate or violence. It is not an injection detector.Open the glossary entryClose decides, for each message going into or out of a GenAI product, whether to allow it, block it or send it to a person. GuardrailsGuardrailA check around a model, on input, output or actions. It reduces risk but does not solve prompt injection.Open the glossary entryClose is the wider term for these checks plus the limits around the model (Guardrails, safety and prompt injection covers the attacks and each layer’s basics; this lesson designs the system around them).

It uses the same chain as Designing a RAG QA system: requirements, scale, components, data flow, failures, evaluation, cost and latency, security, rollout.

Why it matters#

Moderation is a trade, not a switch. Block too little and harmful output reaches users. Block too much and honest users are refused, lose trust, and leave. Neither error can be removed, only moved, so the design question is who bears the cost of each mistake and who decides where the line sits.

Mental model#

Think of airport security with three lanes. Most travellers walk through quickly. A few are clearly stopped. A band in the middle gets a manual check. Staffing the manual lane decides how wide that band can be, and every adjustment to the metal detector’s sensitivity either delays more honest travellers or lets more risk through.

Map it to the system: a fast automatic check, a clear block, a human lane for borderline cases, and a sensitivity dial (the threshold) that a person with authority sets.

Requirements#

For a consumer chatbot:

• Policy: a written list of categories (for example self-harm, hate, sexual content involving minors, regulated advice), each with a stated severity.
• Coverage: screen user input and model output.
• Latency: moderation must not add more than a small, stated share of the response time. Assume 100 ms for the input check.
• Fairness to users: a blocked user gets a clear message and a way to appeal.
• Auditability: every decision is logged with enough context to review it.
• Out of scope: prompt-injection defence for agents with tools, which belongs to the layers in Guardrails, safety and prompt injection. A moderation classifier scores harmful content; it does not by itself detect text that tries to steer a model.

The layers, as a menu#

Layers are options you combine, not a fixed order to recite. Each catches a different failure, and each has a cost.

Serial, windowed or parallel: choose by harm severity#

The old rule of thumb “check output in parallel with streaming” is a choice, not a law. The right pattern depends on what happens if an unsafe token reaches the screen.

Two cautions. First, a stream that has already shown text cannot be un-shown, only retracted, so parallel checks are weak where severity is high. Second, some moderation services return scores only after the whole output is available, according to OpenAI’s documentation (accessed 2026-10-09), which pushes you towards buffering or your own windowed check. Read the current documentation of whichever service you use.

Input checks have a trade-off too. Running them serially saves the cost of a generation you would discard. Running them in parallel with generation saves latency and lets you cancel the generation if the check fails, at the price of some wasted work. For severe categories, serial is the safer choice.

Threshold as a business decision: worked numbers#

Assume 1,000,000 messages a day, of which 0.5% truly violate policy: 5,000 violations and 995,000 benign messages.

Moving from A to B catches 4,900 − 4,500 = 400 more violations and blocks 29,850 − 9,950 = 19,900 more honest messages. That is about 50 honest messages blocked for each extra violation caught (19,900 / 400 = 49.75).

Is it worth it? There is no technical answer. It depends on the harm of one missed violation in this product, which is why a children’s education product, a medical-advice assistant and an internal developer tool should sit at different points. The product owner and policy team set it, with engineering supplying these counts. Notice also that precision depends on the base rateBase rateHow common the thing you are detecting is. When the false-positive rate is not far below the base rate, most blocks are false positives.Open the glossary entryClose: when the false-positive rate is not far below the base rate, most blocks are false positivesFalse positiveIn a guardrail or classifier, a legitimate request or item that is blocked or flagged.Open the glossary entryClose (here a 1% false-positive rate against a 0.5% base rate). A much lower false-positive rate would raise precision a lot.

Set a threshold per category, not one global number. A category with severe harm gets a strict threshold and a human lane; a category about tone can be lenient.

Escalation and capacity#

Instead of a single cut-off, use two: block above a high score, allow below a low score, and send the band in between to a person.

Capacity sizes the band. Assume 2,000 items a day fall in the middle band and a reviewer handles 120 items an hour over an 8-hour shift: 120 × 8 = 960 items a day. Then 2,000 / 960 = 2.08, so you need 3 reviewers (not 2) for that band, or you narrow the band. If you do not staff it, the queue grows and “escalate” silently means “ignore”.

Give users a path: a clear refusal message that does not leak the policy internals, and an appeal that reaches a person. Log blocked content with its context so false positives can be fixed and classifiers improved, with access limited and retention bounded.

Components and data flow#

Diagram: Input passes a classifier that blocks, queues or allows; allowed input reaches the model, whose output is checked in a way chosen by...

Failure modes#

Tuning with shadow mode#

Shadow modeShadow trafficSending a copy of real requests to a new version without showing its answers to users, to compare behaviour safely.Open the glossary entryClose runs a new threshold or classifier on live traffic, records what it would have done, and acts on nothing. You compare its decisions with the current system and with human labels.

Measuring is the hard part.

• Precision is measurable: label a sample of blocked items and count the false positives.
• Recall is hard. Among messages that passed, violations are rare. In the lesson’s numbers at threshold A, 500 missed violations sit among 1,000,000 − 14,450 = 985,550 passed messages, about 0.05%. A random sample of 2,000 passed messages would contain about one miss, too few to estimate anything. So sample near the threshold, add a red-team set with known violations, and use user reports.
• Per-slice metrics: report by category, language and product surface.

Rollback: thresholds and classifier versions live behind flags, so reverting is a config change. Suppose the block rate doubles after a change. If honest users are being refused, roll back first, because that is a config change and it stops the harm. Then check whether the change deployed (versions), whether traffic changed (a new campaign or abuse wave), and which categories moved, and re-test the change in shadow mode. Providers update their moderation models over time, so thresholds you tuned may need recalibration (OpenAI’s moderation documentation, accessed 2026-10-09).

Cost and latency#

Each classifier call costs time and, for model-based classifiers, money. Run cheap rules first, a small classifier next, and the expensive model only on the middle band. Cache decisions for identical inputs if the policy allows. Budget the input check at 100 ms (the assumption above) and count windowed output checks against time to first token. See Cost and latency optimization.

Interview perspective#

Provenance: common industry pattern; no company-specific claim.

Common prompts are “design moderation for a chatbot” and “how do you balance safety with annoying legitimate users”. A frequent mistake is presenting moderation as a single binary filter. Strong answers have layers, per-category thresholds set with the product owner, a human lane sized by capacity, an appeal path, and shadow-mode tuning.

Practice questions#

Provenance: generated practice.

• Why might you buffer output for one category and stream it for another?
• How would you set the threshold differently for a kids’ education product and an internal developer tool?
• What happens to a request that was blocked incorrectly?
• The block rate doubled after a change. How do you diagnose and roll back?

Activity#

Work through it in 25 minutes.

1. Take the table of counts and recompute threshold C: recall 95%, false-positive rate 2%. Find violations caught, benign blocked and precision. (Check: 4,750 caught, 19,900 benign blocked, precision about 19%.)
2. For three messages (clearly fine, clearly harmful, borderline), say which lane each takes and why.
3. Choose serial, windowed or parallel output checking for a medical-advice bot and for a creative-writing tool. Justify each by harm severity.
4. Reflect: who in your organisation should sign off on the threshold?

Homework#

Design the input and output guardrail architecture for a GenAI product of your choice: categories and severities, thresholds per category, output-check pattern per category, the human lane with a capacity calculation, the appeal path, the shadow-mode plan and the rollback trigger.

Quick revision#

• Layers are a menu: input classifier, policy prompt, output check, human lane, logging and appeals.
• Choose serial, windowed or parallel output checks by harm severity, because streamed text cannot be un-shown.
• Thresholds are per category and set by the business; show the counts of both error types.
• When the false-positive rate is not far below the base rate, most blocks are false positives.
• Size the human lane from reviewer capacity.
• Tune in shadow mode; recall is hard to measure, so sample near the threshold and use red-team sets.
• Keep thresholds behind flags so rollback is a config change.

Check yourself#

• Checkpoint question (answer aloud, no notes): Design moderation for a consumer chatbot: layers, thresholds, latency budget, the output-check pattern and how you tune false positives.
• Independent re-test: The block rate doubled after a threshold change. How do you diagnose and roll back safely?
• Common misconception to avoid: One global threshold, no measurement of false positives, and treating a moderation classifier as a prompt-injection defence.

Related#

• Guardrails, safety and prompt injection
• Observability and eval pipelines
• Designing a RAG QA system