A $900 million checkbox
On 11 August 2020, Citibank set out to pay about $7.8 million in interest to Revlon’s lenders. It sent that, plus almost $900 million of its own money: the whole loan, three years before it was due.[1]
The loan software couldn’t do what the team needed in one step. The workaround was to enter the loan as fully paid off and park the principal in an internal account, which meant ticking three boxes: FRONT, FUND and PRINCIPAL. They ticked one.[1]
Three people checked that payment before it left the bank. All three believed one box was enough. Citi’s own filing later blamed human error and “limitations in Citi’s loan processing systems”.[2] Lenders holding about $500 million refused to send it back, and it took the bank two years and an appeals court to win that argument.[1][2]
I keep coming back to this story because nobody in it was careless. Three trained people read the same screen and got it wrong in the same way. When that happens, I stop looking at the people and start looking at the screen.
Nothing on that screen was off-brand. The checkbox was a perfectly standard checkbox. What it meant was the problem.
Now picture screens like that one being built by an AI agent, by the dozen, in an afternoon. That’s where a lot of enterprise software is heading. So this is the question I want to answer: when AI builds the interface, who makes sure every signal still means what it should?
- Consistency gets you halfway. An audit can prove the agent used your red. It can’t prove red was the right call.
- Give every signal one meaning, and write down the judgement the agent can’t see in your code.
- Test behaviour as well as CSS. That’s the layer that protects the person using the screen.
AI builds faster than we check
46%
Don’t trust the output // 2025
of developers doubt the accuracy of what AI tools produce. Only 33% trust it.[3]
32%
Can rely on the output // 2025
of designers and developers in Figma’s survey, though 78% say AI makes them more efficient.[4]
Trust hasn’t kept up with adoption. In the Stack Overflow survey, 66% of developers picked “almost right, but not quite” as their biggest frustration with AI tools.[3]
In most products, almost right is annoying. In a high-stakes one, it’s the dangerous kind of wrong. A broken screen gets caught in review. An almost-right screen ships.
And once a signal stops meaning one thing, people stop reacting to it. Hospitals learned this the hard way. By the Joint Commission’s 2013 estimate, 85 to 99% of alarm signals need no clinical action, so staff learn to tune them out. In three and a half years of voluntary reports it logged 98 alarm-related events. In 80 of them, the patient died.[5] Every red button that isn’t a real hazard is a small step down the same road.
Right value, wrong meaning
Most teams get their design system ready for AI by locking down values. Tokens live in the repo. The agent reads the specs. The build fails on a hardcoded colour. All of that is worth doing, and I’d keep every bit of it.
The catch is that an agent can put the correct red on the wrong button and pass every one of those checks. Try it on the screen below. Everything on it comes from an approved token.
An agent built this screen from approved tokens only. Run both audits.
| No. | The value was right | The meaning was wrong | What it costs |
|---|---|---|---|
| 01 | Warning banner | Wrapped around an AI suggestion | People learn to dismiss warnings. |
| 02 | Data table | Cuts a part number short with an ellipsis | Two different parts look identical. |
| 03 | --color-danger | On a routine “Delete draft” button | Danger now means two things. The real hazard loses its urgency. |
| 04 | Action menu | Test and live options sit side by side | One wrong pick reaches real people. |
That last row is close to a real case, with a twist. On 13 January 2018, an officer in Hawaii sent a missile alert to the whole state. It wasn’t a mis-click. They believed the drill they were hearing was real.[6] What the investigation found is that nothing stood between one person’s wrong belief and everyone’s phone. The menu listed the internal test and the live alert together on one screen. One person could send it with no second sign-off. And nobody had planned for how to take a false alert back.[7] The correction took 38 minutes.
That’s the user to design for: someone who is wrong and sure of it. An AI agent is that user, many times a day.
Value drift is a brand problem. Meaning drift is a safety problem.
One layer, three moves
Keep your tokens, components and patterns. Then add one layer above them that everything underneath has to obey.
Safety layer
Signal meaning, criticality, authority, evidence
Foundations and tokens
One meaning per signal. Normal is quiet.
Components
Criticality, failure modes, degraded states
Patterns
Confirmation, alarm hierarchy, AI output
Move 01Give every signal one meaning
This one is borrowed from the flight deck. In 1982, Airbus delivered an A300 built around what it called a “dark, quiet cockpit”. A switch light stays off while its system runs normally, and comes on when the system is off or has failed.[8] On a dark panel, the one lit switch is hard to miss. It’s the idea this site’s own design system is built on.
Aviation goes further and writes the meanings into law. US certification rules sort every flight-deck alert into warning, caution or advisory by how fast the crew has to respond. Red is for warnings, amber is for cautions, and using either colour for anything else has to be limited.[9] I call my top level hazard, because in most web design systems “warning” already means amber.
Do the same with your tokens. Name each one by what it means, then decide which components are allowed to use it. If a “Delete draft” button reaches for the hazard colour, the build fails.
/* Named by meaning, never by colour */ --signal-hazard: var(--red-700); /* act now: someone could get hurt */ --signal-caution: var(--amber-500); /* act soon: something is getting worse */ --signal-advisory: var(--blue-300); /* good to know: nothing is wrong yet */ --signal-normal: transparent; /* normal is quiet */
--signal-hazard Means: act now, someone could get hurt Allowed in: alarm-banner, e-stop, status-indicator Never in: delete, cancel, form errors, brand accents
Move 02Write the judgement down
An agent can read your component code. It can’t read your head. So for every component that could hurt someone or cost real money, write down four things.
- How critical it isInformational, operational, or safety-related.
- How it could misleadWhat could someone misread, miss, or trigger by accident?
- What it shows when data is stale or missingNever a zero that looks like a real reading.
- Who has the final sayThe system, the AI, or a person.
Then point your agent’s instruction file at those specs, so every session starts from the same rules and not from a fresh guess.
See example spec
# Status indicator ## Metadata Category: atom Status: stable Criticality: Level 3 (safety-related) ## Overview Shows the live state of a monitored parameter. Measured values only. Never use for AI predictions. Use prediction-chip instead. ## States normal, advisory, caution, hazard, stale, no-data, source-fault ## How it could mislead - Colour alone fails for colour-blind users and in glare. Every state pairs colour with an icon and a text label. - A stale value shown as normal is worse than no value. Past the freshness limit, switch to stale and show the last update time. ## Stale or missing data - stale: neutral fill, clock icon, "Last updated 14:32" - no-data: dashed outline, "No data". Never render a zero. ## Final say Display only. Acknowledgement happens in alarm-list, not here. ## Evidence Hazard log HZ-014 (misread stale value). Usability round 3, task 7. (IDs are illustrative.)
Move 03Test behaviour, not only CSS
Add a handful of automated checks that protect the person using the screen. Most of them are as mechanical as a colour audit.
| Check | What it catches | How to automate it |
|---|---|---|
| Restricted signals | Signal colours used outside their approved components | Match usage against an allowlist |
| Colour never alone | Status shown by colour with no icon or label | Component test on every state |
| Safe focus | Irreversible actions with default focus on Confirm | Interaction test |
| No silent truncation | Critical fields cut off with an ellipsis | Style audit on critical fields |
| AI provenance | AI-generated content with no label | Component test on AI components |
One rule on severity. In a prototype, brand slips can start life as warnings. Safety slips fail the build from day one, because prototypes have a habit of becoming the reference everyone copies.
A named person signs off on what counts as critical. The agent can propose. It doesn’t decide.
Where is your team?
A quick way to place yourself. Each stage builds on the one before it.
Stages 0 to 3 // Consistency- 00VibesThe agent guesses every value, every session.
- 01TokensA closed set of values exists. Nothing enforces it.
- 02Specs in the repoThe agent reads your specs at the start of every session.
- 03Value auditThe build fails on hardcoded values.
- 04Safety layerEach signal has one meaning, and critical components carry the four notes from move 02.
- 05Behaviour audit and evidenceSafety rules fail the build, and every critical spec traces back to a real hazard or test finding.
By stage 3, AI output is consistent. Stages 4 and 5 make it something you can trust.
Run it on your project
Paste this into Claude Code, or whichever coding agent you use, at the root of your project. It sets up the scaffolding for every stage above. The evidence is yours to fill in.
Make this project's design system safe for an AI agent to build with, in a high-stakes product. Work in this order and report as you go. 1. Inventory Scan every stylesheet and component. List every hardcoded colour, spacing, font size, weight, radius, shadow, z-index and duration. Group them by category, count them, and name the worst files. 2. Tokens Create tokens.css. Components reference project tokens only. Project tokens point at the upstream design system where one exists. Name status colours by meaning, not appearance: --signal-hazard, --signal-caution, --signal-advisory, --signal-normal. Normal is quiet. 3. Signal semantics Find every status or alarm colour, icon and sound. List where each is used and what it appears to mean. Flag anything with two meanings. Write specs/safety/signal-semantics.md: one meaning per signal, the components allowed to use it, and forbidden uses. 4. Criticality Propose a level for every screen and component: Level 1 informational, Level 2 operational (errors cost time or money), Level 3 safety-related or irreversible. Mark every Level 2 and 3 proposal PENDING until I approve it. When unsure, ask me. Don't guess. 5. Specs Write specs/ for foundations, tokens and every component that exists. Each spec covers purpose, anatomy, tokens, states and an example. Level 2 and 3 specs add: criticality, how it could mislead, stale and missing data states, who has the final say, and evidence. Leave evidence as TODO. Never invent hazard IDs, requirements or test results. 6. Safety patterns Write specs/safety/ files for irreversible actions, data integrity, alarm hierarchy and AI output. 7. Audit Write a script that fails CI on: - hardcoded visual values (suggest the right token) - signal tokens outside their allowed components - status states that rely on colour alone - Level 3 components missing stale or no-data states - irreversible actions with default focus on the confirm button - ellipsis truncation on fields marked as critical data - AI-generated content without a visible label - Level 2 or 3 specs missing their safety sections Safety failures are errors. Brand failures may start as warnings. 8. Replace Swap every hardcoded value for its token. Run the audit until it passes. Never edit the audit or a spec to make it pass. 9. Instructions Add to CLAUDE.md, AGENTS.md or whichever file your agent reads at startup: read the relevant specs before any UI change. Use only tokens. Never give a signal a new meaning. If a request conflicts with a safety spec, stop and say so. Finish with a report: open TODOs, unclassified components, and every place you were unsure.
Three lines do most of the work: “When unsure, ask me”, “Never invent hazard IDs” and “Never edit the audit to make it pass”. An agent that fills gaps with confidence is the exact thing this setup exists to stop.
Treat the first run as a draft. Read every spec it writes before you trust it, the same way you’d review a new teammate’s first pull request.
What should change
The people using the screen
Red means one thing again. Fewer almost-right screens should reach them, and the ones that do should fail in ways they can spot.
Design and engineering
The agent can build faster without someone checking every pixel by hand. Review time moves to the decisions that need a human.
The business
A paper trail you can show an auditor. Two months after the Revlon payment, Citibank’s regulator fined it $400 million for a long-standing failure to fix its risk management and internal controls.[10] That order was about years of gaps, not one wire. But a payment screen is one of those controls.
These are the outcomes the setup is built for, not results I can promise you. Measure them on your own product before you promise them to anyone else.
Consistent pixels are the floor. Consistent meaning is the job.
Your design system is already a safety system. Start treating it like one.
Fair questions
Is a design system really a safety system?
Not in the certified sense, and I’m not claiming that. I mean it’s one of the controls that sits between a hazard and the person at the screen. If that control can drift, it deserves the care you’d give any other one.
Isn’t this overkill for a prototype?
Prototypes become the thing people build from, so their mistakes travel. Start small, with signal meanings and criticality levels. Those two catch the most dangerous drift.
Does this replace a hazard analysis?
No. It gives your hazard analysis somewhere to live in the design system, and puts its conclusions in front of the agent every session. If your domain already has its own criticality classes, use those in place of my three levels.
Can a passing audit make us overconfident?
Yes, if you let it. The audit only checks rules someone has written down. A hazard nobody has thought of passes every check. Keep testing with real users and keep your hazard reviews. The audit’s job is to stop known mistakes from coming back.
We’re not aerospace or medical. Does this apply?
If a wrong screen can cost someone money, a compliance breach, a lawsuit or their safety, yes. The criticality levels let you scale it down for everything else.
What about work that starts in Figma?
Name your signal variables by meaning there too, and keep the specs where designers and agents can both read them. For now, the enforcement lives in code.
Where the facts come from
- In re Citibank August 11, 2020 Wire Transfers (opens in new tab), US Court of Appeals, Second Circuit, 2022. The court’s account of the payment, the three checkboxes and the appeal.
- Citigroup annual report for 2020, Form 10-K (opens in new tab), US Securities and Exchange Commission. Citi’s own explanation, and the $504.2 million that wasn’t returned.
- 2025 Developer Survey (opens in new tab), Stack Overflow. More than 49,000 responses from 177 countries.
- Figma’s 2025 AI report: perspectives from designers and developers (opens in new tab), Figma. Survey of 2,500 Figma users.
- Sentinel Event Alert 50: Medical device alarm safety in hospitals (opens in new tab), The Joint Commission, 2013. PDF. Reporting is voluntary, so the real numbers are higher.
- Worker who sent Hawaii false alert thought missile attack was imminent (opens in new tab), NPR, 2018.
- Report and recommendations: Hawaii Emergency Management Agency, January 13, 2018 false alert (opens in new tab), Federal Communications Commission, 2018. PDF. The menu, the missing second sign-off and the 38-minute correction.
- Airbus cockpit/control milestones (opens in new tab), Flight International, 1995.
- 14 CFR 25.1322, Flightcrew alerting (opens in new tab), US Code of Federal Regulations. The warning, caution and advisory levels and their colour rules.
- OCC assesses $400 million civil money penalty against Citibank (opens in new tab), Office of the Comptroller of the Currency, 2020.