The demo worked. Production is a different job.

Why AI projects stall after the demo

A demo shows that something can work. Production needs it to work every day, on real data, at a known cost, for people who rely on it.

Engineers at Google described the gap in a paper that is still widely cited, presented at NeurIPS in 2015: (Sculley et al., 2015)

only a tiny fraction of the code in many ML systems is actually devoted to learning or prediction (Sculley et al., 2015)

The rest is data handling, connections to other systems, configuration and monitoring. The same paper warns that it is “dangerous to think of these quick wins as coming for free.” Google's own architecture guidance puts it plainly: “the challenge is building an integrated ML system and to continuously operate it in production.” (Sculley et al., 2015) (Google Cloud)

A survey of published deployment case studies, in ACM Computing Surveys, found that practitioners “face issues at each stage of the deployment process”: getting the right data, choosing metrics that reflect the business, testing edge cases, integrating with existing systems, monitoring, and updating the model as the world changes. (Paleyes et al., 2022)

The steps below take those stages one at a time. The AWS Well-Architected Generative AI Lens, a widely used review framework, covers the same ground under its pillars of operational excellence, security, reliability, performance and cost. (AWS)

Agree what success looks like

Agree before you build what the AI must achieve, in numbers the business already tracks.

Google's Rules of Machine Learning put this second, right after “don't be afraid to launch without machine learning”: “First, design and implement metrics.” Without an agreed number, nobody can decide when the system is ready, or whether it still is six months later. (Zinkevich, Google)

Agree the launch threshold too: the minimum quality and the maximum cost at which the business would switch it on.

Get the data right

An AI system is only as good as the data it works with.

Check three things. Is the data complete: are the documents, product details or policies the AI relies on all there? Is it current: who updates it when prices, products or rules change? And is it allowed: may this data be used for this purpose, and by this vendor?

The case studies show data problems at several stages, from too little varied data before launch to data that changes after it. Monitoring for these changes is part of Step 5. (Paleyes et al., 2022) (Google Cloud)

Know whether it's good

You need a way to measure output quality, led by someone who knows your business.

When an AI product disappoints, the cause is rarely the model itself. More often nobody can show whether its output is good enough, so problems are found by customers instead of by checks, and every change is a gamble.

NIST, the US standards institute, advises against judging AI performance “from narrow, non-systematic, and anecdotal assessments,” and recommends evaluating it in real-world scenarios. OpenAI's evaluation guide adds that evaluation is “a journey, not a destination”: it continues after launch. (NIST AI 600-1) (OpenAI)

In practice: collect real cases, have a domain expert grade them, turn the standard into automated checks, and run them before launch and on every change. How strict depends on what a mistake costs.

Keep the design simple, with guardrails

Simpler systems are cheaper, faster and easier to keep reliable.

Anthropic's advice on agents: “you should consider adding complexity only when it demonstrably improves outcomes.” Fixed workflows, where the steps are set in code, come before autonomous agents. Google's Rule #4 makes the same point for machine learning: “Keep the first model simple and get the infrastructure right.” (Anthropic, 2024) (Zinkevich, Google)

Where an agent is used, it needs limits: stopping conditions such as a maximum number of steps, and checkpoints where a person can step in. OpenAI names two triggers for handing over to a person: when the agent fails repeatedly, and before actions that are sensitive, irreversible or high-stakes, such as refunds, cancellations or payments. (Anthropic, 2024) (OpenAI)

Make it run like software

In production, the AI is one part of a software system, and needs the same basics as any other.

Monitoring matters more than for ordinary software. Google's guidance notes that it lets you “detect performance degradation and model staleness”: results can get worse without any change on your side, because the data or the users change. The AWS lens lists consistent output quality, monitoring, traceability and graceful handling of failures among its core practices. (Google Cloud) (AWS)

Control costs that grow with success

AI usage is billed per use, so costs grow with every user and every request.

Know the cost per request and per user before launch, and what it will be at ten times the volume. The AWS lens lists the levers: choosing cost-effective models, balancing the cost of each request against quality, and writing instructions that use fewer tokens. (AWS)

Caching helps when many requests share the same content, such as the same policy documents. With Anthropic's models, for example, reading cached input costs a tenth of the normal input price or less, and responses on long documents start faster. (Anthropic)

Use smaller, cheaper models where the quality checks from Step 3 show they are good enough.

Make sure people use it

A tool that works but isn't used is still a failed project.

Involve the future users early: let them shape it, test it on their real work, and change the workflow around the tool, not just add the tool to it. DORA's research found that a focus on users amplifies AI's benefits, and warns that without it, teams “may just be moving quickly in the wrong direction.” (DORA, 2025)

Measure actual use after launch, not only quality: how many people use it, for what, and where they work around it.

Name an owner, and know your exit

AI systems need ongoing care. Someone has to watch them, decide on changes and answer for them.

The models underneath change and disappear. Anthropic retires older models and notes that “requests to retired models will fail,” with at least 60 days' notice. OpenAI gives at least six months' notice for generally available models, and shorter for preview models. A system built on one model will have to move to another, probably more than once. (Anthropic) (OpenAI)

Prepare for it: keep the test cases from Step 3 so any new model can be compared on your own work, avoid depending on one vendor's special features where you don't need them, and know what switching would take. Anthropic's own advice is to test with newer models well before the retirement date. (Anthropic)

Production checklist

Before launch, check each of these.

• One goal with a number, and a launch threshold, agreed with the business.
• The data is complete, current and allowed to be used.
• Quality is measured on real cases, with a domain expert.
• The design is as simple as the task allows; agents have limits and hand over risky actions.
• Logging, error handling, tests, security review and roll-back are in place.
• Quality is monitored after launch.
• Cost per request is known, including at higher volume.
• Future users were involved, and actual use is measured.
• A named owner, and a plan for when the model changes.

What this would look like for you