Shipping AI in Production: Seven Decisions That Separate a Demo from a Durable Product

Most AI features fail in the system around the model. Uneven traffic, a fixed launch date, a unit-cost ceiling, and customers who notice every bad answer will expose a design that only worked in a demo.

After 30 years of building and operating technology platforms, this is how I take an AI feature from a decision to a product the business can depend on.

Seven decisions cover the work:

1. Let constraints choose the architecture.
2. When the feature slows down at peak, diagnose before you redesign.
3. Route simple work to smaller models, and prove the router before it takes traffic.
4. Design generative output around a business outcome.
5. On a fixed launch date, scope is the only pressure valve.
6. Cut known risk early, and keep flexibility only where the uncertainty is real.
7. Split hybrid workloads by economics, not by where the servers already sit.

Let constraints choose the architecture

Before picking a provider or a pattern, I pin down five things: latency and availability targets, data residency and compliance, expected load and how spiky it is, the unit-cost ceiling per request, and what the team can realistically operate.

With a one-quarter timeline and day-one reliability requirements, the answer is almost always managed services. Buy the undifferentiated infrastructure. Own two things: a model gateway and your evaluations.

The gateway is the most important reliability decision you will make. The application should never call a model directly. The gateway handles routing, retries, fallback to a secondary model, rate limiting, prompt versioning, and cost metering. No single model or provider should be a single point of failure.

I start with a managed foundation-model API, using the cloud provider's own service when residency or procurement requires it. I self-host only when unit cost or sovereignty forces it. Retrieval runs on a managed vector store next to the system of record, because retrieval is where most answer quality comes from. Inputs and outputs are filtered, the feature degrades to a non-AI path when the model path fails, and every request is traced end to end.

On that quarter, the calendar is short:

• Month one. Constraints, architecture, and the evaluation set.
• Month two. Build, then shadow mode.
• Month three. Canary, then general availability.

When it slows down at peak, diagnose before you redesign

Peak slowdowns in AI features usually come from one of three places: provider token limits, scaling on the wrong signal (CPU instead of concurrency), or a slow dependency such as retrieval. Tracing tells you which one. Don't redesign on assumption.

Then fix it in three layers.

Absorb the load. Keep interactive requests synchronous and streaming, and push everything else to queues: summaries, batch enrichment, background analysis. Autoscale on queue depth and in-flight concurrency, and keep warm capacity for known peaks. Reserve provisioned throughput for the baseline and burst above it. Spread load across regions, or across a secondary model, through the gateway.

Reduce the load. This is the cheapest scaling there is. Cache shared prompt context, and cache repeat answers where the response is stable. Tighten retrieval so prompts carry fewer tokens. Fewer tokens means lower latency and lower cost.

Protect the system. Give each hop a time budget. Retry with backoff and jitter, and cap the retries so a failure cannot become a retry storm. Put a circuit breaker on each provider and dependency, and fail over to the fallback model or the non-AI path. Give each tenant or tier its own concurrency pool, so one noisy customer cannot starve everyone else. When the system is saturated, shed low-priority traffic on purpose.

Decide the shed order before the incident, based on business priority. Paying tiers and revenue-critical flows stay up. Each tier has an SLO, and that SLO is what makes the order explicit.

Each of these controls adds a moving part, so add them in order of return and prove each one. Load test at two to three times the forecast peak. Inject faults, such as a provider outage or a slow retrieval layer. Alert on SLO burn rate, not on raw error count.

Keep the synchronous path thin. Push everything else to asynchronous workers. Reduce work before adding capacity. Degradation should be a designed behavior.

Route simple work to smaller models, carefully

Model routing often removes a large share of heavy calls. Keep the implementation simple.

• Classify requests at the gateway with cheap rules first: request type, input length, whether retrieval is required, and customer tier.
• Send simple requests to a small model and complex ones to a large model.
• Escalate automatically when the small model's output fails validation.
• Keep the rules in configuration, behind feature flags.

The main risk is a silent drop in quality. Run the router in shadow mode against the evaluation set before any live traffic moves. Then watch three things: the escalation rate, because a rate that is too high means you pay twice; quality broken out by segment, because averages hide damage; and real cost savings against a baseline. Roll out from shadow to 5 percent, then 25 percent, then 100 percent, with a kill switch at every stage.

Design generative output around a business outcome

If the model writes customer-facing responses, a fluent answer is not a success metric. Define the KPI first: resolution rate, conversion, time to resolution, or customer satisfaction. Then build toward it.

• Ground every response in approved sources and the customer's own data.
• Check every output before it reaches the customer: policy violations, personal-data leakage, unsupported claims, and prohibited commitments.
• Match human involvement to risk. Low-risk responses are automated. Medium-risk responses are drafted by the model and approved by a person. High-risk requests go straight to a human.
• Close the loop. Agent edits and escalations are the most valuable improvement signal you will get.
• Prove the change with an A/B test against the current experience.

Measure quality in production the same way you gate a release. Score a golden dataset before any prompt or model change ships. After launch, review escalations, edits, and negative feedback by segment on a set schedule, and roll a segment back when its quality moves the wrong way. The business KPI and the quality sample have to move together. A faster answer that lowers resolution rate is a failed release.

Run fixed-date delivery with scope as the only pressure valve

When several teams are involved and the launch date cannot move, scope is the only real lever. Five practices keep the plan honest.

• Plan backward from launch, with a minimum launchable scope and a ranked cut list agreed up front.
• Name one owner per milestone, and appoint one delivery lead for the integrated plan.
• Give leadership a one-page weekly report. Green means the exit criteria were demonstrated, not claimed.
• Start long-lead items in week one: security review, compliance, and procurement. Escalate anything blocked for more than 48 hours.
• Hold a scope checkpoint at the midpoint and a go/no-go two weeks out. If you are behind, cut scope or narrow the launch cohort. Never cut testing or guardrails.

Know the tradeoff between cutting scope early and cutting it late

Cutting scope late keeps options open, and it wastes built work and forces the decision under pressure. Cutting early gives certainty and focus, and you decide with the least information you will ever have.

Cut early wherever the risk is already known, especially high-risk, low-value work and anything that depends on a party you do not control. Keep flexibility only where the uncertainty is real and the value is high. Because the cut list is pre-decided, a late cut becomes a planned move.

Split hybrid workloads by economics, not preference

When an existing virtualized environment is in play, decide workload by workload.

Keep on-premises the steady, predictable workloads running on capacity you have already paid for, data with gravity or a residency constraint, and latency-sensitive integrations.

Move to the cloud the spiky, AI-facing request path, managed model APIs and GPU capacity, and new components that need speed to market.

Make the call with a three-year total cost that includes licensing, hardware refresh, staffing, and egress. Design the boundary between the two environments deliberately. Egress and latency can quietly destroy a business case. Revisit the split every year as prices, residency rules, and load change.

The same pattern in every decision

Measure before acting. Buy what does not differentiate you. Protect the customer path. Make failure a designed behavior. Tie every technical choice to a business outcome someone can see on a weekly report.

That is what turns an AI demo into a product the business can depend on.

If a pilot needs to become a fixed-date launch, book a strategy call and we will scope the architecture, evaluations, and delivery plan against the outcome you need. The case studies show the same standard on platforms already in production.