Production AI Agent Architecture: 5 Reusable Patterns from DoorDash & Shippy for Shipping Agents to Production

TL;DR: How do you actually put an AI agent into production?

The hard part of a production AI agent is not choosing the model — it's the five engineering things that sit around the model: (1) wrap the nondeterministic model in deterministic tools; (2) put a shared MCP layer between orchestration and business logic; (3) give it layered memory injected by semantic retrieval; (4) evaluate the agent, not the model; and (5) split the work between a domain team and a platform team, FDE-style. This isn't theory. Below we reverse-engineer a copyable reference architecture from two first-party 2026 cases — DoorDash's "Ask DoorDash" assistant and the Allen Institute's Shippy — where those five patterns show up independently and point to the same conclusion.

What does a production AI agent architecture actually look like? (demo vs. production)

A demo agent is a model with a prompt and a few API calls. A production agent is a system around a model: the model decides, but deterministic infrastructure constrains, remembers, evaluates, and isolates. The clearest way to see the gap is to line the two up side by side — and every "production" cell below is backed by DoorDash or Shippy, not by opinion.

Dimension	Demo-grade agent	Production-grade agent
Tool calls	Model assembles raw API calls	Purpose-built deterministic CLI/tools (auth, pagination, structured output)
Architecture	Business logic stuffed into the prompt	Orchestration decoupled from business logic via a shared MCP layer
Memory	Single-turn context / no memory	Layered memory (long-term / session / agent) injected by semantic retrieval
Evaluation	Manual spot-check, run once, eyeball it	Automated eval framework scoring the exact version + real data
Isolation / safety	Shared context	Per-user ephemeral sandbox; guardrails written into the system prompt (auditable)
Organization	One person / one model	Domain team builds agents × platform team runs orchestration, tools, memory, eval

DoorDash's assistant, as an InfoQ analysis of "Ask DoorDash" reports, is the production column made concrete: an assistant runtime that only orchestrates, backed by shared tools, layered memory, and a standing evaluation harness. Shippy — the Allen Institute's maritime-monitoring agent built on Skylight — is the same story from a different industry. The rest of this piece walks the five patterns.

How do you make a nondeterministic agent reliable? Give it deterministic tools

The founding insight comes straight from the Shippy team, and it is worth quoting in full: "Agents are nondeterministic. You can't control what the model decides to do, but you can make the tools it reaches for predictable." (Hugging Face Blog, 2026-07-15.) You don't fight the model's uncertainty; you engineer certainty into everything it touches.

Shippy learned this the hard way. Early prototypes let the model assemble API calls directly and produced "a steady stream of subtle bugs" — pagination errors that silently dropped data, geometry-encoding mistakes, calls that looked right but returned the wrong data. The fix was to replace ad-hoc API wrangling with purpose-built CLIs that own authentication, pagination, and structured output, then test each layer in isolation. As the team puts it, "Each layer narrows what the next layer can get wrong."

DoorDash reaches the same conclusion at a different altitude. In its architecture, business capabilities — product search, recommendations, cart, checkout, order history, user memory — are not written into prompts. They are exposed through a shared MCP layer, so the assistant runtime only orchestrates reusable tools rather than improvising business logic. Even resource updates run as deterministic operations that don't call the LLM at all. Same principle, both cases: the model is the unpredictable part, so you shrink the surface where its unpredictability can do damage.

How does an agent remember context without losing control? Layered memory + deterministic operations

Memory is where demos quietly break: they either forget everything between turns or dump an unbounded history into the prompt. DoorDash's answer is three tiers of memory — long-term offline memory, session memory, and agent memory — ranked by semantic vector retrieval and injected into the prompt only when relevant. Memory is a retrieval problem, not a "stuff more tokens in" problem.

And the payoff is measurable, which is the part demos never get to show. Per the InfoQ write-up, computed consumer memory lifted grocery checkout conversion by ~24%, cart size by 17%, and cut conversation turns by 7%; on restaurant search, open-ended queries saw conversion +15%. Co-founder Andy Fang framed the user-facing effect bluntly: "Ask DoorDash builds a cart about 5× faster than doing it manually — one prompt, done in under two minutes." Layered memory plus deterministic updates isn't architectural elegance for its own sake; it moves the business metrics.

How do you evaluate whether an AI agent is any good? Evaluate the agent, not the model

This is the pattern most teams skip, and it's the densest evidence in both cases. The reframe is simple: you are not benchmarking a model, you are grading the exact agent you're about to ship, on real data.

DoorDash operationalized this into standing infrastructure — more than 2,000 automated evaluations per day, which raised its quality score by 8 points and compressed regression testing from six hours to twenty minutes. In one model migration, the eval harness let them cut latency by 35% with no drop in quality. The reason they invested so heavily is captured by Raghav Saboo, DoorDash's head of recommendations and search: "Building a useful AI assistant is hard. Knowing whether it's actually good is harder."

Shippy states the principle in one line — "evaluate the agent, not the model" — and runs it through the Harbor framework with expert-weighted rubrics, always against the exact version and real data in play. That discipline is what surfaces the failures a model benchmark never would: the agent overstepping to give tactical advice, boundary simplifications that caused missed detections, and — memorably — "inventing a CLI command that doesn't exist." An eval loop that scores the shipped agent on real inputs is the difference between "the model scored well" and "the product works." For teams extending this into ongoing production monitoring, our companion pieces on agent reliability and observability and observability, audit, and evaluation go deeper on the runtime side.

How should teams organize to ship an agent to production? FDE × platform split

Architecture is also an org chart. DoorDash draws a clean line: domain teams build the specialized agents, while a platform team maintains the orchestration, MCP tools, memory, evaluation, and shared components. Domain experts own behavior; the platform owns the reusable machinery underneath. This is precisely the Forward-Deployed Engineering (FDE) × platform division 6AM builds around — and we're citing it because two independent production teams landed on it, not to sell it.

Shippy shows the same split at the component level. Its agents are assembled from three versioned parts — Soul (a system prompt that sets persona and behavioral boundaries), Skills (frontmatter-tagged markdown following the agent-skills spec, versioned and auditable), and Config (model and harness runtime settings). Each user interacts with Shippy in an ephemeral, per-user Kubernetes sandbox with the user's JWT injected at provision time, so every API call is scoped to that user's data — which is how Shippy serves 300+ partners across 70 countries without leaking data between them. And its guardrails live in the system prompt, not in fine-tuning: as the team notes, keeping boundaries explicit "makes them auditable and easy to revise" — for example, Shippy refuses to make legal determinations, because "that is a determination for people, not an agent." Boundaries you can read and change beat boundaries baked into weights.

The production agent checklist (the copyable blueprint)

Collapse the five patterns into something you can tick off before you call an agent "production-ready":

• Deterministic tools — the model never assembles raw API calls; every capability is a purpose-built tool that owns auth, pagination, and structured output.
• MCP / decoupled orchestration — business logic lives in a shared tool layer, not in prompts; the runtime only orchestrates.
• Layered memory — long-term / session / agent memory, injected by semantic retrieval; prefer deterministic (non-LLM) operations wherever you can.
• Evaluate the agent, not the model — an automated eval harness that scores the exact shipped version on real data, with weighted rubrics, running continuously.
• Per-user isolation + explicit guardrails — ephemeral, scoped sandboxes; safety boundaries written into the system prompt so they stay auditable and revisable.
• FDE × platform org split — domain teams build agents; a platform team owns orchestration, tools, memory, and evaluation.

If your agent demos well but you can't check most of these boxes, you don't have a production problem with the model — you have the gap between demo and production that why AI agents fail in production describes, and this blueprint is the "how to do it" other side of that story.

This is exactly the work 6AM does — forward-deployed engineers who grow AI into your business processes with the deterministic tooling, evaluation loops, and platform underneath. If you're trying to close your own demo→production gap, start with a production-readiness diagnosis, or browse more answers in our FAQ.